Agent-first: The gap behind the AI productivity promise

AI agents are increasingly positioned as digital workers that can own an entire task. But in an agent-first model, real productivity counts only when the output has been reviewed, corrected, and accepted.

Key takeaways

• Measure AI productivity from task assignment to accepted output, not to the moment the agent responds.

• Review capacity, not generation capacity, may become the limit on scaling digital workers.

• Codify expert acceptance criteria, and automate the checks that can be expressed as rules.

• Set agent autonomy based on the impact of an error and how easily it can be detected.

• Protect learning opportunities for junior staff, who will become tomorrow’s reviewers.

In early 2025, a controlled study found that experienced developers believed AI made them 20% faster, while measurements showed they were actually 19% slower [3]. A year later, the same research team said AI had likely begun to speed developers up, but acknowledged they could not yet measure by how much [4]. The tools are changing faster than our ability to measure them. That makes this a good moment to take a closer look at the promise of digital workers.

Digital workers come with a compelling promise: more work done in less time. But between an output that has been generated and an output that is good enough to use lies the work of the people who review it, fix it, and take responsibility for it. That work must be fully accounted for when organizations evaluate the value of AI.

Consider a sales quote. An agent prepares the draft, fills in the product list, and summarizes the commercial terms. The account owner still has to verify pricing, scope of supply, payment schedules, and customer commitments. A single wrong clause can wipe out much of the time saved.

If the number of quotes generated goes up but review capacity stays the same, work piles up on the reviewer’s desk. The organization produces more output, but customers don’t necessarily get results any sooner.

This is the risk to plan for when moving to an agent-first model: a way of organizing work in which AI agents own complete tasks, rather than suggesting next steps like an assistant (copilot). It also requires people to rethink how they supervise and evaluate work. Review capacity may become the ceiling on how far digital workers can scale.

What the demo doesn’t show

According to a case study published by Anthropic, Doctolib piloted Claude Code with 30 engineers before rolling it out to the entire team [1]. The tool was integrated into software maintenance and technical documentation workflows. The platform team also built a shared library of prompts, custom commands, and subagents.

What stands out is that usage was standardized alongside deployment. Engineers had common guidelines for assigning tasks and bringing results into their workflows.

In Vietnam, digital workers are also moving closer to everyday business operations. Local solutions have been introduced for tasks such as preparing quotes, supporting contract drafting, and consolidating input from multiple departments. As that happens, the question is no longer what the agent can do, but whether its output is solid enough to use.

Real-world processes always involve conditions that a demo can’t fully capture. 

Which prices are currently valid? Which agreements has the customer confirmed? Which legal comments are still unresolved? Who has the authority to approve exceptions?

An output can look complete and still lack the grounds to be used. A person or a suitable review mechanism is still needed to catch those gaps.

That’s why effectiveness has to be measured up to the point where the work is accepted. Stopping the clock when the agent responds misses the effort that comes after.

Where does the definition of “done” live?

Imagine an organization that has kept complete records of past projects. When a similar project comes along, the team still has to track down the people who ran it to ask why a particular approach was chosen, why a line item went over budget, or why a seemingly simple requirement took several rounds of discussion.

Records may capture decisions without the reasoning behind them. Experienced people remember the constraints, the warning signs, and the options that were ruled out.

That knowledge directly shapes acceptance. Two proposals that meet the same list of requirements can still differ significantly in feasibility. Reviewers need to understand the context to judge them.

When assigning work to agents, as much as possible of what used to rely on experience and implicit understanding should be written down. If the instruction is simply “do it like the previous projects,” reviewers will have to fill in the context for every output themselves.

Scaling then remains dependent on a small number of experts. Organizations can provision more tools, but these experts’ time is limited.

A practical way to prepare is to document why changes were made, the conditions for accepting exceptions, and common mistakes. That content should be reviewed and turned into shared guidelines and test cases.

Not every judgment can be written as a rule. That’s exactly why organizations need to decide which parts can be checked against clear criteria and which still require an expert’s call.

The parts that have become clear criteria can be handed to an automated verification layer: checking prices against the current price list, scanning contracts for mandatory clauses, running tests against code, or using another agent to cross-check the work. When machines filter out measurable errors first, experts can spend their time on the parts that truly require judgment.

What does a meaningful approval require?

Putting a human in the approval step isn’t enough to conclude that the output is under control. Reviewers need expertise, information, and time. They also need the authority to send work back or stop it when the grounds are insufficient.

An Anthropic study of Claude Code published in June 2026 [2], based on roughly 400,000 sessions from October 2025 to April 2026, found a link between user expertise and session outcomes. Under the study’s strictest success criterion, novice sessions succeeded about 15% of the time, while sessions at intermediate expertise or above reached 28–33%.

These figures should be read within the limits of the methodology. Expertise and outcomes were assessed from session content using classification models. The study only estimates the relative value of tasks; it does not measure whether the final output was actually used.

Even so, the findings offer a basis for rethinking the role of experts in agent deployments. Experts should be involved from the start, when tasks and evaluation criteria are defined. If they only appear at the final step, they may end up correcting both how the work was assigned and what was produced.

An approval step is meaningful only when reviewers are genuinely able to challenge the work. If too many outputs arrive at once, review time shrinks, schedule pressure pushes approvals through, and the mechanism risks becoming a formality.

Who will become the next experts?

This raises a hypothesis worth considering: handing most execution work to agents may reduce learning opportunities for new employees.

People learn to build cost estimates by doing the calculations themselves, missing things, and being corrected. Developers build understanding by tracking down why code doesn’t run. If newcomers start out by reviewing agent output, they may be asked to evaluate work they have never done themselves.

This hypothesis has not been confirmed by long-term research. Still, it raises a question about workforce development: how will the people who replace today’s experts build their experience?

Team leads can reserve some tasks for new employees to complete on their own before using agents. For complex work, newcomers should evaluate results alongside experts, who explain why an output is accepted or rejected.

This approach may take more time in the short term. In return, the organization preserves hands-on practice and passes on its evaluation methods. This training cost should be factored in when deciding which work to assign to digital workers.

If organizations count only the time to complete today’s tasks, they may overlook the time needed to develop the people who will be accountable tomorrow.

Measure the rework, too

The feeling of working faster needs to be checked against measurement.

In a METR study published in July 2025 [3], 16 experienced open-source developers predicted AI would make them 24% faster. After the study, they still estimated they had been 20% faster, while their measured completion time had increased by 19%.

These results reflect early-2025 tools. In February 2026, METR published an update [4]. Raw data from the new experiment pointed toward AI speeding developers up, but the research team considered the data unreliable. Many developers declined to participate or excluded tasks they didn’t want to do without AI. Agent users often switched to other work while waiting, making completion time hard to measure. Output quality, such as the amount of testing and documentation, also differed between the two conditions.

These challenges show that AI productivity can’t be judged by perception or by a single universal number. Each organization has to measure it in its own work, all the way to the point of acceptance.

Time should be measured from task assignment to acceptance. Costs should include tooling, reviewers, rework, and error handling. These measurements should be compared against how the work was done before deployment.

A minimum set of metrics might include:

 Permissions to act also need to be defined precisely. Preparing a quote is different from sending it to a customer. Proposing a data correction is different from updating the system directly. The level of autonomy should depend on the impact of an error and the ability to detect it before it causes harm.

One way to frame autonomy is along two axes:

With this evidence in hand, leaders can decide which tasks to scale, which to adjust, and which should still be performed directly by people.

The productivity promise of digital workers must therefore be verified where the work ends: when the output has been accepted and can be put to use.

Before assigning more work to agents, organizations must be able to answer three questions: who will review it, against what criteria, and how long it will take. The time after an agent reports “done” is exactly what must be counted to know how much the technology has really helped.

References

[1] Anthropic. Doctolib accelerates developer productivity with Claude Code (Customer case study). https://claude.com/customers/doctolib 

[2] Hitzig, Z., Massenkoff, M., Lyubich, E., Zhang, S., Heller, R., & McCrory, P. (June 16, 2026). Agentic coding and persistent returns to expertise. Anthropic Research. https://www.anthropic.com/research/claude-code-expertise

[3] Becker, J., Rush, N., Barnes, B., & Rein, D. (July 10, 2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

[4] Becker, J., Rush, N., Cunningham, T., Rein, D., & Mahamud, K. (February 24, 2026). We are Changing our Developer Productivity Experiment Design. METR. https://metr.org/blog/2026-02-24-uplift-update/