Before changing your startup hiring plan because of an AI benchmark, map the role's actual tasks, identify which have relevant evidence, and test the missing work in your own workflow. Record the human responsibilities that remain before approving or cancelling the vacancy. A model score can inform that investigation; it cannot make your staffing decision.

Imagine you are about to hire someone for customer onboarding. A new model performs well on a benchmark, and a demo produces an impressive onboarding document. Your team asks whether the vacancy is still necessary. Examine what the document leaves out: missing customer evidence, conflicting commitments, internal handoffs, and someone available to resolve exceptions.

Mercor's [benchmark map, currently being discussed on Hacker News](https://news.ycombinator.com/item?id=50121179), gives that question a timely starting point. It examines how much economic work appears in a selected set of AI benchmarks. Its findings warrant checking what has been measured before turning a score into a staffing commitment.

## What does AI benchmark coverage tell you?

Benchmark coverage tells you whether particular tasks appear in an evaluation. It does not tell you whether a model succeeds at every covered task, how it performs with your tools, or whether a whole job can be automated.

Mercor's initial [map of the US economy](https://www.mercor.com/research/labor-index/benchmark-map/) includes **29 selected benchmarks**. It reports **924 covered tasks out of 18,797**, using **923 occupations**. In its main definition, a task counts as covered when it maps to at least five benchmark prompts. A single prompt can map to more than one task.

The headline estimate is that roughly **90% of American working hours** are spent on tasks with little or no coverage in that index. Keep the scope attached to the number. It describes coverage in a selected benchmark collection, with estimated time weights. It does not mean AI fails at 90% of work, or that 90% of jobs are safe from automation.

There are material limits. Mercor says the collection is incomplete and provisional. It limits benchmarks by job family and selects against benchmarks already saturated by frontier models. Those choices can push estimated coverage down, by an amount the researchers cannot bound. Treat the map as evidence of measurement gaps, rather than a census of everything that has ever been tested.

The mapping also involves judgment. Mercor uses a language model to associate prompts with occupational tasks. Experts reviewed first-choice mappings for an APEX subset, accepting **76.7% of 1,763 judgements**. The map counts other suggested mappings too, which that review did not fully assess. That is a useful check with a defined boundary. [Mercor's methodology](https://www.mercor.com/research/labor-index/benchmark-map/methodology/).

Independent research raises a related concern. Wang and coauthors analyzed **43 benchmarks and 72,342 tasks**, finding a mismatch between programming-heavy agent development and the distribution of human work. Their study uses a different sample; it does not independently confirm Mercor's headline estimate. [How Well Does Agent Development Reflect Real-World Work?](https://arxiv.org/abs/2603.01203v2).

For your hiring plan, find out which parts of the proposed role the evidence actually represents.

## Which tasks does your hiring decision depend on?

Start with the work that needs doing, its arrival pattern, and the consequence of leaving it unfinished. A job title is too broad to establish what an AI result would need to change.

The official [O*NET occupational framework](https://www.onetcenter.org/overview.html) organizes information about tasks, activities, skills, and other job characteristics. It can help you notice missing work. Its occupation profiles cannot describe the exact mix at your startup, where one person may combine several functions and depend on informal handoffs.

Ask the people handling the workload to walk through recent cases. Include a routine case, a delayed case, and one that required an exception. Use existing work records to reconstruct what happened. Identify who supplied missing information, who made a decision, and who checked that the result was usable.

Write tasks as observable actions. “Support onboarding” gives you little to test. “Prepare a customer onboarding pack from approved account records” gives you an input and an output. “Resolve a conflict between the signed agreement and the account notes” identifies a different task, even if both appear under the same job title.

Include coordination. A finished draft may still wait for an account owner, a customer reply, or access approval. If those dependencies determine when onboarding finishes, leaving them outside the inventory would make the evidence less relevant to the vacancy.

Also distinguish work you need from work you could stop doing. A test of an unnecessary report cannot establish the staffing requirement for a growing customer queue. Agree on the required outcomes before debating whether a person or a system should produce them.

Our broader [headcount planning guide](/blog/headcount-planning-humans-ai-agents-new-hr-role) covers the capacity decision. Here, you are preparing one input to it: a clear account of the tasks whose performance would change your plan.

## Build a task-to-evidence table before testing AI

Connect each important task to a real case, an acceptance standard, and the evidence available. This is a proposed planning method, not a procedure validated by Mercor's research.

The table describes a **hypothetical customer-onboarding workload**. It contains no observed results, customer data, or industry performance targets. Its purpose is to show what you would need to learn before changing the hiring plan.

| Task | Case to include | What a usable result requires | Evidence to collect |
|---|---|---|---|
| Prepare an onboarding pack | Complete, consistent account records | Required facts supported by the approved records | Missing facts, incorrect details, and edits needed before use |
| Handle incomplete inputs | A required customer detail is absent | Identify the gap and request the relevant clarification | Whether the gap was noticed and who resolved it |
| Resolve conflicting information | Account notes disagree with the agreement | Preserve the conflict and route it to an authorized owner | Evidence supplied, escalation quality, and remaining decisions |
| Coordinate a handoff | The next owner is unavailable | Leave a clear state and follow the agreed fallback | Who receives the work and whether the handoff finishes |
| Recognize an action limit | A request exceeds the system's authority | Stop at the boundary and explain what needs approval | Whether an unauthorized action was attempted or taken |

Add a column for any relevant public benchmark. Record what matches and what differs: inputs, tools, context, task length, or grading. “Related benchmark exists” is useful discovery information. It does not establish that the case has already been tested under your conditions.

Use separate states for **untested**, **attempted but unacceptable**, and **accepted under the stated conditions**. An untested escalation is an unknown. Calling it a failure would overstate the evidence; treating it as a pass would conceal the gap.

Choose the acceptance standard before viewing results. For the pack, that could mean every required factual statement has support in an approved record. For a conflict, the acceptable output could be a documented escalation rather than a completed pack. A system that correctly stops may have done the right work.

Keep the evidence manageable. You need enough detail to explain the decision, including important omissions. A small initial set can reveal problems and guide another test, but it cannot establish a general success rate. There is no universal number of cases or pass threshold supplied by the research above.

## Test the workflow you would actually run

Evaluate the proposed setup with its real inputs, permissions, tools, and human assistance. Document the conditions so a successful result does not quietly acquire a broader meaning.

If the proposed workflow includes a person checking every pack, test that arrangement. If it assumes access to account history, supply the access you could actually authorize. Record help from colleagues, including explanations that the system would need again on an unfamiliar case.

Prepare cases using information you are entitled to use in the chosen environment. Remove unnecessary customer details while preserving the difficulty you intend to test. If removing context makes the example materially different, record that limitation instead of presenting it as a complete reproduction of the work.

Compare with the current process under comparable conditions. Staff should not receive the hardest cases while the proposed system receives only complete records. Conversely, stripping people of ordinary colleagues and context can make the baseline artificial. Write down what each side could see and what assistance it received.

Mercor's accounting research illustrates why those conditions matter. In an [October study of four simplified month-end-close scenarios](https://www.mercor.com/blog/human-baselines-for-benchmarks-ai-now-outperforms-junior-accountants/), Mercor reports that leading models outperformed the 12 participating licensed CPAs. That comparison used unassisted people and does not establish gains from AI assistance. Mercor also explains that the setting removed coworkers, accumulated context, and parts of the wider job. Strong bounded-task performance can coexist with uncertainty about an entire role.

For onboarding, inspect the finished state as well as the generated text. Was the pack usable? Was the missing evidence obtained? Did the conflict reach an authorized person? Preserve cases that remain unresolved at the decision deadline, with the reason and next owner.

If you change the model, instructions, or available tools, identify the revised setup. Better results afterward describe that setup. They do not isolate which change caused the improvement, or prove that the earlier configuration would now pass.

The financial comparison deserves its own treatment. Our [pilot cost guide](/blog/ai-agent-pilot-cost-per-accepted-outcome) covers software, retries, review, and unfinished work. Keep those records alongside this table. The table shows whether you tested the work that matters to the vacancy; the cost record shows the resources it consumed.

## Turn the result into a bounded hiring decision

Use the test to explain what changed in your understanding of the workload. Then decide whether the evidence supports hiring, changing the process, or conducting another limited experiment.

Return to the hypothetical onboarding team. If preparation becomes easier but ambiguous cases still require the founder, you have learned something about the division of work. The next role may need operational judgment and authority to resolve cases. That conclusion would depend on the actual results and workload; it is not a finding from the example table.

Look at remaining work by person and availability. Small amounts of time freed across several colleagues do not automatically provide someone to cover a customer queue. An escalation requiring a specialist who is rarely available can constrain delivery even if document preparation improves.

Keep demand in view too. Testing today's cases does not establish how the workflow behaves when volume rises or the customer mix changes. A vacancy intended to cover growth needs an explicit demand assumption. Record the assumption beside the observed evidence so readers can distinguish what happened from what you expect.

A short decision record can make that reasoning reviewable:

1. **Decision:** State which vacancy or workload is under consideration and the deadline for deciding.
2. **Evidence:** Link the tested cases, setup, acceptance standards, and observed results.
3. **Remaining work:** Name the decisions, exceptions, and handoffs people must still handle.
4. **Limits:** List important conditions that were absent or untested.
5. **Action:** Approve the role, revise its scope, change the workflow, or run a further bounded test. Name the owner and review date.

Avoid making the record a formality for a decision already taken. Write down what evidence would change your view. If the test can only produce a justification for cancelling the vacancy, it cannot fairly assess whether a hire is needed.

A promising result may justify a narrower experiment while you continue recruitment. An inconclusive result may justify gathering missing evidence. A persistent gap in ownership may strengthen the case for a person with the right authority. These are possible decisions, with different commitments and evidence requirements.

Once you approve a hire, use the remaining work to define the role. The separate guide to [AI-assisted roles with real responsibility](/blog/ai-assisted-roles-meaningful-work-decision-rights) covers that next step. Your planning record should explain why the role exists; the job description should tell a candidate what they will own.

## Carry the approved role into Kit

Keep the benchmark comparison and internal workflow test in your planning document. Kit can help organize the hiring process that follows an approved role, but it does not provide a benchmark-coverage analyzer, workforce model, or internal AI test harness.

Put the actual work in the job description and explain the relevant exercise in candidate-facing stage instructions. Kit supports job-relevant work samples and team review stages. Adapt those stages to the role you approved, with clear expectations about what candidates should demonstrate.

The useful handoff is a concrete brief: the outcomes the person owns, the decisions they can make, and the work your experiment left unresolved. A public benchmark helped you ask a question. Your own evidence supplies the boundaries of the answer.

> [!CTA]
> **Have an approved role to fill?** Start with a [Kit hiring template](/templates), then adapt its stages and instructions to the work your team actually needs.