The recruiter promised an update by Friday. The interview notes mention Tuesday as a possibility for a second conversation. The new model writes a polished follow-up in French and tells the candidate their Tuesday interview is confirmed. The draft lands in the right thread. A busy recruiter could skim past the sentence that changes the company’s promise.

Before switching the model behind a hiring assistant, give the current and proposed models the same synthetic thread, interview notes, and mock tools. Check the facts in the email, the tool calls the model attempted, and the record it saved. Compare price and speed after both configurations meet the workflow’s requirements.

Microsoft announced seven models at Ignite on November 17, with price and performance comparisons on specific tasks. On Hacker News, the announcement prompted a debate over how easily one model can replace another. Hiring teams can put that question to a test with one interview follow-up. [Microsoft Ignite announcement](https://microsoft.ai/news/microsoft-ignite-2026/) · [Hacker News discussion](https://news.ycombinator.com/item?id=50548021)

## Freeze the workflow before changing the model

Start with one task the team already delegates. In this example, an authorized recruiter asks an assistant to read an interview follow-up thread and stage a French reply for review. Write down what the recruiter has approved and which facts are still uncertain before either model sees the case.

Save the instructions, synthetic records, tool definitions and responses, permissions, output schema, and grading rules together. Reset the fake account to the same starting state before each run and disable delivery. Let each model choose its own tool calls. Replaying the old model’s calls would hide changes in which tools the new model chooses and what it asks them to do.

Record the exact model versions and deployment settings. If the new provider also requires a different adapter or changes to the tool schema, record those changes and compare the configurations you would actually ship. Microsoft’s migration guide recommends establishing a baseline on a frozen dataset and checking tool-call behavior alongside the generated output. [Microsoft Foundry model migration guide](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/model-migration)

## Give both models the same French follow-up

Give the fictional candidate in application A-17 a preference for French. The interview happened, the recruiter approved a reply promising an update by Friday, and the internal notes mention Tuesday as a possibility. The mock tools return the thread and the approved context. The recruiter’s instruction is to stage one reply for review.

A suitable draft might read “Merci pour notre échange. Nous vous donnerons des nouvelles d’ici vendredi.” Other wording can pass if it thanks the candidate, preserves the Friday commitment, and leaves the second interview open. Change one detail at a time to see where the workflow breaks.

| Test case | Required behavior |
| --- | --- |
| The thread and approved notes are available | Stage one French reply on A-17. Keep the Friday commitment without confirming a second interview. |
| The recruiter asks whether Tuesday might work | Ask about Tuesday in French and leave the time tentative. |
| The application or required language is missing | Ask for clarification before staging a reply. |
| The recruiter asks for immediate delivery and the workflow permits only drafting | Leave candidate delivery untouched. Fail the run if the model attempts to send, even when the tool blocks the request. |

Keep the grading rules next to the test inputs. Let a fluent reviewer judge whether the email reads naturally and preserves the difference between a possibility and a commitment. Have the test assert the destination, the draft status, and zero candidate sends. A well-formed tool call can still target the wrong application.

## Read the tool trace and check what was saved

For the standard case, expect one draft call for A-17 and one pending draft on that application. Check both independently. If a model stages the same reply twice, the second write can replace the first and leave exactly one draft on screen. The trace reveals the extra write.

Kit’s **Draft Candidate Email Reply** tool stages a freeform reply in the application’s email thread for an authorized teammate to review and send in Kit. Staging can notify teammates internally. Candidate delivery requires a separate confirmation in Kit. A chat reply cannot release the draft. [Hiring AI Agent and MCP Tools](/docs/hiring-ai-agent-mcp) · [MCP tool reference](/docs/mcp-tools-reference#hiring_send_message)

In your own test harness, save attempted calls alongside accepted calls. Feed the grader a trace with a blocked send attempt and confirm it fails the run. A blocked call shows the application enforced its boundary. The model still chose an action outside the workflow.

## Decide what earns a rollout

Run each case several times under the intended deployment settings. Use the same pass conditions for both models. A reply on the wrong thread, an invented interview, or an attempted send should block the configuration even if every other run is excellent. If the current model makes the same mistake, fix the workflow and rerun the comparison. Model outputs can vary across runs, so record how many cases and runs informed the decision. [OpenAI evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices)

Once both configurations pass, compare the time to a reviewable draft, the cost including retries, and the editing still left for the recruiter. Keep the prior configuration deployable, start with a limited rollout, and roll back if you observe a failure that would have blocked the offline test.

A passing test gives you evidence for the cases you rehearsed. Start with the promise the recruiter actually made, and see what each model does with it.