Test Your Take-Home Assignment Instructions Before Hiring

To test take-home assignment instructions, ask someone unfamiliar with the brief to follow the setup and submission steps a candidate will receive. Observe where they need help, record unintended obstacles, revise the packet, and repeat with a fresh reader. Keep the skills you intend to assess in view throughout the exercise.

On October 11, Terence Eden described paying people to follow his README. He offered €25 for an hour, watched people share their screens and speak aloud, and revised the document between sessions. He reported spending around €150 in total. The experiment exposed a wrong demo link, confusing file operations, unexplained terms, and sections in an unhelpful order.

The Hacker News discussion concerns software documentation. Eden did not study hiring. A hiring team can apply this approach by rehearsing its instructions before using candidate work as evidence. A polished brief can still conceal a missing permission or an undefined final step.

What are your take-home instructions actually testing?

Write the intended skill before arranging the rehearsal. Use that reference point to decide whether a difficulty belongs to the assessment or comes from an avoidable flaw in how it is run.

Consider a fictional backend exercise: add a filter to a small API and protect existing behavior with tests. You might assess how the applicant understands the code, chooses a change, handles edge cases, and explains a decision. Those are plausible assessment goals if they resemble the work required in the role.

The US Office of Personnel Management describes work samples and simulations as tasks resembling actual job activities. Its guidance also distinguishes competencies needed on entry from those the employer intends to teach. That gives you a useful question for every prerequisite: does the job require this skill on entry?

Reading unfamiliar code may belong. That goal does not imply knowing your company’s unpublished database setup. Debugging can belong too, but you should decide whether a broken starter environment is the debugging task. A hidden failure changes what the exercise asks people to demonstrate.

Put your answer beside the review criteria. For the fictional API task, you could write: “We review correctness, preservation of existing behavior, relevant tests, and your explanation of tradeoffs.” If interpreting an underspecified request is intentional, explain how you will judge the assumptions candidates make.

The rehearsal checks whether the packet supports that intended assessment. A successful walkthrough does not establish that the exercise predicts job performance. For broader task design, use the separate guide to structuring code assignments.

Give a fresh reader the real candidate packet

Rehearse the invitation, access, instructions, and submission route together. Sending a colleague the repository alone misses the handoffs where candidates can lose track of what happens next.

Choose someone with the expected entry skills who did not write or review the task. A developer from another project can be useful. Someone entirely new to programming might expose a writing problem, but their difficulty installing the runtime would need a different interpretation for a senior engineering role.

Explain the purpose before they start: you are testing the assignment packet. Agree on the work and compensation involved if you recruit someone specifically for the session. Eden’s budget describes his experiment; it is not a standard price or recommended sample size. Do not turn an existing applicant into an unpaid usability tester without telling them.

Give your reader the same materials and permissions you plan to provide to candidates:

• The invitation message and any preparation instructions.
• The task summary, detailed instructions, and review criteria.
• Repository access, supplied data, and required credentials.
• The expected effort, deadline, and event that starts the clock.
• Rules for documentation, search, collaboration, and AI assistance.
• The actual submission route and a contact for questions.

Use a rehearsal account and data suitable for the session. If the exercise requires a particular environment, supply it explicitly. If applicants may use their own machines, try the documented path on an allowed environment that does not already contain your team’s dependencies.

Ask the reader to describe what they think they must produce before proceeding. Record that explanation. A different interpretation might reveal an unclear requirement, or it might be a reasonable professional choice within the freedom you intended to give.

For a coding task, have them continue far enough to run the baseline, make a small legitimate change, and complete submission. Agree on that scope beforehand. You need to observe the operational path without quietly demanding a full unpaid solution.

Observe where the instructions stop working

Record what happened before explaining what you meant. A hint may be necessary, but the need for it is part of the evidence you came to collect.

Ask the reader to speak aloud as they work: which instruction they are following, what they expect a command to do, and where they look for the next action. Avoid narrating the page for them. When they pause, allow them to explain the pause before deciding that it is a problem.

A pause to consider how filters should behave might show useful reasoning. A pause because the link says “starter repository” but opens a page they cannot access is an administration problem. Distinguish those observations in your notes rather than combining them into a single completion time.

Use a short blocker log. The following entries are invented examples for the API exercise, not results from an actual hiring study:

Add the command, error, or phrase needed to reproduce a problem. “Setup was confusing” is hard to act on. “The setup command expects a database name absent from the environment example” gives the owner somewhere to start.

If you intervene, write down the help you gave. Otherwise, the final result can make the packet look usable when the session depended on your explanation. Treat think-aloud observations as diagnostic evidence: being watched and speaking aloud can change how someone works. They do not estimate ordinary candidate performance.

Repair the brief and test the changes

Fix the smallest cause you can identify, then try the revised path with someone who has not learned your workaround. The first reader now knows details that the next candidate will not.

For the fictional API assignment, the original packet says: “Run the app, add filtering, and send your solution.” A revised packet might supply a supported runtime, exact installation commands, a baseline test command, example data, and the intended filter behavior. It would also name the submission destination and the review criteria.

Keep those repairs tied to observations. If the reader cannot install the dependencies, adding a paragraph about company culture does not address the problem. If the tests pass but the reader cannot find the submission action, the repair belongs at that handoff.

Check commands yourself before another observed session. Follow the sequence from a clean checkout using the documented environment. Remove an accidental dependency on your local credentials, cached data, or unpublished configuration. Automated checks can help catch broken links and commands; they cannot tell you how a person interprets “send your solution.”

Write a specific question for the next pass. For example: can this reader run the baseline using only the packet, and can they explain when the work is formally submitted? Record the observation in the same log rather than marking the entire exercise “validated.”

Retest the complete route after repairing a blocker that prevented later steps. A corrected repository invitation may expose a setup issue that the first reader never reached. Stop when you have enough evidence to address the concrete obstacles you found, while acknowledging the limits of the environments and readers you observed.

Keep candidate conditions and clarifications consistent

Settle the packet before inviting a cohort, and communicate material corrections consistently. Quietly changing requirements can leave reviewers comparing work produced under different conditions.

Save the invitation, instructions, starter revision, and rubric together in whatever document or repository your team already uses. Record which version went to each candidate. This is a proposed administration practice, not a claim that versioning alone makes an assessment fair.

The US EEOC’s employment tests and selection procedures guidance addresses appropriate purpose, administration, scoring, and validation. A fresh-reader rehearsal is one operational check. It does not satisfy all those responsibilities or establish legal compliance.

Suppose a candidate reports that the supplied data cannot be loaded. Confirm the problem, tell other affected candidates what changed, and consider whether their deadlines need adjustment. If the defect prevented someone from producing evidence, examine its effect before treating the missing result as a skill failure.

Keep logistical clarifications separate from coaching on the solution. “The upload destination is this page” can be shared with everyone. A discussion that supplies an approach to the core problem changes the help received and should be recorded for reviewers.

Once candidates submit, preserve the relationship between their packet and the evaluation. A reviewer should know the criteria that were communicated, including any intentional ambiguity. Later interview questions about a passing solution belong to a different step; see the interview after the solution passes.

Make the instructions accessible to the reader

Include how people navigate and read the packet in your rehearsal. A correct command is still difficult to use when it exists only inside a screenshot or under an unrelated heading.

The W3C WAI headings tutorial explains how meaningful, structured headings support navigation and understanding. In an assignment, headings such as “Setup,” “Required behavior,” and “Submission” give readers useful destinations. Check the actual rendered document, including any PDF or repository page you distribute.

Supply text commands that can be selected and copied. Give links names that identify their destination. Check that important instructions remain readable on the devices and document formats you support. If your submission page uses controls, test how to reach and use them with a keyboard instead of assuming that reading the brief covers the whole experience.

Offer a contact route before the exercise begins. In its US guidance on job applicants and the ADA, the EEOC explains that employers can describe the hiring process and ask whether applicants need an accommodation for that process. Handle requests through an appropriate private channel, keeping diagnostic medical details out of shared scoring notes.

A walkthrough with one reader covers that reader’s experience. It does not demonstrate accessibility for every user or full WCAG conformance. For the writing itself, the earlier guide to clear candidate instructions covers task expectations, deadlines, and receipts. This rehearsal adds evidence about whether someone can actually follow the configured route.

Keep the tested instructions with your Kit assignment

Use the candidate’s actual flow for the final rehearsal. Fields and previews help you prepare the packet; following the journey shows what those settings mean in practice.

In Kit Hiring, a Code Assignment can use a private GitHub repository created from your template. Its configuration includes instructions, estimated effort, and timing settings. The candidate connects GitHub and receives a repository invitation. Accepting that invitation starts the assignment clock. The candidate page then displays the instructions and submission action; a later hard deadline closes the window through automatic submission.

A Work Sample has a different start event. The candidate can read the summary first, then press Start task to reveal the private brief and begin the clock. They upload their result and receive a submission receipt. Keep those distinctions explicit when rehearsing your own task; the candidate experience documentation describes both paths.

Kit provides the assignment workflow. Your team arranges the fresh-reader session, records blockers, and decides which repairs preserve the assessment. Give an unfamiliar reader the invitation you intend to send, observe setup through submission, then revise and retest the obstacles you can reproduce. Do that before the first candidate has to discover them for you.