## What an Experiment Tests

An experiment compares separate campaigns using one outcome:

- **Reply rate** — any reply divided by messages sent
- **Positive reply rate** — replies classified as positive or interested divided by messages sent

Use one campaign as the control and at least one as a variant. Change one meaningful variable at a time—such as the subject, opening angle, call to action, or audience—so a difference is interpretable. Existing campaign history counts in its results, so use comparable cohorts and timing.

## Set It Up

1. Open **Outreach > Experiments** and create a draft with a name, outcome metric, and minimum sample size. The allowed minimum is 1–500 sends per campaign.
2. Open each campaign and assign it to the experiment as **Control** or **Variant**. You need at least two assigned campaigns before the experiment can start.
3. Check that sending identity, audience quality, follow-up policy, and timing are comparable unless one of them is the variable under test.
4. Start the experiment.

Kit does not automatically route new prospects across running experiment campaigns. Build comparable cohorts in each campaign before launch, or add prospects to each variant deliberately. Do not load one variant with a materially different cohort and call it a clean test.

Archiving a campaign from a draft experiment removes it from that experiment. After an experiment starts, Kit locks its campaign set so sample thresholds and results keep referring to the same variants.

## Read the Result

Kit shows sends and total reply rate for each campaign. Once every campaign reaches the minimum sample, Kit notifies the eligible team that the experiment is ready to review.

The minimum is a review trigger, not proof that a result is statistically significant. Small samples, unequal audiences, seasonality, deliverability, and multiple simultaneous changes can all mislead. If you selected **Positive reply rate**, inspect how replies were classified before choosing a winner—the current results table still shows total reply rate.

## Conclude, Then Apply

**Conclude** records the campaign you chose as the winner and stops the experiment from being a running test. It does not rewrite campaign configuration or move prospects.

The conclude form initially selects the campaign with the highest total reply rate. Treat that as a starting point, especially when the experiment's outcome is **Positive reply rate**: verify the result and select the intended winner before submitting.

A completed campaign cannot be selected as the winner and does not appear in the list. If the selected campaign is completed after conclusion, Kit stops **Apply winner** before changing any configuration or moving any prospect or message.

**Apply winner** is the separate rollout action:

- If the winner is not the control, its configuration replaces the control's configuration.
- Pending and drafted prospects move from losing campaigns to the winner.
- Prospects already being researched, already sent, or otherwise further through the workflow stay where they are.
- A losing prospect whose email already exists in the winner stays behind to preserve per-campaign uniqueness.
- Losing campaigns are marked complete.

Applying a winner is irreversible in the UI. Review the selected campaign, its sender and AI settings, and the remaining prospect counts before confirming.

## Experiment Checklist

- [ ] One outcome metric was chosen before launch
- [ ] Control and variants differ in one deliberate way
- [ ] Cohorts, timing, and sender conditions are comparable
- [ ] Every campaign has reached the sample floor
- [ ] Reply classifications and practical impact justify the decision
- [ ] Winner settings and remaining prospects were reviewed before rollout