A/B test calculator

Find the sample size a cold email A/B test needs, or check whether the reply rates you got are a real difference.

Updates as you typeRuns in your browser, nothing is sent anywhere

Send each version to 2,518 contacts.

To tell 3% from 4.5% with 80% power at 95% confidence, send each variant to 2,518 contacts (5,036 in total).

Per version

2,518

Emails delivered

In total

5,036

Both versions

Baseline

Your reply rate today

Target

4.5%

50% more replies

What a bigger or smaller lift costs

From a 3% baseline

10% lift to 3.3%

53,211 per version

25% lift to 3.75%

9,100 per version

50% lift to 4.5%

2,518 per version

100% lift to 6%

749 per version

200% lift to 9%

245 per version

Small lifts need far more emails. If you send 1,000 a month, a test for a 10% lift is not worth running: test a bigger change.

The stop rule

Fix the sample size before you start. Send to both variants at the same time, stop only when each has reached it, and then read the result once. A significant result before that point is not a result.

Email rate calculator

Then

Subject line tester

Put Ken on the rest of your outbound

Inboxes, warmup, leads and copy in one place. Ken runs these checks on every inbox it manages.

Start free - 1,000 credits, no card

What the A/B test calculator does

It answers the two questions every cold email test raises.

Before the test: how many emails each version needs to detect the lift you care about.
After the test: whether the difference you got is real or noise.

It uses a two-sided two-proportion z-test on reply or positive reply rates, at 95% confidence and 80% power by default. Baselines start at Ken's published rates, and you can enter your own.

How to read the result

In sample size mode you get the emails needed per version for the lift you entered. In significance mode you get the p-value, a confidence interval for the difference, a winner if there is one, and a plain verdict.

If you planned a sample size and have not reached it, the verdict says to keep collecting, even when one version looks ahead.

How to run a fair test

Decide the sample size before you start and do not stop early because one version is winning. Checking every day and stopping at the first good-looking result makes false winners far more likely.

Change one thing per test: the subject, the opener or the offer.
Split the list at random and send both versions on the same days, from the same inboxes.
Measure replies or positive replies, not opens.

Why it matters for cold email

Cold email rates are small, and small rates need big samples. Going from a 3% to a 4% reply rate needs 5,301 emails per version to detect reliably. A 10% relative lift, 3% to 3.3%, needs 53,211 per version. Most tests stop at a few hundred sends, which is why most cold email winners are noise.

Questions

How many emails do I need for a cold email A/B test?

It depends on your baseline and the lift you want to detect. At 95% confidence and 80% power, 3% to 4.5% replies needs 2,518 emails per version, and 3% to 6% needs 749. Smaller lifts need far more, which is why it pays to test bigger changes.

How do I know if my A/B test result is significant?

Enter the sends and replies for each version. If the p-value is under 0.05, the difference is unlikely to be chance. 30 replies against 45 on 1,000 sends each looks like a 50% lift, but the p-value is about 0.08, so it is not significant yet.

Can I stop the test early if one version is clearly winning?

No, not unless you planned for it. Checking repeatedly and stopping at the first significant result raises the chance of a false winner well above 5%. Set the sample size first and read the result when you reach it.

What is statistical power?

The chance the test detects a real difference of the size you set. At 80% power, a real lift of that size is missed one time in five. Higher power needs a bigger sample.

Should I A/B test subject lines on open rate?

Not anymore. Apple Mail Privacy Protection and security scanners open emails automatically, so open rates no longer measure what the subject line did. Test subject lines on reply rate.