Glossary
A/B testing
A/B testing is a controlled experiment that shows two versions of something, such as an email subject line or a landing page, to randomly split groups of people and compares a chosen metric to see which version performs better.
What Is A/B Testing?
In short You change one thing (the variable), keep everything else the same, and assign people at random to version A (the control) or version B.
After enough people see both, you compare the metric you chose in advance, such as reply rate, and check whether the difference is larger than chance would produce. Optimizely describes statistical significance as a measure of how unusual your results would be if the change had no effect.
Small samples are the common mistake. If version A gets 6 replies from 200 emails and version B gets 9, the gap is well within random noise. Detecting a lift from a 3% to a 4% reply rate at 95% confidence and 80% power takes about 5,300 sends per version, so cold email teams test bold changes, such as a new angle or offer, rather than small wording tweaks. For marketing email, HubSpot recommends sending an A/B test to at least 1,000 contacts.
A/B Testing Example
Lift = (rate of B - rate of A) / rate of A x 100For example, a team splits 6,000 prospects at random into two groups of 3,000. Version A asks for a 20-minute demo; version B offers a free audit of the prospect's current setup. A gets 96 replies (3.2%) and B gets 150 (5.0%), a 56% lift that is well above noise at that sample size, so B becomes the new control.
A/B Testing in Practice
In outbound, teams A/B test subject lines, first lines, offers, calls to action, sequence length and send times. Opens are inflated by privacy features and bots, so judge tests on replies, positive replies and meetings booked. Run one test at a time per segment, and set the sample size and the winner rule before you start.
FAQ
What is statistical significance in A/B testing?
A check on whether the difference between versions is likely real or could be random chance. A common threshold is 95% confidence: a gap that large would appear less than 5% of the time if the versions truly performed the same.
How many people do you need for an A/B test?
It depends on your baseline rate and the smallest lift you care about. For marketing email, HubSpot recommends at least 1,000 contacts. For cold email reply rates of a few percent, detecting a 1-point lift takes thousands of sends per version.
What can you A/B test in an email?
Subject line, sender name, opening line, body copy, offer, call to action and send time. Mailchimp, for example, lets you test subject line, From name, content or send time with up to 3 variations.
What is the difference between A/B testing and multivariate testing?
An A/B test changes one variable between two versions. A multivariate test changes several variables at once and tests the combinations, which needs much more traffic to reach a clear answer.
Do This in Ken
Do This in Ken
Split the list at random yourself, run each half as its own campaign in Campaigns, and compare replies in Inbox before you scale the winner.
- Module
- Campaigns
- Open in Ken
- /features
The new Ken opens to the waiting list in batches.
Related
- GTM Glossary
Every GTM term, grouped by topic.
- Cold email
- Email sequence
- Email open rate
- Spintax
- Marketing funnel
- Ken features
Sources
| Source | Checked |
|---|---|
| Optimizely: Statistical significance (Optimization Glossary) | |
| HubSpot Knowledge Base: Run A/B tests for marketing emails | |
| Mailchimp: About A/B testing campaigns |
Changelog
- Page published.