By the end of this lesson you can
- Explain how A/B tests work and when to use them.
- Design a valid A/B test with a clear hypothesis and metric.
- Avoid common mistakes in running and interpreting tests.
The idea
An A/B test randomly shows different versions of a design to different users and compares results. Version A is usually the current design (the control); version B is the change (the variant).
When A/B testing fits
- You have enough traffic to detect meaningful differences.
- The goal is measurable, such as sign-ups or purchases.
- You are choosing between specific options.
It does not tell you why one version wins, and it is poor for testing big new concepts or low-traffic pages. Use qualitative research for those.
Designing a test
- Write a hypothesis: "Showing delivery costs on the product page will increase checkout completion, because people abandon when surprised by costs."
- Choose one primary metric and guardrails.
- Calculate the sample size needed before starting.
- Run the test for the planned time, usually at least one or two full weeks to cover weekly patterns.
- Analyse once the planned sample is reached.
Statistical significance
Significance indicates how unlikely the difference would be if there were no real effect. It does not tell you the difference is large or important. Look at the size of the effect too.
Why it matters in real projects
A/B tests provide strong evidence of cause and effect, which helps settle debates and avoid shipping changes that hurt results. Misused, they produce false confidence: stopping early, testing too many things or ignoring effect size leads to poor decisions. Designers who understand testing can design better variants and interpret results correctly.
A realistic example
An online furniture retailer debates whether to show delivery costs on product pages. The team writes a hypothesis, picks checkout completion as the primary metric and refund requests as a guardrail, and calculates they need about 20,000 visitors per variant.
After two weeks, the variant shows completion up from 3.1% to 3.5% with no change in refunds. The designer pairs this with session recordings showing fewer people leaving at the delivery step, and the change is rolled out to everyone.
Step by step
Check suitability
Confirm you have enough traffic and a measurable goal.
Write a hypothesis
State the change, expected effect and reason.
Choose metrics
One primary metric plus guardrails.
Calculate sample size
Use a sample size calculator.
Run without peeking
Run for the planned duration.
Analyse and learn
Check significance, effect size and qualitative evidence.
Common mistakes
Mistake Stopping as soon as results look good.
Instead Run until the planned sample size is reached.
Mistake Testing many changes at once.
Instead Change one meaningful thing or use proper multivariate methods.
Mistake Ignoring effect size.
Instead Ask whether the difference matters in practice.
Check your understanding
Choose an answer, then check it. There is an explanation either way.
Practice exercise
Exercise · about 45 min
Test design
Design an A/B test for a sign-up page.
Steps
- Write a hypothesis.
- Choose primary and guardrail metrics.
- Estimate sample size with an online calculator.
- Design the variant.
Deliverable
A one-page test plan and variant design.
Self-assessment
Tick each item when it is true of your exercise. Be honest; this is for you.
4 items
Summary
- A/B tests compare versions with random assignment.
- They need traffic and measurable goals.
- Write a hypothesis, choose metrics and calculate sample size.
- Do not stop early; consider effect size.
Further reading
Third-party sources, chosen for reliability. They open in a new tab.
- A/B testing 101 (opens in a new tab)Nielsen Norman Group
- Sample size calculator (opens in a new tab)Evan Miller
Your notes
Marking complete updates your course and roadmap progress.