By the end of this lesson you can
- Design experiments with clear hypotheses and metrics.
- Analyse funnels and cohorts.
- Avoid common statistical pitfalls.
The idea
Experimentation lets teams learn what actually works by comparing versions with real users. Analytics shows what people do at scale.
Experiment design
- Hypothesis: "If we show delivery dates on product pages, checkout completion will increase, because people will not be surprised at checkout."
- Primary metric: one main measure of success.
- Guardrail metrics: things that must not get worse, such as returns or complaints.
- Sample size and duration: calculated in advance; usually run full weeks.
Funnels and cohorts
A funnel shows how many people complete each step in a journey, revealing where they drop off. Cohorts group users by when they started or what they did, showing how behaviour changes over time (for example retention).
Statistical pitfalls
- Peeking: stopping a test as soon as results look good inflates false positives.
- Too many metrics: test enough metrics and something will look significant by chance.
- Novelty effects: new things get attention that fades.
- Small samples: low traffic makes results unreliable.
Work with analysts or data scientists, and combine experiments with qualitative research to understand why.
Why it matters in real projects
Experiments replace opinion with evidence, but badly run tests produce false confidence. Understanding the basics helps designers propose good tests, interpret results correctly and avoid shipping changes that only appeared to work. It also builds trust with product and data teams.
A realistic example
A team tests a new sign-up page and checks results daily. On day three the new version is "winning" by 12%, so they stop. A month later sign-ups are unchanged. Their next test sets the sample size in advance, runs for two full weeks, uses one primary metric and a guardrail for account activation, and checks results only at the end.
Step by step
Write a hypothesis
Include change, expected effect and reason.
Choose metrics
One primary metric and guardrails.
Plan sample and duration
Calculate in advance with an analyst.
Run without peeking
Wait for the planned end.
Analyse
Check results, segments and guardrails.
Explain
Use qualitative research to understand why.
Common mistakes
Mistake Stopping early when results look good.
Instead Set duration in advance.
Mistake Too many metrics.
Instead Pick one primary metric.
Mistake Testing with tiny traffic.
Instead Check sample size or use other methods.
Check your understanding
Choose an answer, then check it. There is an explanation either way.
Practice exercise
Exercise · about 45 min
Experiment plan
Plan an experiment for a design change.
Steps
- Write a hypothesis.
- Choose a primary metric and two guardrails.
- Estimate sample size with an online calculator.
- Write how you will analyse and explain results.
Deliverable
An experiment plan.
Self-assessment
Tick each item when it is true of your exercise. Be honest; this is for you.
4 items
Summary
- Experiments need hypotheses, metrics and planned samples.
- Funnels and cohorts reveal behaviour.
- Avoid peeking, too many metrics and small samples.
- Combine experiments with qualitative research.
Further reading
Third-party sources, chosen for reliability. They open in a new tab.
- A/B testing 101 (opens in a new tab)Nielsen Norman Group
- Sample size calculator (opens in a new tab)Evan Miller
- Analytics and user experience (opens in a new tab)Nielsen Norman Group
Your notes
Marking complete updates your course and roadmap progress.