Skip to content
Design Academy My progress

Stage 2 · Lesson 5 of 13

Surveys: when they help and when they do not

Good survey questions, sample bias, and why surveys cannot tell you why.

  • Beginner
  • 30 min
  • 3 check questions
  • Exercise: 30 min

By the end of this lesson you can

  • Decide when a survey is the right method and when it is not.
  • Write clear, unbiased survey questions.
  • Recognise sampling bias and interpret survey results with care.

The idea

A survey asks many people the same set of questions. Surveys are good at measuring how many and how much: how many customers use a feature, how satisfied people are, which of five problems is most common. They are poor at explaining why.

When surveys help

  • Measuring how common something is that you already understand from interviews.
  • Tracking satisfaction or a metric over time.
  • Reaching many people quickly, or people in many places.

When they do not

Surveys cannot follow up. If you do not yet know which problems exist, your answer options will reflect your own guesses. Use interviews first to discover; use surveys later to measure.

Writing good questions

  • Ask one thing per question. "How easy and fast was checkout?" mixes two.
  • Use neutral wording. "How much did you enjoy..." assumes enjoyment.
  • Offer balanced scales and an "I don't know" or "Not applicable" option.
  • Prefer behaviour ("How many times did you order last month?") to prediction ("Would you order more if...").
  • Keep it short. Every extra question lowers completion.

Sampling bias

Who answers matters as much as what they say. A pop-up survey on your site reaches existing users, not people who left. Very happy and very unhappy people are more likely to respond. Always report who responded and how many.

Why it matters in real projects

Surveys are easy to send and easy to get wrong. A badly worded question or a biased sample can produce confident numbers that point the team in the wrong direction. Knowing what surveys can and cannot do lets you use them to size problems and track change, while leaving the "why" to methods that can answer it.

A realistic example

After eight interviews, a library app team has found four reasons people stop renewing loans online. They do not know which matters most.

They send a short survey to members: one question about how they renewed their last loan, one ranking the four problems (plus "Something else" with a text box), and two demographic questions. They receive 1,240 responses and note that 70% came from people who visit the library weekly, so occasional users are under-represented.

The ranking shows "I didn't know my loan was due" leads by a wide margin. The team prioritises reminders, while running a few extra interviews with occasional users to check whether their picture differs.

Step by step

  1. Check it is the right method

    Do you already understand the problem well enough to write good answer options? If not, interview first.

  2. Write the goal

    State what you want to measure and what decision it will inform.

  3. Draft few questions

    Aim for under ten. Cut anything that will not change a decision.

  4. Review for bias

    Check each question for double meanings, leading words and missing options.

  5. Pilot it

    Ask three people to complete it while talking aloud. Fix anything they misread.

  6. Report with context

    State the sample size, who responded, how they were reached and what that means for the results.

Common mistakes

  • Mistake Using a survey to find out why people behave a certain way.

    Instead Use interviews or observation to understand reasons, then surveys to measure them.

  • Mistake Double-barrelled questions.

    Instead Split them into separate questions.

  • Mistake Reporting percentages without the sample.

    Instead Always state how many people responded and who they were.

Check your understanding

Choose an answer, then check it. There is an explanation either way.

Question 1 of 3What are surveys best at?
Show answer

Measuring how common something is. Surveys measure scale; they cannot ask follow-up questions.

Question 2 of 3What is wrong with "How quick and easy was it to find a product?"
Show answer

It asks about two things at once. Speed and ease are separate; one answer cannot cover both.

Question 3 of 3A website pop-up survey shows 90% satisfaction. What should you be cautious about?
Show answer

It only reaches people still using the site. People who left are not in the sample, so the result is likely too positive.

Practice exercise

Exercise · about 30 min

Rewrite a bad survey

Find a real survey you have received, or write a deliberately flawed one about a commuting app, and fix it.

Steps

  1. Identify every leading, double-barrelled or unclear question.
  2. Rewrite each with neutral wording and balanced options.
  3. Cut questions that would not change a decision.
  4. Write a note on who the survey would reach and who it would miss.

Deliverable

Before and after versions of the survey and a short note on sampling bias.

Self-assessment

Tick each item when it is true of your exercise. Be honest; this is for you.

4 items

Summary

  • Surveys measure how many and how much; they cannot explain why.
  • Discover with interviews first, then measure with surveys.
  • Write neutral, single-topic questions with balanced options.
  • Sampling bias can make results misleading; always report who answered.

Further reading

Third-party sources, chosen for reliability. They open in a new tab.

Your notes

Marking complete updates your course and roadmap progress.

Back to UX research and problem definition Roadmap

PreviousUser interviews and ethical research practice Next lessonQualitative versus quantitative research