Designing a good survey

February 17, 2017 by No Comments | Category Digital Public Services

This is a blog post by Philip Kirk, one of our user researchers.

A survey can be an effective way to get information on who is using your service. They can investigate the characteristics, behaviours and opinions of a group of people and, if designed well, can help you understand your users and what they need.

A poorly designed survey can undermine the results and findings. When beginning work on a new survey, ask yourself these questions:

  1. Is the goal of the survey clear?
    • What is the question I am trying to answer, and is a survey the right research tool to do it?
    • Do I have an overall ‘research question’ or survey goal that each survey question is helping to answer?
    • Do I have a clear picture of who I need to answer this question and a plan to reach them?
  2. Is it clear how I will gather and analyse the results?
    • How will each question feed into the survey goal?
    • How will the responses be categorised and then interpreted?
    • If using numbered scales or open questions, do I have the resources and skills to analyse and make sense of them?
  3. Are the outcomes of the survey actionable?
    • How will I action the insights I get from the survey?
    • Is the research goal and data giving me relevant insights?

If you don’t know how the answer to a question will help make changes to the work you’re doing, there is no need to ask that question.

Survey questions

Designing the questions is the most difficult part of making a survey. Each question should relate to the survey goal and be written from the perspective of the person answering the survey.
Questions should:

  • relate to the survey goal
  • be written from the perspective of the reader
  • be obvious
  • be interesting
  • be appropriate
  • be reliable
  • be valid

Questions should not:

  • lack purpose
  • use abbreviations and acronyms
  • be written in passive voice
  • have a strong point of view
  • be unnecessary
  • cause distraction or confusion

Reliable – written so if the same group answered the questions at different times, they would give the same responses.

Valid – the question is answering what you think it is asking.

Question types

Questions can be open or closed. Open questions allow the person to write their answer to a question in their own words. Closed questions have a list of responses that the person chooses from. Each type of question has positives and negatives:

Open Closed
Positives Increased reflection Easier to answer
Accurate responses Easier to analyse
Unanticipated answers Less time and thought
Shows up bad questions
Negatives More interpretive Adds more bias
Harder to analyse Answers not clear and defined
Require more time to answer Not all answers are included

Survey structure

A good survey should have four sections; introduction, consent, the questions and a thank you. It should be clean, simple and have a logical flow.

  • Introduction – a short explanation of the survey and its goal. It should clearly describe, in a few sentences, what the reader needs to do and how long the survey will take to complete.
  • Consent – this is essential and you need to comply with the Data Protection Act. You should get consent at the start and this needs to be explicit. Consent can’t be inferred from pre-ticked boxes or from starting the survey. Consent has to be a positive indication that personal information can be processed, such as an un-ticked box.

Consent should clearly tell the reader:

    • what they have to do
    • what the data will be used for
    • who will see the data
    • how data will be kept secure
    • how long data will be kept, including the date this starts from
    • how the data will be disposed

A survey isn’t a tool to collect email addresses. You can’t use email addresses from a survey for any purpose other than what was set out in the consent. If you want to follow up from the survey, explicitly state this in the consent with information about what the follow up will be.

  • Questions – these should be set out in a logical order. The number of questions needed depends on your survey goal. Questions should be checked to ensure they are in plain English and ideally at a reading age of 11 years old. There are many tools to check this, such as Hemingway Editor.
  • Thank you – thank the person for filling out the survey. You may want to add your contact details if you want any feedback or questions from the audience directly.

TIP: If using closed questions and are not able to statistically test them then:

  • don’t use the same wording in the questions as an answer
  • have the most negative answer on the left side of the scale

Cross symbol

Do you agree this question is biased?

Agree Neutral Disagree

tick symbol

Is this question biased?

Disagree Neutral Agree

Finally, take the survey out and see how people use it. Is it the way you expect them to? This is the best way to make sure your questions make sense to your audience.

Tags: ,


Leave a comment

By submitting a comment, you understand it may be published on this public website. Please read our privacy policy to see how the Scottish Government handles your information.

Your email address will not be published. Required fields are marked *