Survey Design: Understanding Through Thoughtful Questions
By Clay
What is a survey, and how is it different from a questionnaire?
Put simply, a questionnaire is a written list of questions. A survey distributes the questions and interprets the responses. Survey design is the process from start-to-finish, from choosing questions and their layout to organizing and analyzing the results.
Insights derived from surveys
Program Feedback — Powerful ideas come from within your organization if you know how to listen! Use surveys to collect ideas on how to improve and better understand your programs and events from those you serve. Don’t forget to ask your volunteers and employees; who knows your organization better than those who keep it running?
Market Research and Strategic Planning — Use surveys to acquire information that helps you in decision making processes. You can also use them to help gauge interest in potential programs and find out what needs are not currently being met.
Customer/Participant Satisfaction — You’ve seen these before. How satisfied are your customers with their experience? Would they recommend you to a friend?
Survey goals — What do you want to know?
First, consider the intended audience: who will be responding?
The people you serve, or would like to be serving, can directly help you determine how best to fulfill your mission!
Next, list your goals for the survey. Along with participation and other numbers, your organization can measure your impact using survey responses. Think about “metrics” —measuring your organization’s outcomes that are most useful for grants, marketing, budgeting, and much more.
Some grantors will want to see your impact before deciding whether to fund a project, while others will ask you to report how their funding made a difference. (This is why it is helpful when creating programs to consider how you might measure success as the program progresses.)
When will you distribute your survey? You can also conduct a survey more than once! It can be useful to measure changes over time — running the same simple survey regularly enables you to compare results and identify what’s working well and what needs improvement.
Types of questions
There are many types of survey questions, useful for different purposes. Along with yes/no questions, a few common types are:
Rating scales — Respondent rates their satisfaction level or how well the question applies to them, etc.
Multiple choice — Respondent chooses from a selection of predetermined options
Ranking — Respondent may place items in order of importance or their preference, etc.
Any question that has respondents choosing from predetermined options is a closed-ended question. Open-ended questions are those that leave a text box for filling one’s own custom answer. Effective surveys stick to mostly closed-ended questions that are specific and fairly short. Open-ended options are best used when you really want to hear ideas that you may not have considered how to ask for.
Survey designers recommend starting with some “warm up” questions, transitioning into the deeper or more difficult questions, and ending with any easy-to-fill-out demographic data.
I like to include a general open-ended question space at the very end, along the lines of “Is there anything else you wish we knew” or “Do you have any other suggestions for how we can improve?” You might be surprised how much important info can come from these ‘comment card’ style options, as they allow respondents to feel heard in their concerns and ideas.
Survey principles to keep in mind
1. Respondents’ time is valuable, and you only get so much of it
Most people don’t have more than a few minutes, so make your questions count. Don’t add extraneous detail and respect their time. The more questions your survey has, the less time (on average) someone will spend replying to each question. And if your survey is too long, people start to drop out before they reach the end.
SurveyMonkey found that surveys longer than 7-8 minutes resulted in higher “abandon rates” — participants simply won’t finish a survey that doesn’t hold their attention. The abandon rate increases sooner for customer surveys — assume this applies to anyone who isn’t closely involved with details about your program.
Even someone who really wants to respond to your survey will be fatigued by the number or complexity of your questions after a certain point. If your survey is lengthy, explain why you are requesting certain information, or how their answers will help you. Don’t forget to thank your respondents for their time and insights!
Remember that survey participants are doing you a favor; you don’t want to wear out your welcome and cause someone to not reply next time or to fill out incomplete or rushed answers just to get it over with. Make it easy for them to provide you with helpful information. This leads us to another important principle...
2. “Garbage in, Garbage out” — Ask the right questions
A staple phrase of data science, “garbage in, garbage out” is about the importance of quality information (‘in’) to draw effective conclusions (’out’). If you’re collecting information, you want to be sure you’re tracking the right stuff. When you’re assessing how effective a program is for its target audience, pause to reflect on the specific criteria that define success in that context.
You want to really know how your program is doing, how your event went, or how you can best serve your audience. Make sure your questions remain neutral to elicit honest feedback that helps you better understand needs, successes, and areas of growth.
Don’t use overly technical language/jargon, and don’t assume your survey participants will recognize the vocabulary you use internally in your organization.
Ask one thing at a time and keep each question simple and clear.
3. Ask the right people!
If you’re gauging the success of a children’s program, ask kids and their parents or instructors.
If you’re looking to learn about how you can improve your services, consider soliciting responses from a variety of folks who interact with your services in different ways.
If you’re trying to understand the experience of your volunteers, maybe ask them what you could be doing better or what keeps them coming back to serve.
4.Recognize the value of anonymous feedback
People are oftentimes more open if they’re able to share their thoughts anonymously. If you don’t need to follow up with respondents, don’t ask for contact information or names.
Don’t require answers to sensitive questions by leaving an option like “prefer not to answer.”
Keep in mind your responsibility to protect the data you collect; only request info you really need.
Distributing your survey
Think about the survey, from your participants’ point of view
Are questions worded in a way that makes them difficult to answer? Are there a dizzying number of questions that you find yourself starting to speed through? Revise with your reader in mind. You can even try out a potential survey draft on a friend or a small group to collect feedback about the questions.
Consider your delivery method
If using a web survey, create a QR code and share it on-site for your in-person event or program or send your survey link in a follow up email. Many of the survey sites listed below offer built-in QR code options, or you can use this secure QR code generator.
Whether as an alternative or a complement to an online survey, you can also create paper surveys. Paper is sometimes easier for “exit surveys” at the end of an in-person event, for example. Just be sure you're using paper surveys carefully — someone will have to manually organize all those responses (probably you)!
Don't start from scratch!
More than ever before, resources are available to help you create thoughtful surveys in record time!
Websites like SurveyMonkey, Qualtrics, SurveyPlanet, Microsoft Forms, Google Forms, ProProfs, and JotForm all offer free plans that allow you to ask a limited number of questions and send to a select amount of respondents. Many provide pre-built templates, and you can readily find tutorial articles and videos on the web for how to create a survey with any of these sites.
Analyzing responses can also be easier with these online platforms, many of which offer tools to create charts, graphs, and summaries with ease.
Canva also offers handy templates for creating paper surveys.
Don’t reinvent the wheel
Look for inspiration through examples of surveys similar to what you’re considering.
Create reusable templates. If you’re looking for similar feedback across multiple age groups or asking questions you’ve asked previously, you won’t need to start from scratch.
Get help by brainstorming with others about what kinds of information you’re hoping to collect.
Check out the links below and feel free to get in touch with the Ghost Writer team with questions!
Conclusion
Don’t fret about making “the perfect survey” but by approaching survey design with intention, you’ll get a lot more value out of the responses. A small time investment in the planning phase will provide your best shot at collecting the information you need to understand your impact, make informed decisions, and share your outcomes with others.
