Questionnaire Design Boot Camp: An Introduction to Survey Writing | Jessica Broome | Skillshare

Questionnaire Design Boot Camp: An Introduction to Survey Writing

Jessica Broome, Opinion researcher and research evangelist

Play Speed
  • 0.5x
  • 1x (Normal)
  • 1.25x
  • 1.5x
  • 2x
6 Lessons (30m)
    • 1. Trailer

      1:18
    • 2. Finding and Operationalizing your Objective

      5:46
    • 3. Applying the Response Process Model

      6:31
    • 4. Some survey dos and don'ts

      5:13
    • 5. Key Points About Answers

      6:22
    • 6. Questionnaire from Soup to Nuts

      5:10

About This Class

This class is for anyone who wants to know what other people think or do! Maybe you want to understand what design features your customers want from your product, why people are buying from your competition, or what your employees are saying about you.

Basic survey research is a common tool, but most people never learn the key principles. Survey design is an art and a science-- better surveys get better data, which help you make smarter decisions. This class will prepare students to write effective questionnaires.

Students should be prepared with an idea for research they want to conduct using a survey. The class will begin with understanding your research aims and writing questions to meet these aims. Specific modules offered include: asking about sensitive topics (drinking! drugs! mean thoughts!); picking the right scale (numbers, letters, labels?); and avoiding pitfalls (more common than most people realize). Lectures will include basic theory but will focus primarily on "real world" examples.

Transcripts

4. Finding and Operationalizing your Objective: I'm Jessica Broom, and this is questionnaire design. Boot Camp Unit one is on finding an operational izing your objective. First, I want to talk for a second about different uses of survey research. Yes, every research can be used to get publicity. For example, I've done surveys about things women worry about and asked people how much they worried about their clothes, their skin, their hair. Then we end up with a headline. 99% of women worry about their hair, and a company can use that to Segway into talking about how their hair product will keep you from having this worry. Surveys could be used to make decisions. This could be anything from How can we, as a company, spend our money to? What should we do to keep our employees happy? To Where should we open our next branch? We can ask people questions and make a decision based on their answers. Service could be used to design content where content is anything from an advertisement. So what words to people like, What colors do they like to a product where we try to understand what features their most important? What are people willing to pay to a website. How useful is it? How easy is it to navigate? Do people have a need for it? Or service could be used to make changes. A lot of places do customer satisfaction surveys, and I like to think they're making some changes based on what they hear. If everyone says the line is too long, maybe they add more cashiers. If a lot of people are saying the chicken is too salty, maybe they switch up the recipe. These are some uses of survey research. The first step in any survey is to answer three questions of your own. What do I want to know? Who do I want to talk to and what am I going to do with the results? So quick example. Right now I'm working on a survey for a hearing aid manufacturer. They want to know how their target audience reacts to a few different advertisements, which won people prefer and why. So that's my research question. I want to make sure I'm talking to the right people, their customers, people who use hearing aids, And what am I gonna do with the results? I asked this so I don't end up with answers to questions that aren't useful. My client is gonna decide which add to run based on the ad that's most preferred. It's after I know what I'm doing, who I'm talking Teoh and what I'm going to do with the information I get. I'll start coming up with some question ideas, brainstorming some questions. I like to write these on a white board or a big piece of paper, and I like to keep my research objective nearby so I can refer back to make sure all my questions have a purpose. I also like to make sure some of the questions I brainstorm are the ones that are going to get me the right people. So here I have. Do you regularly wear a hearing aid I have to have. For how long have you weren't a hearing aid, which add do you like best? And why does it catch your attention? Would you take some action and tell someone about it? Uh, I'll go back through these a few times and make sure all my questions are getting me what I need. So for how long have you weren't hearing? Aid? Probably isn't that useful across that off the rest of my questions and answers, I'll keep. For now, I like to go through some questions and right out potential, uh, findings. So 100% of respondents were hearing AIDS. That's what my client wants. Um, 75% of respondents like ad number one the best. Remember, this was my clients objective, which add to potential customers prefer and why respondents report liking the ad because it's colorful, interesting and relevant to them. I do this to make sure I'm asking the right questions. So your first step before you write a single question is to answer these three questions yourself. What do I want to know? You could want to know multiple things, but before you start writing questions, you need to be super clear about what you're after in this survey. Who do I want to talk to? You mean to include some questions that are going to get you to that? So you really need to know who these people are. If there are people who use a certain product, or do they have to use it a certain amount. Are the people who attended an event Who were you trying to talk to and what are you going to do with the results are using them for publicity are using them to make a change. You need to ask questions that you'll be able to use for the results that you want. So just knowing that people didn't like your event doesn't help you make a decision about what to change. So your first step is to answer these three questions. Your next step is to sketch out some questions and answer choices for your servant. These might or might not be the questions that you eventually use in your survey. This is just your first step. I'd suggest between five and 25 questions. Um, that's not a hard and fast rule. But once your questionnaire starts getting really long, you need to ask yourself if all these questions are necessary for your objective. Or maybe, do you have too many objectives for one survey? Ah, you're gonna keep your objective nearby to make sure that all your questions help you get to it, and you're going to make sure you include in your questions of question or two that will help you get help. You sorry, make you make sure that you get to the people you want to talk to, so do you Wear a hearing? Aid will help you get to people who were here intense. And you're gonna write some sample results. You need to know what you're going to do with the answers. And if the potential answers you could get are going to help you to get to this end, that's all we have for Unit one. 8. Applying the Response Process Model: I'm Jessica Broom. And this is questionnaire design Boot camp. Unit two is on applying the response process model. I try to keep this class very non academic and applied, but this is a theory that you can really use Teoh improve. Almost any question that you write to the response process. Model has five steps and code comprehend recall map and report encoding is the question. Did I noticed this happening? If respondents aren't aware that something happened, it'll be hard for them to answer. Covering hand. Obviously. What is this question asking me if I don't understand your question of the words in it? I have a hard time giving a good response. Recall. Can I remember this? And mapping? Is there a place to put my answer? Just my answer. Fit in an answer category. Finally reporting. Do I want to admit this? So here are some examples of the response process model places it might break down. How often do you eat foods containing vitamin A? This is something a lot of people just aren't aware. They haven't encoded this information, so they won't be able to tell you comprehension. Were you see, not an inpatient or an outpatient basis, These words or not commonplace for a lot of people, it might be easier just to ask. Did you stay overnight in the hospital? Recall. How many apples have you eaten in the last year? This is tough of its ah, mundane sort Every day activity. People aren't necessarily going to remember for a longer reference period like the last year mapping. How much cocaine do you use on a daily basis? This was, ah, question from a survey I worked on, and all the answer categories were in kilograms. But all the respondents would hear this question and answer in dollars. So $50 worth $100 worth. Um, and they There was no way to map their responses onto the responses that were given in the survey and finally reporting. So what is your annual income? This is something a lot of people don't like to admit, especially to maybe a stranger who's asking them Survey questions, uh, one place. We run into a lot of problems in the response process. Model is comprehension on one area, and specific is unclear terms. So words that can mean different things to different people. You really want to make sure all your respondents are answering the same question. So if I ask how many people are in your family, this means different things to different people. It could mean my family of origin. It could mean my family now, my extended family. I don't know what this is asking me. Same thing with how safe do you feel in the area where you live? What is this area? Is it? My house is at my block. Is that my state? Is the country une easy solution here? If you have any terms, that you think might not be clear is to give a definition. So by area we mean X y z and it can mean whatever you want. You just need to make sure all your respondents get the same definition and her answering the same question. Another stumbling block is often reporting. We sometimes ask questions about things that are sensitive or socially desirable. They might not be sensitive topics to you, but they are sometimes to a lot of people. People don't always like to talk about how many sexual partners they've had their income, whether they vote, how much they drink and smoke how much they exercise. So if you're asking about any of these topics or any topic that could be sensitive by your respondents, there are a few ways to sort of minimized the effect of social desirability. You can assume the behavior so you can ask people, How many drinks have you had in the past week, assuming that they've had at least one? But of course, giving them the option to say zero rather than asking, Have you had any alcohol to drink in the past week? Um, question where people might I feel pressured to say no. You can soften your question with a leading. So a lot of people have told us that they wanted to vote, but they couldn't maybe because of this, that and the other. And then ask your question. How about you? Were you able to vote in the last election? Or you can ask about another person first? So do you know anyone who's shoplifted? Yes or no? And then how about you? Have you ever shoplifted? It's a lot easier to stomach after I just told you that all my friends shoplift, so your next step for this unit is to apply the response process model to your questions. Uh, encoding. You want to make sure you're asking about something your respondents will know, Or if not, can you make it about something they'll know? Comprehension is your question. As simple as possible are their words that could have multiple meanings. Do you ask about one thing? Onley in each question to make it as simple as possible for your respondents Recall. If you're asking respondents to remember a quantity, what's more useful and what's easier for for them a rate or account? So how many apples do you eat in a typical week? Or how many apples did you eat last week? Uh, mapping? Will everyone be able to put their answer into one of your answer categories? You need to make sure, um, everyone has somewhere Teoh somewhere to go somewhere to put their answer and reporting. Is this something people will be resistant to answering for any reason? How can you soften this question your step. Another step here is to pre tests problematic questions, so you wanna find someone who looks as much like your respondents as possible. So if you're doing a survey with elderly people. I don't find someone who's 25 years old and have them answer any questions you have doubts about. Just go through the survey and respond to the questions as if they were responding. Then they'll answer some additional questions for you. Asked them things like, What does this word mean to you? What was hard to understand in this question? How did you remember this? Have them walk you through their thought process for coming up with an answer, especially about a quantity that can be really shut a lot of light on where people are having problems. Um, this exercise might show you where people are having problems. It might show that people aren't having problems at all. Um, but it will give you. I'll take you one step closer to having final questions that are easily understood and responded to by the people you ultimately want. Talk to. That's all for Unit two. Applying the response process model 10. Some survey dos and don'ts: - I'm Jessica Broom, - and this is questionnaire design. - Boot Camp. - This is Unit three, - some things to do and some things to avoid in your questions. - I wanted to start this one off with some survey results. - This was a survey about, - at what point do you think a survey is too long? - And these researchers found that 8% of people fought anything more than a few minutes was - too long and 26% thought more than 10 minutes was too long. - So if you're doing a 20 minute or 30 minute questionnaire, - you're really alienating a lot of your respondents. - My my first advice is to be considered of your respondents by keeping your surveys short. - Another way to be considerate of your respondents is to think about who they are. - So if you're a college educated person, - it's easy to forget that in the United States only 22% of people have a bachelor's degree. - The average reading level in this country is somewhere around eighth or ninth grade, - and one in five people reads at or below 1/5 grade level. - So if your questions include a lot of complex concepts and uncommon words, - people might not be understanding what you're saying and you might run into problems. - Do be considered of your respondents by not giving them a grid that looks like this, - Or you will end up with data that look like this. - Um, - if you have to have if you absolutely have to ask all of these questions, - I would suggest cutting the grid at least into two pieces so people aren't suddenly staring - down the barrel of 17 questions they have to answer. - Otherwise, - they're not gonna put a lot of thought into their answers. - You want to consider the context of your questions. - So here on the left hand side, - we ask people how often they exercise, - how often they eat fast food, - how often they had they rate their stress level and then question 20. - After you've been thinking about all these things I've been thinking about how little I - exercise and how many Big Macs I eat. - Then I have to rate my health. - Um, - on the other side, - we asked, - how would you rate your health as the first question? - So neither one of these is right or wrong It when emphasizes air, - both just different and you really just need to consider what you want. - Do you want people to think about all of these things before they read their health? - Or do you want to actually get a clean read on how they rate their health before you've put - all these other things in their mind again? - Neither one is right. - Neither one is wrong. - Just something to think about. - Something you want to avoid in in survey design is double barrelled questions. - Most people know this, - but people aren't always aware of where double barrelled questions can hide. - So this is a pretty straightforward when, - how likely is it that you'll go out for dinner and a movie this weekend? - Um, - if I'm going afer from going out for dinner and a show or ice cream in a movie, - I don't know how to answer this question. - To make sure every question on Lee asks about one thing back to my hearing aid client, - for which, - if any, - of the following reasons, - have you been hesitant to use a hearing aid? - They're too expensive. - I don't believe I need one. - I'm concerned about the stigma of looking older. - This is a sort of a hidden double barrel. - Um, - someone might be concerned about looking older, - but not about this stigma. - Or they might be concerned about the stigma of looking death rather than the stigma of - looking older. - So, - really, - make sure you separate out different concepts into different answer choices. - Finally, - do you favor or oppose President Obama's health care plan? - This can be tricky if we're sort of lumping a person together with a concept. - People are often hesitant to oppose the concept if they generally favor the person. - So I for something like this, - I would suggest taking President Obama out of it. - If it's the health care plan that you're interested in your project. - Step for this unit is to check your questionnaire for any of these problems. - Um, - we started out with length. - Your first problem could be that your questionnaires too long. - I would do a time test, - have someone who's not familiar with your questionnaire, - respond to it to go through it the way a respondent would and time how long it takes them - much longer than 10 minutes. - See where you can cut, - do some really close at its go through your questionnaire with a fine tooth comb and ask - yourself questions like, - Is this question absolutely necessary? - What can I cut Do I use words that my respondents will understand? - Where can I simplify? - This, - of course, - is depends on who you're respondents are, - Um, - but put yourself in their shoes and try to make it as simple as possible. - Do I have any very long sets of questions like that? - Terrible grid I showed? - How can I cut them up if you can't reduce at least cut ah long grid into different pieces? - Does the order make sense? - Are there early questions that might influence responses to later questions? - If you're okay with the early questions influencing the later questions, - that's fine. - Just be aware of it, - um, - and make a conscious decision on how you want to order your questions. - Do any of my questions ask about two things at once? - We don't want any double barrelled questions or even double barreled answers. - That's it for Unit three. - Some things to do and some things to avoid in your questions 14. Key Points About Answers: - I'm Jessica Broom, - and this is questionnaire design. - Boot camp. - This is Unit Four key points about answers. - The first thing I want to talk about is using open ended questions. - Uh, - these are questions that just ask. - Ask something and your respondents have to write their answer or speak their answer. - There are no choices given for them. - Uh, - these are questions like what features would you like to see on this product? - Or what feedback do you have about this event? - Questions like that. - Questions like this have both pluses and minuses. - On the positive side that can give you ideas. - If you don't have any, - respondent might come up with something really good. - They have respondents a chance to express themselves freely, - and some people really like that on the negative side. - Unopened and a question makes respondents do a lot of work. - They have to go through an extra step of thinking about an answer and writing out there - answer rather than just selecting one. - They don't always give you useful information. - If you ask someone what feedback they have about an event, - they might just say it was good or it was bad, - doesn't really give you anything to work with. - And the results are hard to analyze. - Depending on how many respondents you have, - you could end up with a really long list or really big stack of papers. - Um, - with an open ended question that you have toe go through a lot of times for answer choices - , - we use different kinds of scales. - Um, - some that I've seen lately are a 1 to 5 scale with labelled endpoints. - Not at all. - And extremely, - um, - it's great to label your endpoints something you want to think about in scales that have a - midpoint. - So in this case, - the three is what is the midpoint mean? - Um, - does it mean sort of ROK or I don't have an opinion? - Um, - so the label your endpoints and also label your midpoint can be helpful. - For your respondents a 1 to 7 scale, - um, - same issue with the midpoint. - Also, - an issue with the 1 to 7 scale is the way that it can be interpreted. - If you're asking something like, - how often do you watch TV? - Um, - people might, - even though you don't say it. - And the question they might think one means once a week and seven means every day. - So be careful not to accidentally convey any extra information in your scale. - Ah, - not at all. - Not very somewhat. - And very This is ah, - scale. - I use a lot and gets you away from the midpoint issue a lot of little none. - Ah, - this could be useful if you're planning to just collapse your, - uh, - answer choices anyway. - So if you're gonna put one and two into a category and four and five into a category when - you're doing your analysis, - sometimes it's easier to just give respondents three choices instead of five or seven. - The two point scale can be dangerous. - Um, - favor. - Oppose, - Agree, - Disagree. - Yes. - No. - Sometimes people feel like they're backed into a corner. - So I generally caution people not to use two point scales and often, - sometimes rarely, - never has its own set of problems. - Um, - you know what's sometimes for me might be often for you or rarely. - For someone else. - I these air hard Teoh interpret and I generally more of a fan of asking people about actual - units of time. - So once a week, - once a month, - etcetera beyond scales, - sometimes we give people response categories your categories should be mutually exclusive - and comprehensive, - so everyone should fit one and only one category. - Ah, - an example. - This is actually pretty common. - In which of the following categories is your annual hustled income less than $30,000.30 to - 40,040 to 50 $50,000 or more? - If I'm on one of these border points 30,000 or 40,000 or 50,000 I could conceivably put - myself into two different categories. - Um, - this is just messy and confusing for respondents. - Same thing with this one. - What's the age of your youngest child? - 1234 to 67 or older? - Have a six month old, - but I don't know where to put him. - So keep in mind everyone should fit in one category. - Everyone should not fit in more than one category. - Keep your response categories clean. - So here's an example. - How important do you think it is to provide universal access to the Internet? - Not at all important, - because information corrupt somewhat important, - because some people need to communicate with friends and family. - Very important for economic development. - Extremely important, - everyone should have home Internet access. - I mentioned that I really like this? - Not at all. - Somewhat, - very, - extremely scale. - Um, - but here it's sort of junked up by combining the rating scale with the reasons why this can - actually be two separate questions. - So keep your response categories or your scales clean. - Make sure your answers match your question. - Ah, - when I see this question, - how likely are you to enroll in this program? - I think the answer categories they're going to be not at all likely, - somewhat likely etcetera. - So I'm a little taken aback when I see Definitely Will. - Probably will probably will not, - Definitely will not. - People will answer this there. - There's not a problem in answering it. - Um, - it's just gonna take them an extra minute. - Um, - and a little extra brain work that you want to not put on your respondents. - This is actually a pretty common mistake. - If you're editing questions a lot, - you might change your question and not your answers. - Seeming to be really aware of this, - your project step for this unit is to take a really close look at all your answer choices - in all your scales. - Ask yourself questions like am I using any open ended questions? - Do I really need them. - How can I limit them? - How can I? - How am I gonna handle the responses to open ended questions? - How am I gonna analyze them to my scales? - Convey any information I don't want conveyed. - Do I want a midpoint on my scale? - What is the midpoint mean? - Well, - every respondent fit in one and only one answer category. - Do I have any double barreled answers or any places where maybe I'm combining reasons with - ratings and these can be split apart and finally do my answers match my questions? - That's it for Unit Four key points about answers. 16. Questionnaire from Soup to Nuts: I'm Jessica Broom, and this is questionnaire design Boot camp. This is, you know, five questionnaires from soup to nuts. This unit talks about things besides the questions and answers that can help you have the best and most effective questionnaire. Your first step is gonna be inviting your respondents. So, Harrow, are you gonna communicate to respondents that you want their participation? If they're getting the survey by email and suggest sending a warning out a week or a few days ahead of time to let them know they will be receiving the survey and giving them a heads up? If it's an in person survey, maybe you're at an event. Um, you can accost people walking by and try to get them to respond. You could have signs up. Um, telling people this is an opportunity to share your experience or to take a survey you need to communicate to respondents that you value their participation. Some information people often like to know before they take a survey. Is there an incentive? If you can offer people even a dollar or a coupon, um, they feel a little more valued, and they might be more likely to respond tell them what you're using the information for. Um, they're giving you all these answers. They would like to know probably where their answers were going or what they're gonna be used for. Is the survey confidential? Who is going to know what they said? My recommendation is to, if at all possible, keep people anonymous. Don't even ask for their names, um, on the survey and and let them know that their names will never be attached to their responses. If that's true, how long will it take? We've talked about in other units, the shorter the better. People really like to know how much of their time they're giving they're giving to you for this, for this survey during the survey can communicate to respondents that you value their participation by making the survey process as easy for them as possible. So another example from my hearing aid client. Which of the following were most important to you when you purchased a hearing? A prize price, functionality, size and comfort? When I see most important, I think you're just asking me for one thing. But this question seems like it's asking for multiple because it says which of the following were most important. Um, but I really don't know if I'm supposed to select one or if I can select multiple. This is a place where you really need to add instructions so people know exactly what you expect them to do. Another example of making it easy for respondents is this set of questions. Do you use a hearing aid? Yes or no? And then question to for how long have you used to hearing it? So if I just answered Question one? No, I don't use a hearing aid. What am I supposed to do with questions? Who am I supposed to skip it? Am I supposed to write something? Ah, uneasily around. This is to ask question one and then give instructions and you see that I've written these in capital letters and read So they really stand out to respondents. They know that they're only supposed to answer Question two if they answer yes to question one. These things might seem obvious, but it's always better to air on the side of telling giving, making it easier for people rather than harder. You really need to say thank you to your respondents as much as possible. You can thank them for taking the time to participate. Thank them for sharing their experiences. Thank them for helping you out by giving you by giving you their thoughts. Um, whatever you say, it should be genuine. And people should know why you're so grateful to them. Because he really would not have a survey if you didn't have respondents, Your project step for this unit is to figure out your implementation plan. So where will you find people? And how will you let them know that you have a survey you'd like them to take? How will they take your surveys it online? Is it paper and pencil? How are you going to administer it? What mood and how you analyze the results. If you do an online survey with a su surveymonkey or zoo meringue, take program. You can get graphs from that. If that's enough for you. If you do a paper and pencil survey, you'll end up with a big pile of paper. You need to think about what you're going to do with it to make. To make it into usable data. You should also finalize your questionnaire. Add some upfront language telling people how long this every is gonna take, what it's gonna be used for and, of course, thanking them. Do A final read through of your survey really should have not have any typos in this. Ah, if respondents think you're not taking it seriously, they won't take it very seriously. Make sure your questions do match your mode. If it's ah, face to face in our survey interview or on the phone, you should make sure questions sound conversational, which is not as important as if it's on online or paper and pencil survey and finally adds amending language again thanking people for their time and participation. That's it for Unit Five questionnaires from soup to nuts.