OEA QuestionnaireTips

Transcript

1 - Tips for Writing Questionnaire Items Office of Educational Assessment Last Updated: October 2006 handout is provided by the UW Office of Educational Assessment (OEA) as a resource for This WebQ workshop attendees who want some basic tips for writing questionnaire items. It does not address on found site: web our broader aspects of survey and research design, but additional resources can be support . OEA also provides a http://www.washington.edu/oea variety of consulting and research or a with assistance research, assessment, services; please feel free to contact us if you are interested in ( project evaluation 543 ‐ 206 , [email protected] 5190). ‐ TIP 1: BE RELEVANT Before starting your survey, you should specify one several over ‐ arching questions you wish to or address (i.e., research questions or evaluation questions). Your questionnaire items should be focused around these questions. Different Domains of Questions: Behavior, Beliefs, and Evaluation Does providing optional, online practice quizzes help students learn the material in my course? Of the ten optional online 0 1–2 3– 5 6–7 8– 9 10 practice quizzes, how many Behavior did you complete? Completing the online Strongly Strongly practice quizzes helped me Agree Disagree Neutral Beliefs Disagree Agree learn the material in this course. How satisfied were you with Somewhat Extremely Somewhat Extremely Evaluation Dissatisfied Dissatisfied Satisfied Satisfied the online quizzes? three items above have different foci. The first could help answer your research question if you were The to responses correlate to grades. actual with able item the Assessment © 2006 University of Copyright Washington Office of Educational

2 Different Types of Questions: Open- and Closed-Ended transfer students benefit from going to “Transfer Thursday” sessions in the Gateway center? How do ‐ ended: Which of the following Closed do you see as benefits of attending Transfer Thursday sessions? (check all that apply) Learning about major requirements o o Meeting with a UW adviser o other transfer students Meeting Other (specify): __________________ o Open ‐ ended: What do you see as the benefits of attending Transfer Thursday sessions? Data from an open ‐ ended question can help you create “check all that apply” lists or other closed ‐ format if data; the analyze to takes it primary items. The time drawback of open ‐ ended questions is the ended resources limited, use open ‐ questions sparingly. are TIP 2: BE SPECIFIC Avoid abstract terms and jargon The in-class activities in this class facilitate active Strongly Strongly Neutral Agree Disagree BAD Disagree Agree learning. The in-class activities (e.g., illusory correlation Strongly Strongly GOOD Disagree Neutral Agree demonstration, eye-witness testimony activity) help Disagree Agree keep me engaged in class. In the revision, the terms “in ‐ class activities” and “active learning” have been specified. Provide clarifying details How many times did you visit CLUE? BAD the abbreviations or acronyms Write out further Provide if descriptors first time. How many times did you visit the Center for GOOD appropriate. Learning and Undergraduate Enrichment (CLUE) -- evening tutoring in Mary Gates Hall? Frequency estimates Never Not often Fairly Often All the time How many times do you visit CLUE? BAD During a typical quarter, how many About every About once 2 or more times Never 1-3 times GOOD other week per week per week times do you visit CLUE? vary in will The less definite the response options, the more ʹ noise ʹ or error in the data (i.e., participants interpretation their options). the of Writing OEA: Tips for Questionnaire Items 2 UW

3 TIP 3: AVOID CONFUSION Avoid double-barreled questions and The TAs in this course were helpful Strongly Strongly Neutral Agree Disagree BAD Disagree Agree responsive. Rate the performance of the TAs on the following dimensions. GOOD Poor Excellent Helping you understand the material 5 2 1 3 4 Poor Excellent Responding to inquires about your grade 5 3 4 1 2 Avoid double-negative wording The instructor for this course did not respond to Strongly Strongly Neutral Agree Disagree BAD Disagree Agree students' comments during class. The instructor for this course was response to Strongly Strongly Disagree Neutral Agree GOOD Disagree Agree students' comments during class. to have would trait) positive (a responsive was In the first case, students who thought the instructor disagree for puzzles. mental choose strongly out (a such negative answer). Watch TIP 4: USE APPROPRIATE SCALES Provide a spectrum of options Did you enjoy the interactive online activity about Yes No BAD Newton's 3rd Law? To what extent did you enjoy the interactive Very much GOOD Not at all A little Somewhat online activity about Newton's 3rd Law? Typical scales Strongly Strongly Agree Neutral Disagree Agreement Disagree Agree Extremely Somewhat Somewhat Extremely Dissatisfied Satisfied Satisfaction Dissatisfied Satisfied Dissatisfied Satisfied Good Fair Poor Excellent Very good Evaluation Very unfair Unfair Very fair Fair Not too Very Somewhat Not at all important important important important Somewhat Not too Not at all Very Knowledge familiar familiar familiar familiar for Writing UW 3 Questionnaire OEA: Items Tips

4 Unipolar vs. Bipolar scales Unipolar None Extremely Some Very Very Somewhat Very Somewhat Extremely Extremely Bipolar positive negative positive positive negative negative the presence of one attribute Often, not necessitate the absence of another. For example, one can does both arts and sciences, so they should enjoy not be presented with a scale that forces them to choose one or the other. Beware of imposing a structure or model that may not be valid. Providing a (neutral) midpoint Extremely Neither negative nor Very Extremely Very With midpoint negative positive negative positive positive Extremely Extremely Very Very Somewhat Somewhat Without midpoint negative negative positive negative positive positive regarding the appropriateness or usefulness of a midpoint, particularly for agreement Opinions vary Research scales. a forced response will yield roughly the same proportion of agree/disagree as a suggests scale with a midpoint. For example: Results a midpoint: 30% con, 50% neutral, 20% pro with Results without a midpoint: con, 40% pro 60% Number of response anchors general, Should one label all of the response options? In the more specific you are about the response options those of interpretations idiosyncratic the by options, the less about you will have to worry you respondents. some instances (e.g., frequency estimates) it is imperative that label all options. In In cases, you could other crease error by trying to ʹ force ʹ a label. Whatever system you choose, make sure in ʺ labeling is symmetrical (e.g., do not label only ʺ 1 ʺ and ʺ 2 ʺ and ʺ 5 your on a 5 ‐ point scale). that N/A: Use sparingly! I would have completed the online practice Strongly Strongly NO Agree N/A Disagree quizzes, even if they did not count towards my Disagree Agree grade. The optional online practice quizzes improved Strongly Strongly Agree Disagree N/A OK Disagree Agree my grade on the midterm. Another alternative would be to direct respondents to skip a set of items if they had not completed any of the online practice questions, making optional the N/A option unnecessary. ʺ Not applicable ʺ and ʺ don ʹ t know ʺ should be avoided for opinion and attitude items. "Check all that apply" lists lists as exhaustive as possible. Use previously Make collected open ‐ ended data if available. Unless all in ‐ write a for space a response. ʺ other ʺ options are known, it is almost always appropriate to provide Writing OEA: Tips for Questionnaire Items 4 UW

5 TIP 5: AVOID BIAS Avoid leading questions The UW is considering having wireless capacity fo r the entire campus within 3 years. Given the BAD potential impact on students' behavior in the classroom, what do you think of this proposal? How do you think all-campus wireless capacity will affect students' behavior in the classroom? IFFY if at al l, do you think all-campus wireless capacity wi ll affect students' behavior in the classroom? How, GOOD Do you think all-campus wireless capacity will affect students' behavior in the classroom? BEST Definitely not Possibly Definitely yes If yes or possibly, in what ways do you think it will affect students' behavior? the first example, there is an implicit assumption that expanded capacity will affect students’ In behavior in class. The second wording is better, but it still not make clear that respondents are free to deny does any effect; the third wording provides that option. In the last example, the addition of a closed ‐ format introductory allows for easy tabulation of question from overall open ‐ ended question. opinions the Beware hidden assumptions and contingencies During the past quarter, how often have you skipped evening social events to study? BAD In the course of a typical seven-day week, how many hours do you spend doing course-related work GOOD outside of class? In the course of a typical seven-day week, how many hours do you spend socializing with friends for non-academic purposes? The overall question or objective is to find out how students in different majors balance their social lives the and study time. The first example assumes that respondent would normally take part in social the in place take only that evening, events. Furthermore, events there is an implication that all social evening events ʹ count ʹ , or that respondents typically social only study during the evening. Finally, the respondent would be obliged to think back and average over an entire quarter. REFERENCES/RESOURCES N. M., Sudburn, S. & Wansink, B. Bradburn, (2004). Asking questions: The definitive guide to questionnaire design. San Francisco, CA: Jossey ‐ Bass. to Czaja, & Blair, J. (2005). Designing surveys: A guide R. decisions and procedures. Thousand Oaks, CA: Sage Publications. Fowler, F. J. (2002). Survey research methods, 3rd Ed. Thousand Oaks, CA: Sage Publications. Fowler, F. J. (1995). Improving survey questions: Design and evaluation. Thousand Oaks, CA: Sage Publications. effective Publications Sage CA: Oaks, Thousand questionnaires. Constructing Peterson, R. (2000). Writing OEA: Tips for Questionnaire Items 5 UW

Related documents