1 Su rv eying Clients ABOUT O UTCOMES SERIES ON OUTCOME MANAGEMENT FOR NONPROFIT ORGANIZATIONS The Urban Institute
2 Su rv eying Clients UTCOMES ABOUT O ide is part of a series on outcome management for nonprofit This gu organizations. Other guide topics include keys st eps in outcome analyzing and interpreting management outcome data The Urban developing community-wide llowing up with former clients fo using outcome management indicators Institute
3 Copyright © 2003. The Urban Institute. Permission is granted for reproduction of this document, with attribution to the Urban Institute. The nonpartisan Urban Institute publishes studies, reports, and books on timely topics worthy of public consideration. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders.
4 Cont ents v Preface Acknowledgments vii Introduction ix About This Guidebook x SECTION Survey Basics 1 I What Are Client Surveys? 1 Limitations 3 5 Challenges SECTION Initial Decisions II Step 1 7 : Identify What Information Is Needed Step 2 : Determine Which Clients to Survey, and How Many 8 : Decide Who Should Design, Conduct, and Analyze the Survey 9 Step 3 SECTION Develo ping the Questionnaire III Step 4 13 : Prepare the Questionnaire Step 5 : Conduct a Pretest 18 18 : Decide When and How Often to Administer the Survey Step 6 SECTION Admin istering the Survey IV Step 7 : Choose How to Administer the Survey 21 Step 8 23 : Take Actions to Achieve a High Response Rate : Ensure That Surveys Do Not Harm Clients Step 9 24 26 : Administer the Survey Step 10 SECTION Co llecting and Using the Data V Step 11 : Enter and Tabulate the Data 29 31 : Analyze the Results Step 12
5 iv Surveying Clients about Outcomes Step 13 : Prepare and Disseminate Reports 32 : Encourage Use of the Survey Information Step 14 34 SECTION Cost and Time Requir ements VI 37 Initial Decisions Developing the Questionnaire 38 Administering the Survey 38 Collecting and Using the Data 39 41 References Appendices 45 Sample Client Satisfaction Survey A. Sample Client Survey about Program Outcomes and Benefits B. 47 49 C. A Sampling Primer 51 Issues to Consider before Developing a Questionnaire D. Exhibits 1. Key Steps to Conducting Effective Client Surveys xi 2 2. Some Outcomes Tracked Using Client Surveys 3. 4 Outcome Information from Client Surveys 12 Issues to Consider When Hiring a Survey Contractor 4. 15 Guidelines on Wording Questions 5. 6. 16 Guidelines for Question Response Categories Ways to Administer Surveys 7. 22 Suggestions for Improving Response Rates 25 8. 27 9. Guidelines on Disclosure of Client Responses Quality Control Tasks 28 10. 11. Ways to Analyze Client Survey Data 33
6 Preface This guide provides detailed information about developing and using client sur- ys, one very important method that nonprofit organizations can use to assess serv- ve ice outcomes. Staff members of nonprofit organizations are frequently asked how well they are doing or what difference a particular service makes. In times of tight budgets they are often faced with decisions about which services to cut. Clients will benefit from changes to services that make them more effective. All of these management con- cerns can be well served by information generated from client surveys. When evidence from client surveys is used to improve services by making them more efficient and effective, and used to report to the public about the performance of the organization, the value of this evidence can far outweigh the cost of collect- ing it. Is the evidence provided by client surveys sufficient, as the only source of infor- mation available to decisionmakers? The answer is, definitely not. But when used in conjunction with other information gathered by the agency, defensible decisions are likely to result. The series of guides on outcome management developed by the Urban Institute arms nonprofit managers with a variety of important assessment tools. Multiple methods, and multiple sources of information, provide a strong foundation for mak- ing decisions about services and for providing performance information to the public. The Urban Institute is to be congratulated for taking the practice of outcomes measurement and management to a higher level through the development and publi- cation of these guides. J ames R. Sanders Professor Emeritus of Educational Studies W estern Michigan University
7 Acknowledgments This guide was written by Martin D. Abravanel. The guide benefited greatly from the assistance, comments, and suggestions of Harry P. Hatry, Linda Lampkin, and Timothy Triplett of the Urban Institute. The editors of the series are Harry P. Hatry and Linda Lampkin. We are grateful to the David and Lucile Packard Foundation for their support.
8 Introduction Executives, program managers, and those who fund nonprofit organizations have an inherent interest in learning about service outcomes—how their services affect client condition, situation, or behavior. Collecting and using such results are essen- tial to the practice of outcome management. Good outcome management, of course, helps to improve service delivery, make services more effective, and modify or elim- inate ineffective activities. It also helps to assess organizational achievement and v alue. The first guide in this series , identifies , Key Steps in Outcome Management v arious methods for obtaining outcome information, such as reviewing agency records, using trained observers (such as caseworkers), and administering client surveys. Each has its benefits and costs. Systematic client surveys, for example, can be more expensive than other methods, but are likely to yield some of the most important kinds of outcome information for a nonprofit organization. Although few nonprofit organizations regularly survey their clients, feedback from clients is likely to be a major, if not the major, source of information on service outcomes. This guide is intended to encourage the use of client surveys to measure service 1 outcomes on a routine basis. Extensive and complicated surveys are not necessary. At the same time, some aspects of the process can be technically challenging. That 2 is why this guide contains more detail on procedures than others in the series. Nonprofits do not have to become survey experts, as technical expertise and support are available from a variety of sources. But nonprofits must understand what surveys involve, recognize good survey practice, and make decisions about the role that staff members—as opposed to technical specialists—ought to play. This knowledge helps ensure that useful, high-quality information is collected and provides reasonable confidence that the information collected will stand up to external scrutiny. 1 The guide draws from several sources on survey design and practice, including Don A. Dillman, Mail and Internet Surveys: The Tailored Design Method ; Harry P. Hatry et al., Customer Surveys for Agency Managers ; the American Statistical Association’s “What Is a Survey?” series of brochures; Kathryn E. Newcomer and Timothy Triplett, “Planning and Using Surveys to Collect Useful Prog ram Information”; Boys & Girls Clubs of America, Youth Development Outcome Measurement Tool Kit, 1998 ; and the web site of the American Association for Public Opinion Research, http://www.aapor.org. 2 Another guide in this series, Finding Out What Happens to Former Clients , also discusses client surveys as a means of gathering information on program outcomes.
9 Surveying Clients about Outcomes x About This Guidebook This guidebook includes the following sections: SECTION Survey Basics outlines some features, limitations, and challenges of the survey process for nonprofits. I SECTION Initial Decisions includes steps 1 through 3 of the client survey process, involving actions taken before survey development begins. II SECTION includes steps 4 through 6, covering drafting the sur- Develo ping the Questionnaire y instrument and pretesting. ve III SECTION Admi nistering the Survey includes steps 7 through 10, covering distributing and administering the survey, and handling response rates and client confidentiality. IV SECTION includes steps 11 through 14, involving entering, tab- Co llecting and Using the Data ulating, and analyzing the data. V SECTION ements discusses the efforts required for implementing and Cost and Time Requir maintaining a client survey process. VI Exhibit 1 summarizes the key steps to conducting client surveys.
10 Introduction xi EXHIBIT 1 Key Steps to Conducting Effective Client Surveys SECTION Initial Decisions II Identify what information is needed Step 1. De rmine which clients to survey, and how many te Step 2. Step 3. Decide who should design, conduct, and analyze the survey SECTION Develo ping the Questionnaire III epare the questionnaire Step 4. Pr Conduct a pretest Step 5. when and how often to administer the survey Decide Step 6. SECTION nistering the Survey Admi IV Choose how to administer the survey Step 7. Step 8. s to achieve a high response rate Ta k e a c t i o n Ensure that surveys do not harm clients Step 9. Step 10. Administer the survey SECTION llecting and Using the Data Co V Enter and tabulate the data Step 11. Analyze the results Step 12. epare and disseminate reports Step 13. Pr Encourage use of the survey information Step 14.
11 SECTION I Survey sics Ba What Are Client Surveys? Client surveys can be an indispensable source of outcome information. They pro- vide a systematic means of gathering data on service outcomes from all or a portion of clients. Here are some key features of such surveys: Clients are the recipients, beneficiaries, users, or They focus on clients. patrons of services. They include direct recipients of those services, but may also include others, such as parents of children who receive tutorial, recre- ational, or day care services. Clients can also include those who visit museums, attend performing arts presentations, use parks or recreation facilities restored by environmental clubs, or shop in neighborhoods where community-based organizations engage in reinvestment activities. The survey’s focus on clients reflects the organization’s primary focus on serving people. They seek information about outcomes. Client experiences and satisfaction are important program outcomes that are often best tapped through surveys. F or many organizations, surveys may be the only feasible way to obtain major outcome indicators. Surveys not only provide a way to rate service quality, but also allow clients to report on their condition, behavior, activities, and, most importantly, any changes. Finally, they permit clients to indicate how they think the service provided contributed to bringing about any changes. They are appropriate for many types of services. Surveys can produce use- ful information about outcomes in areas as diverse as animal protection and welfare, arts and culture, child-support services, community improvement, education, environmental protection, health care, housing, job training, men- tal health care, mentoring, recreation, rehabilitation, senior companion serv- ices, shelter, youth development, and many others. Exhibit 2 provides some e xamples of outcomes that can be tracked using client surveys. While nonprofits always have some information about They are systematic. service outcomes from their clients, it may be anecdotal, rather than system- atic and generalizable. Systematic, generalizable data is what sets client sur- ve ys apart from more informal means of obtaining feedback. Properly done,
12 2 Surveying Clients about Outcomes EXHIBIT 2 Some Outcom es Tracked Using Client Surveys Outcome Service type addiction recovery becoming substance-free facility cleanliness animal shelters child abuse prevention child well-being facility convenience child day care enrollment volume community recreation service helpfulness counseling disaster relief responsiveness to victims economic development resources provision employment job placement rates environmental beautification visual improvement food ba nks nutritional improvement group homes staff responsiveness effective representation legal advocacy libraries availability of sought-after material Meals on Wheels on-time delivery patient care dignity of treatment performing arts patron enjoyment rape victims’ support counselor compassion rural development infrastructure improvement shelters condition of accommodations transportation on-time performance vo ca tional training skills attainment youth development behavior improvement
13 Survey Basics 3 surveys are organized efforts to obtain information. Even so, client surveys are subject to error (biases, inaccuracies, or imprecision); identifying and dealing with potential sources of error are discussed in section V. It is not good outcome management practice They are conducted regularly. to conduct surveys just once, or even every now and again. Client surveys should be used to provide a recurring flow of information to the organization and need to be conducted routinely. They encourage client feedback. It is not always easy to get people to respond to surveys. When respondents are selected from lists of telephone numbers or addresses, the connection to the survey sponsor is limited. Nonprofits conducting client surveys, however, have a distinct advantage because they have a relationship with their respondents. Clients should be less likely to dismiss or discard a questionnaire seeking information. Many are likely to have a personal interest in the survey topic and in helping the organ- ization to improve services. Indeed, some may even appreciate the opportu- nity to provide feedback. The major They are done to help the organization improve its services. purpose of a client survey is to identify what works and what doesn’t work for clients so that modifications can be made. Exhibit 3 lists the types of outcome information generally obtained through client surveys. Some sample survey forms can be found in the appendices. Appendix A is an American Red Cross survey that cov ers client satisfaction, and appendix B is a proposed survey by the Mental Health Statistics Improvement Program asking about client outcomes and benefits. Limitations As with all outcome indicators, client surveys should not be the sole means used not to obtain outcome information. They are good when data are more easily, more accurately, or less expensively collected from 3 agency records, trained observers, or other sources, clients need to recollect perceptions of experiences and events that occurred in the distant past, say, more than a year before the survey, questions require complicated or detailed answers or explanations, information is requested on the experiences of others about whom survey respondents are not likely to be well informed, or respondents are asked to provide especially sensitive or private information that is only peripherally relevant to service provision. 3 Such data might include medical diagnoses, school grades, or service use patterns (such as number of hours of service received).
14 4 Surveying Clients about Outcomes EXHIBIT 3 Outcome Information from Client Surveys 1. Satisfaction with quality of service provision Service quality, timeliness, accessibility Fa cility convenience and condition—location, safety, hours of operation Staff quality, accessibility, pleasantness, courtesy Overall satisfaction with the services provided Wh ether the client would recommend the service to others 2. Outcomes/benefits realized by clients at the conclusion of services or after Indications of client behavior or condition, such as whether a tutored student has obtained a degree, GED, or become employed, or a previously addicted client has become or remained drug-free Client perceptions of service benefits and results Client perceptions or indications of degree of improvement in behavior or condition as a result of services 3. Information to help explain client ratings Client opinions and preferences that have a bearing on outcomes, or can help a nonprofit organization review or rethink policy or service-delivery issues Reasons for any poor ratings Suggestions for service improvement Service characteristics, such as the amount or type of services received Demograp hic characteristics not available from organization records
15 Survey Basics 5 Challenges Obtaining feedback from clients presents several challenges: Burden on the organization. Client surveys can be somewhat difficult and costly to undertake, and may require skills unfamiliar to nonprofit staff. Need for reasonably high levels of client participation and good infor- mation collection practices. While many clients have an interest in provid- ing feedback about service results, it is not always easy to obtain a high enough level of response to ensure that survey information is a valid repre- sentation of client experience. Success requires good survey techniques. P ossible unwanted or negative publicity. Some organizations may be con- cerned about possible misuse of client survey findings by their funders, the media, or others. From an outcome-management perspective, the primary purpose of a client survey is to assess whether clients are benefiting and, if not, to identify why and what can be done to improve results. However, oth- ers may not view negative results as a way to improve, but simply an indica- tion of organizational failure. The possible misuse of negative results may discourage some organizations from doing surveys. Setting up a client survey process requires that nonprofits understand what is involved; set aside sufficient staff resources, obtain outside expertise, or provide for special staff training, if necessary; and consider how to make the best of whatever information is generated.
16 SECTION II Initial Decisions Identify What Information Is Needed Step 1: Client surveys help organizations learn whether services are producing antici- pated or desired results and, if not, provide clues for how to improve them. What sur- ve ys should cover and how they should be done will vary from organization to organization, following from each organization’s unique mission, services provided, client base, and intended results. Issues covered by a client survey should correspond to the key service outcomes an organization wishes to track. Because survey length generally affects response rates, issues not pertinent to improving outcomes should probably be limited. The goal is to develop the shortest possible list of questions consistent with the survey’s objective of assessing outcomes. Organizations involved in numerous programs or activities should decide which of them, if not all, will benefit from having client survey information. If multiple services are provided to the same clients, should all services be covered in one sur- ve y, or should each service be surveyed separately? It may be useful to get advice from staff members, board members, or represen- tatives from funding organizations, advocacy groups, and public agencies on what outcomes should be tracked through surveys. Input from clients and former clients should also be sought. Involving more people in the planning process also helps to obtain their “buy-in,” lends credibility to the survey, and increases the likelihood that survey findings will be used. Finally, the content of surveys conducted by others providing similar services may be helpful. Another consideration is whether any “issue-of-the-moment” information will be collected. In addition to collecting information on outcomes, it may be useful to include questions about a limited number of other time-specific matters. A small por- tion of the questionnaire can be reserved for covering special issues that may change each time the survey is administered. For example, the organization may wish to ask about proposed service delivery options or program changes. Adding one-time ques- tions adds some cost and burden as the questionnaire, data entry, and tabulation pro- cedures will need to be modified for each survey.
17 8 Surveying Clients about Outcomes Step 2: Determine Which Clients to Survey, and How Many A number of questions about survey respondents need to be considered. For e xample, not all service recipients will be able to respond to a survey because of their v ery young age, illiteracy, language barriers, or physical or mental communication barriers. Instead, other people, such as family members, may be able to provide the needed information. Co nsidering Whom to Survey Other than direct service recipients, who should be surveyed—parents? Guardians? Caregivers? Should informants be asked to respond on behalf of clients who are not in a position to be surveyed? Should program dropouts be surveyed? Should former clients be surveyed to identify a program’s sustained outcomes? What minimum amount of service should clients receive to be eligible for surveying? Once the relevant client group is defined, the organization needs to be able to enumerate all members of that group so that the information collected is not biased in one way or another. Generally the basis for the enumeration is a list maintained in the normal course of operation. Some organizations keep an unduplicated roster of . In the latter, the same gister of service-use instances clients while others keep a re client’s name may appear multiple times. Which type of list is appropriate for use in conducting a survey depends on how an organization’s outcome indicators are spec- satisfied with their serv- percent of clients ified. For example, if the indicator is the ice experience, then an unduplicated client roster is appropriate. If the indicator is the clients are satisfied with the quality of service received, then it percent of times is appropriate to use a service-use instance list. To obtain valid results, client lists need to be up-to-date, complete, and accurate. Leaving out some portion of the client group presents the possibility of bias. Omitting records that do not contain current addresses, telephone numbers, or other contact information may result in overlooking an important segment of the organiza- tion’s clientele. If all clients cannot be enumerated, such omissions should be noted when reporting findings. It may also be necessary to adjust the analysis of survey results. or some programs, such as those involving performing arts or recreation center F use, a roster or list of participants may not be available. In that case, surveys should be done in such a way that all clients have a chance to be included. Another consideration is the minimum amount of service that a client must receive before being included in the survey. For example, some clients may come in only once, or only for a short time, and it may be decided that they should not be considered clients for outcome measurement purposes.
18 9 Initial Decisions Similarly, a decision must be made about whether to survey clients who drop out before completing the full complement of service planned for them. For dropouts, a different set of information would normally be used that would focus on the reasons for dropping out. Because many nonprofits serve relatively few clients, it is often possible to sur- ve y all of them, rather than a sample. Surveying all clients is often simpler than sam- pling a portion of them. It also eliminates a potential source of inaccuracy in surveys known as sampling error; the “luck of the draw” of a sample may result in a group of clients whose responses do not represent the experiences of the entire group. But surveying all clients is not always feasible, for example, if the client base is very large. Surveying every client on a regular basis could be very burdensome, time consuming, or costly. This is probably the most important reason for not surveying all clients. Examples are users of rec- if the full range of beneficiaries is unknown. reation facilities refurbished by a citizen association, or boaters who use w aterways cleaned by an environmental group. if clients are extremely difficult to locate (for example, those without per- manent addresses who are no longer in contact with the organization). These clients could be omitted if they represent a small segment of the client population and the omission is noted during analysis and in reports. If difficult- to-locate clients represent a core service constituency, then a different way of assessing service outcomes may have to be devised. In these cases, sampling may be appropriate. It may save the organization money, speed up the information collection process, reduce the overall burden on the client population (since only a fraction of them need to provide information), and result in sufficient precision for most outcome-management purposes. Because sampling involves selecting only some clients, a degree of uncertainty is involved in generalizing to all of them. The extent of that uncertainty depends on the type of sampling used and the size of the sample, but surprisingly does not depend on the total size of the client base. If samples are used, expert advice on the sample selection method and sample size should be sought. See appendix C for more information on sampling issues. Step 3: Decide Who Should Design, Conduct, and Analyze the Survey Surveying clients on a regular basis requires a commitment of resources. Organizations can choose whether to work on surveys entirely in-house, primarily in-house but with assistance from outside people, or entirely contracted out. Each option has its benefits and costs.
19 10 Surveying Clients about Outcomes Co nsidering Who Should Do Client Surveys Staff. Are there staff members who possess (or can acquire) the appropriate skills and expertise? Time. Is there sufficient staff time available to do the work involved? Is there adequate technological support, such as computer hardware echnical support. T and software? Schedule. Can staff complete surveys on a timely and regular basis, consistent with the organization’s needs for outcome information? Community support. Are there volunteers with survey expertise available? unds. F Are there funds to pay for outside assistance? F or most nonprofit organizations, client surveys will only be practical if they are done in-house. Because the surveys should be conducted regularly, there are signif- icant advantages to developing an internal capacity to do them. But it is not always necessary to start from scratch. Similar organizations may have questionnaires and e xamples of reports that can be used as a base. Even if such materials are available, an in-house effort will still probably require a considerable amount of staff time, especially at the beginning. Some organizations may be able to plan and manage client surveys entirely by themselves, but most are likely to need technical assistance. Assistance with setting up the initial survey, developing the questionnaire, advising on a sampling strategy (if needed), and analyzing the initial survey’s results is likely a good investment for the organization. Outside support may be desirable for mailing questionnaires or entering the responses into a database. Nonprofits may be able to obtain free or low-cost help from faculty members or student interns at local colleges or universities, high school students needing to fulfill community service v olun teers. requirements, or community Finally, some organizations may choose to use an outside contractor. University- based and commercial survey and market research organizations, as well as smaller, independent consultant-type businesses, organize surveys. This approach may be appealing if survey design issues are complex or if survey implementation is excep- tionally burdensome for one reason or another—and if the organization has funds to pay for such help. Hiring a contractor is also advantageous if client survey activities occur at irregular times, leaving dedicated staff with periods of inactivity. And when external credibility of the surveys is very important, an outside organization may be seen as more independent and less invested in survey outcomes than the organization itself. Exhibit 4 lists some issues to consider when hiring a survey contractor.
20 Initial Decisions 11 Even if surveys are conducted by a contractor, the responsibility to provide direction and oversight of the process still lies with the organization. Management must make the basic decisions—about survey objectives, activities and issues to cover, kinds of clients to survey, and the level of precision. As a general rule, the more in-house involvement, the more likely the results will be valued and useful. Appendix D summarizes issues to consider before developing a questionnaire.
21 12 Surveying Clients about Outcomes EXHIBIT 4 Issues to Consider When Hiring a Survey Contractor Cost . Cost often varies cons iderably from contractor to contractor, so get bids from several. Make sure that survey requirements are clear, and carefully evaluate proposals before selecting a contractor. The firm’s experience. Find out about the contractor’s past experience in conducting similar surveys. Check with previous clients to learn about their experiences with the firm. Contin uity. If surveys are to be conducted on a regular basis, the contractor should either be available to do so over the longer term or be willing to transfer its materials (and guidance in how to use them) to the organization so that subsequent surveys are compatible with those conducted by the contractor. Style and compatibility. Seek a firm whose staff members are easy to work with and have a good understanding of the organization’s need for, and use of, client survey data. Consider whether the contractor’s staff members listen and respond well to the organization’s needs and understand the constraints under which the organization operates, and whether they are willing to allow organization staff to participate in the survey process in order to reduce costs and assume ownership of the results. Confident iality. If data obtained from clients need to be maintained as confidential, it is essen- tial to find a contractor that employs strict procedures to protect the privacy of client- supplied information.
22 SECTION III Develo ping the Questio nnaire Step 4: Prepare the Questionnaire Asking the wrong questions, wording them badly, or designing questionnaires poorly can produce erroneous or useless survey results. It is important to ask questions that clients can and are willing to answer; make questions pertinent to service outcomes; make questions and response alternatives unbiased, clear, and easy to understand; ensure that questions can be consistently interpreted by all clients, and can be answered accurately and truthfully; and review order, format, and design of the questionnaire to ensure that clients are encouraged to provide good quality responses and complete the form. When expertise is not available in-house, obtaining advice and assistance from a survey specialist is recommended. Even when outside expertise is used, nonprofit staff should be acquainted with the basic issues involved in questionnaire develop- ment. At a minimum, questions must be reviewed to make sure the wording is appro- priate for the organization’s clients and outcomes. Questions should be written so they require as little effort as possible on the part of respondents (in answering them) and the organization (in recovering the informa- tion). They can be closed-ended (limited to choosing an answer from those listed on the form) or open-ended (given a blank line to write any response). If most questions are closed-ended, survey administration is easier and less costly, and tabulations can be done efficiently and consistently. Because closed-ended questions have fixed, pre-established answer categories for selection by respondents, the quality of the information elicited depends on how well questions and response categories are crafted. Good questions are generally simple, clear, short, and unbiased. According to the American Statistical Association,
23 Surveying Clients about Outcomes 14 It is the questionnaire designer’s greatest challenge to take important topics and trans- late them into simple concepts, simple behaviors, and simple words. The style of the questionnaire must not get in the way of respondents’ providing their information; oth- erwise the results could be incomplete or misleading data, item refusals, respondent fatigue effects—even the respondent’s refusal to complete the questionnaire. (American Statistical Association 1999, p. 11) While books and manuals provide advice on how to word questions, conferring with a survey specialist is strongly recommended. See exhibit 5 for suggestions on how to word questions. Although most questions will likely be closed-ended, it can be quite useful to include a few open-ended questions, such as “ Why do you feel that way? ” or “ Please It is especially useful to ask for explanations of negative ratings to provide e xplain. ” guidance for corrective actions. It is also good practice to end a client survey with a ” or, “ Is there In what additional ways could we have helped you? question like, “ anything else you would like to tell us about the service you received? ” or, “ What can we do to improve our performance? ” Open-ended questions give clients an opportunity to explain some of their closed-ended answers, and permit them to give feedback in a manner that reflects their own perspectives. In essence, open-ended questions allow clients to express themselves in their own words and feel they have been heard. Equally important, the organization can use such questions to ensure that important information is not missed, help interpret the patterns and findings from responses to closed-ended ques- tions, and learn about concerns not covered in the questionnaire. If answers to open- ended questions show some consistency over time, they can be developed into closed-ended questions for use in subsequent surveys. More guidance on how to structure responses to survey questions is included in e xhibit 6. After writing the questions, the next task is assembling the questionnaire. Important considerations include length, question order, and construction. A survey’s length depends on both the amount of information needed and the e xtent of client tolerance for supplying it. The questionnaire should be long enough to provide adequate coverage of priority outcomes yet short enough to encourage client response. One or two pages are adequate in many instances. In general, the longer a questionnaire, the more incentive or client motivation is needed to obtain an acceptable response rate. Even so, a stimulating and inviting multipage question- naire may encourage more response than an uninteresting and non-involving shorter one. The order that questions appear on a questionnaire is important. It can play a sig- nificant role in establishing and maintaining client interest, and facilitate client abil- ity to navigate the questionnaire and interpret (react to) individual questions.
24 Developing the Questionnaire 15 EXHIBIT 5 Guidelines on Wordi ng Questions e thing at a time. Asking clients if the service facility they visited was “safe and Refer to on clean” does not allow for determining whether a “no” answer refers to safety, cleanliness, or both. Choose terms clients can understand. Asking if clients are substance abuse “recidivists,” for e xample, may be beyond the comprehension of some. . Questions should not require a client to respond “yes” in Eliminate “negative” questions order to mean “no.” For example, the question, “Was the service not provided in a timely manner?” is difficult to understand. The answer is equally difficult to interpret. Asking, “Isn’t the new collection of books in the library Avoid leading or biased questions. much better than the old collection?” invites a “yes” answer, as opposed to a more neutral version—”W ould you say the current collection of books is better or worse than the previous collection?” Another option, although longer, is, “Some library clients believe that the current collection of books is better than the previous collection. Others believe that the previous collection was better than the current collection. What do you think?” Provide a Make clear which time period is being ppropriate and reasonable time references. asked about, but avoid asking clients to recall events that occurred a long time ago. For example, when asking about the promptness of ride-on transportation, it is appropriate to specify a time period, but asking if service was generally on time a year ago is unlikely to produce a reliable answer. Use complete sentences to ask questions. Make questions consistent if comparisons over time or across groups are to be made from survey to survey. Even if prev iously used wording is not ideal, word-for-word consistency is important to ensure that comparisons are valid. Use ranges for numerical responses. Ranges may be especially useful if precise answers are not expected from clients. Avoid unnecessary calculations, especially those that discourage completing the questionnaire. Tr y to phrase questions so clients have accurate, ready-made answers.
25 16 Surveying Clients about Outcomes EXHIBIT 6 Guidelines for Question se Categories Respon Make sure that categories do not overlap. Fo r e xample, the question, “How many times did yo u come to the service facility during the past month—1 to 3 times, 3 to 6 times, or more than 6 times?” does not give a clear choice of category to select if the answer is “3 times.” Cov er all possible responses. Fo r e xample, the question, “How long did you have to wait before y our name was called the last time you came to the center—10 to 20 minutes, 21 to 30 minutes, or more than 30 minutes?” does not allow for the possibility that the client had to wait less than 10 minutes. Or, the question, “The staff were courteous to me—always, sometimes?” does not allow for the possibility that the staff were never courteous. Use broad numerical ranges to make it easy for clients to respond. e often Ranges ar appropriate when asking about income or service frequency, where clients may not know the exact amount. Category sizes should also reflect the distinctions needed for analysis. For example, instead of asking, “What was your total household income last year?” ask, “Which of the following categories best describes your total household income last year—less than $10,000, $10,001 to $20,000, 20,001 to $30,000, $30,001 to $40,000, or $40,001 or more?” Include balanced numbers of favorable and unfavorable response categories on rating scales. Fo r e xample, the response scale “completely agree, agree, neither agree nor disagree, disagree” should also contain the possibility “completely disagree.” Wh ere appropriate and feasible, use the same response categories across questions. It is easier and less confusing to clients if response categories are consistent and do not change from question to question.
26 Developing the Questionnaire 17 Guidelines for Order of Questions Set priorities. Questions that are more important or interesting to the client should precede less important and less interesting ones. Consider the question sequencing logic. Questions should be asked in a logical order—by topic, time sequence, or another appropriate sequence. Cluster questions. Logically related subjects should be clustered together. Also, where appropriate, group together questions that use the same response categories. Consider whether specific questions should precede or follow general questions . More specific questions often precede more general ones. For example, asking about facility convenience and staff accessibility would come before asking about overall satisfaction with the service, thereby suggesting that overall satisfaction may be based on these more specific evaluations. Alternatively, clients could be asked to give overall ratings without any guidance on what to consider in making a judgment. Each organization should calculate which approach would produce the desired result. F P actual, easier questions are generally put at the ut easier-to-answer questions first. beginning of a questionnaire. Sensitive or possibly objectionable questions are generally placed near the end. How a survey will be administered—whether by an interviewer asking the client questions or by the client filling out a questionnaire—influences the construction. Since most organizations are likely to use self-administered questionnaires, attention needs to be paid to format, appearance, layout, instructions, and overall design. The objective is to facilitate understanding, encourage participation, and produce com- plete and accurate responses. Guidelines for Questionnaire Construction Cover letter. Having a separate cover letter (on organization stationery and signed by a high-level organization official, such as the executive director or board chairperson) is r ecommended when questionnaires are sent in the mail. Introduction. A questionnaire should generally begin with a brief introduction stating what the survey is about, whom it is for, how the information will be used, and whatever confidentiality provisions are in place. It is good form to politely ask clients to participate, indicate the importance of getting their feedback, and thank them for taking the time to participate. The introduction should also contain simple, straightforward instructions on completing and returning the survey. D esign. A questionnaire should be visually appealing, simple, and easy to follow. Where to begin and how to proceed should be obvious. If there are multiple columns or pages, ensure that the layout does not confuse respondents. A badly formatted questionnaire can contribute to missed questions or failure to see the full range of response alternatives Conclusion. A questionnaire should conclude with a friendly thanks for participating.
27 Surveying Clients about Outcomes 18 Step 5: Conduct a Pretest Although the process need not be elaborate, a new survey (and survey proce- dures) should be pretested before it is administered for the first time. No matter how well a survey is planned or designed, it is extremely difficult to anticipate all possi- ble problems in an instrument, or with procedures for administering it. Pretesting is the major way to determine if the instructions, questions, and format are under- standable. Pretesting helps determine if questions are interpreted by clients and the survey developers in the same way. Something as simple as how clients define the word How would you rate the convenience of this serv- “convenience” in the question, “ ice? ” can be very important in interpreting survey results. Pretesting also provides the opportunity to try out alternative wordings, question order, questionnaire for- mats, and the like. Even questions borrowed from other surveys need to be pretested with the organization’s own clients. Only a small number of clients—perhaps ten or so—is necessary for pretesting, if they represent a range of client types or service experiences. Formal pretesting can include r espondent debriefings —structured one-on-one interviews immediately following completion of a questionnaire to evaluate interpretations and e xperiences; ocus groups f —in-depth group discussions with client respondents to hear their reactions; and cognitive interviews —asking respondents to paraphrase, think aloud, or answer probing questions while they are filling out a questionnaire, to under- stand how they interpret and process the questions. A less formal option is unstructured discussions with clients who have completed the questionnaire to obtain their comments and input. Based on pretest results, the questionnaire or procedures should be modified where necessary. If they change considerably, a second pretest may be appropriate. Questionnaires that need to be translated into other languages should be pretested in each language. Step 6: Decide When and How Often to Administer the Survey Organizations must decide when during service provision clients will be sur- ve yed, how many times they will be surveyed, and how frequently they will be surveyed. Depending on the nature of the service and program objectives, there are several options regarding when to survey clients. A questionnaire could be administered at
28 Developing the Questionnaire 19 the time of departure, shortly thereafter, or at a fixed interval following service completion. or services that only seek to improve clients’ immediate condition, it is appro- F priate to survey them at the end of service or very soon thereafter. This category applies to clients who, for example, are provided a short-duration stay in an emer- gency shelter or treatment at a clinic. It also applies to those who attend an event (such as a performance) or visit a facility (such as a library or museum). Clients can also be asked about the frequency of attendance and, where applicable, about previ- ous experiences. F or services that seek to help clients sustain an improved condition, it is appro- 4 priate to survey them later, such as six or 12 months after service completion. This category applies to clients who, for example, are provided assistance intended to result in retained employment, remaining drug-free, or long-term relief from a nonabusive environment. If questionnaires are mailed to clients, they should proba- bly not be surveyed more than 12 months after departure from service because of the difficulty and cost of locating them. A special study is normally used to track clients after 12 months, if funds are available. Avoid the temptation to administer question- naires to clients who received and departed from service at substantially different intervals. F or ongoing services, it is appropriate to survey clients periodically while that service is being provided. This category would include services in an institutional- ized setting. Surveys could be conducted, for example, at one-year intervals or at fixed points in time after service is initiated. How many times clients are surveyed should reflect the organization’s outcome- management needs as well as the administrative and cost burdens imposed. In most instances, clients need to be surveyed only once. Surveying once minimizes cost and administrative complexity, since having to link clients’ responses across multiple surveys requires considerable effort. The nature of the service and service objectives generally determine how fre- quently clients are surveyed. Some organizations may require information at multi- ple points, such as when important intermediate outcomes can be measured and then when end outcomes can be measured. Others may need annual feedback from clients who are served for long periods. The more times clients are surveyed after leaving service, the more costly the client survey process will be. When considering survey frequency, certain distinctions need to be made—such as between programs that serve a stream of clients who enter and leave service at dif- ferent times and those that serve clients who start and end service simultaneously (classes of tutored students or attendees at a performance). 4 See Finding Out What Happens to Former Clients , in this series, for information on the issue of tracking clients for follow-up purposes.
29 Surveying Clients about Outcomes 20 If an organization has a steady stream of short-term clients who receive service at irregular times, it may be appropriate to survey them on a rolling or continuing basis. For example, a small portion of an organization’s clients—those who com- pleted service at about the same time—could be surveyed each week. This arrange- ment minimizes the administrative effort at any point in time and spreads the w orkload throughout the year. Survey results can be accumulated, tabulated, and summarized frequently, and the findings made available to program managers on a timely basis. F or organizations providing services to groups who start and conclude service at the same time, it may be appropriate to survey clients in periodic waves, such as ev ery few months or so. Questionnaires would be batched for clients who received (or completed) service within that time period.
30 SECTION IV Administ ering the Survey Choose How to Administer the Survey Step 7: Selecting the most appropriate way to administer surveys is a key decision. Nonprofits most often use either mailed or handed-out questionnaires for self- administration or they survey by telephone, but there are other possibilities. Each has advantages and disadvantages to consider. Exhibit 7 briefly summarizes these options. One consideration is where surveys should be done. Options are at clients’ resi- dences, where they receive services (such as at service centers, clinics, shelters, recreation centers, libraries, auditoriums, museum, parks, and so on), or elsewhere (such as at a school where children who receive counseling services at another site are enrolled). A second consideration involves whether to use interviewers. In-person inter- views give greater control over the questionnaire-administration process and are par- ticularly well suited to asking complex or detailed questions, providing clarification, obtaining information from clients who are less responsive to written documents, or asking more questions than is feasible when interviewers are not involved. elephone interviews have some of the same advantages, although sometimes to a T lesser degree. It is generally costly, however, to use interviewers, with in-person interviews at a client’s home as the most expensive survey mode. Labor and training costs are incurred in all cases, as well as travel and time costs when interviewers go to clients’ homes. In-person interviews conducted at a service facility and telephone interviews are less expensive than in-home interviews. Finally, telephone interviews obviously require clients to have telephones, which in some instances may not be av ailable. In general, interviewer-administered surveys require significant resources. If interviewers are not used, clients must take responsibility for completing the survey. Self-administration gives less control over the process but tends to be less resource-intensive, burdensome, and costly, and is often quite adequate for routine surveys. Questionnaires can be handed to clients in person or mailed, faxed, or e- mailed to them. Or, clients may be directed to a web site. Questionnaires sent through the mail are relatively low cost and may be particularly well suited when
31 22 Surveying Clients about Outcomes EXHIBIT 7 Ways to Admin ister Surveys —In-home interviews are likely to be prohibitively expensive because of Interviews, in-person time and travel costs. Interviews conducted at service sites are less costly, although costs will be higher than for self-administered questionnaires. Interviewers should generally not be service providers, and should be trained in techniques to reduce bias in responses. Interviews, by telephone —Although more expensive than self-administered questionnaires, in some instances telephoning clients offers advantages that are worth the extra expense—for example, if probing is important or if a low response to self-administered mailed questionnaires is expected. This mode is feasible if staff, volunteers, or funds are available; clients have telephones; and accurate, up-to-date telephone contact information is available. inistered, by mail Self-adm —If client addresses are accurate and clients do not object to re ceiving mail from the organization, mail is a practical and efficient way to do routine surveys on a limited budget. Clients do not have to be physically present at a service site so there is flexibility about when in service provision client feedback can be solicited. Inconvenience to the client is minimized, and completing questionnaires in the privacy of one’s residence may encourage honest responses and provide confidentiality. A key disadvantage is the potential for a low response rate. Actions must be taken to achieve as high a response rate as possible. Another disadvantage is having to deal with possible duplicate responses from the same clients if more than one mailing is done. Proced ures m ust ensure that duplicate questionnaires are not counted. Duplication may be difficult to avoid if responses are anonymous. inistered, handed out —Mailing questionnaires may not always be possible or Self-adm appropriate; for example, if clients have no fixed address, if addresses are unknown, if an organization does not keep client records (such as a museum or a recreation center), or if clients do not want mail sent to their residences. In such cases, clients can be given questionnaires at service sites to com plete at the service site. This opti on is low cost, allows for quick feedback, and mini- mizes recall problems if clients are queried about services just received. It is also advan- tageous if clients are unlikely to respond to mail questionnaires. Clients may complete a questionnaire alone, be assisted by staff members (if there are literacy or other impedi- ments to self-completion), or enter answers at a computer terminal. This option may be particularly attractive when there is no contact information on clients but they are will- ing to come to a facility to complete a questionnaire—especially if incentives are offered. com plete elsewhere and return. Th is option requires that clients are handed question- naires in an organized fashion—making sure that each client who is supposed to get one does, and that multiple questionnaires are not given to the same client. Questionnaires should not be left for clients to pick up as they please, since some will not bother to pick them up—increasing the likelihood of bias. Response rates may be low, however, as follow- up is difficult.
32 23 Administering the Survey EXHIBIT 7 (continued) Handed-out questionnaires, whether completed on-site or elsewhere, have a number of dis- advantages. Post-service results may be difficult to obtain unless clients are asked at the time of service provision to return to the service site in the future to complete a question- naire. Incentives may be necessary. Because of inconvenience for the client and the likelihood of a low rate of response, this approach should be used only if other approaches are infeasi- ble and if a reasonable proportion of clients are expected to return to be surveyed. Also, pro- cedures must be in place to avoid having staff members who provide the service influence client responses. Other survey modes —Internet- and e-mail-based surveys are generally not feasible alternatives for many nonprofit organizations, but this may change as access to and facility with computers improves. If service sites include computer-training facilities, clients may be able to use the terminals to complete questionnaires. This has been successfully done in local Boys and Girls Clubs, for example. Automated telephone surveys ask clients to call a telephone number and enter or speak answers to pre-recorded questions. While this mode is used primarily for commercial or institutional customer-satisfaction surveys now, it could become more prevalent over time. confidentiality is an issue and sensitive questions are included. E-mailed or web- based questionnaires are quick and reduce data entry costs, but require specialized design expertise of the nonprofit organization and computer access and knowledge of the client. Surveys can be administered using a combination of modes. Typically, for exam- ple, questionnaires are mailed or handed out and then clients who do not respond are called. Clients may also fill out questionnaires while questions are read aloud to them by an interviewer. Such combinations are likely to increase response rates and help to deal with special circumstances or client types. However, using multiple modes makes the survey process more complex and requires additional effort to ensure that the information collected in different ways is comparable. Step 8: Take Actions to Achieve a High Response Rate Regardless of which survey mode is used, it is important to get as high a response rate as possible. If all clients are surveyed, the response rate is the number of clients who complete a questionnaire divided by the total number of clients. If a sample of clients is surveyed, the response rate is the number of clients who complete a ques- tionnaire divided by the number in the sample.
33 24 Surveying Clients about Outcomes Whether or not sampling is used, the lower the rate of response, the more likely a survey will produce biased, inaccurate information that is not representative of the organization’s clients. According to the American Statistical Association, “A low response rate does more damage in rendering a survey’s results questionable than a small sample, because there may be no valid way of scientifically inferring the characteristics of the population represented by the nonrespondents” (American Statistical Association 1997, p. 7). While there is no absolute acceptable response rate, a general rule of thumb is that a response rate of 70 percent or better is very good. Surveys that obtain response rates below 70 percent but above 50 percent can be decent, but may require some special analysis to assess their representativeness. The findings from surveys with response rates that fall much below 50 percent need to be viewed with some skepticism. Since the findings of low-response-rate surveys may be called into question, it is xtremely important to take actions to encourage clients to respond. A number of e techniques for increasing response rates are outlined in exhibit 8. Another guide in 5 this series provides additional suggestions. It is not appropriate for organization staff to engage in any behavior that could appear to coerce participation or bias responses. It should be clear to staff and clients that participation is voluntary and that no penalties (such as reduced further service) will result if the survey is not completed. Step 9: Ensure That Surveys Do Not Harm Clients Some survey questions may be especially sensitive to clients and damaging if their answers are attributed to them—for example, if the behavior is illegal or con- sidered undesirable. Questions that ask for ratings of the organization or service or staff members may be problematic if clients perceive their answers could have reper- cussions. Even basic demographic information (such as whether the client is a citi- zen) could be considered sensitive if clients fear that various (including legal) consequences could result from their answers. Finally, certain kinds of questions— such as those that remind a respondent of threatening experiences—may produce emotional distress. The organization should ensure that collection of information for outcome management purposes does not cause harm to clients or pose a risk to them. Client privacy needs to be respected. In some instances, federal or state laws, reg- ulations, or program rules protect client privacy or require that personal information be confidential. An organization’s public or private funding sources may also cover issues of privacy, confidentiality, and risk or harm associated with information collection. 5 See Finding Out What Happens to Former Clients.
34 Administering the Survey 25 EXHIBIT 8 Suggestions for Improving Respon se Rates an advance postcard (or letter). Send In a personalized letter, explain the survey’s importance and use. Include the organization’s name and logo on the postcard or envelope if it will result in clients taking the mailing more seriously. The lett er should be written on Fo r m a iled surveys, include a transmittal letter. organization stationery and signed by the executive director. Add endorsements by others if available (such as prominent board members or community leaders) and if judged to increase se rates. Clearly state that the survey’s purpose is to help to improve services. Be respon polite and respectful, and express appreciation for the client’s time and effort. Referencing specific changes the organization has made as a result of past client surveys may be an incentive to respond. Do not give up after the first try. If questionnaires are mailed, send a reminder postcard or make reminder telephone calls. Send a second questionnaire two weeks later. If returned questionnaires contain identifying information, send follow-ups only to those who have not ye t returned their first questionnaire. If questionnaires are returned without identifiers, send follow-ups to all who received the initial mailing, but try to reduce the likelihood that clients will return multiple questionnaires. Although more costly, mailing by delivery service or priority mail will get more attention than mailing by first-class mail. Fo r phone interviews, try calling at different times of the day and different days of the w eek. Call as many times as possible if there is no answer. Include stamped, self-addressed return envelopes if mailing. Include one each time a questionnaire is mailed out. They should not be pre-printed business reply envelopes, however, but have real stamps affixed. Completing the survey should be a convenient, Make the questionnaire easy to complete. positive experience. The questionnaire should have a simple, easy-to-follow format, and clear instructions. Provide incentives. Monetary incentives can be effective in attracting a recipient’s attention. Offer a ticket to the movies or a sports event, or a chance to win a prize in a lottery— although lotteries require knowing who responded. If questionnaires are completed at a service site, incentives might be bus passes or coupons for use at a fast food restaurant, grocery store, movie theater, and so on. Local businesses might be willing to provide such awards. Use multiple modes. Use telephone interview follow-ups for those who do not respond by mail, or allow clients to come to a service site to be personally interviewed. Allow clients the option of either mailing their questionnaires back, bringing them back, or coming to a service site to complete them. Ask caseworkers to encourage clients to cooperate in planned surveys. During service, ca seworkers can emphasize to clients the importance of surveys for helping identify needed improvements to the program. Caseworkers should be careful, however, not to make clients f eel coerced into participating in surveys.
35 Surveying Clients about Outcomes 26 Client concerns about confidentiality can negatively affect response rates, cause bias, and affect the validity of the information gathered. Some clients are only likely to give honest answers or answer sensitive questions if they believe their responses are anonymous (that is, cannot be traced to them) or confidential (that is, can be traced to them but will not be inappropriately disclosed). More information on dis- closure of client responses is presented in exhibit 9. Step 10: Administer the Survey Survey administration is the single most time-consuming step in surveying clients. It involves both collecting the information from clients and ensuring the quality of the information-collection process. A staff person should be responsible for directing these activities and adhering to the planned procedures and schedule. If an outside contractor conducts the surveys, there should be ongoing coordination with the contractor throughout the process. Regardless of the survey mode used, administration includes identifying which clients will be surveyed and when, and tracking the survey’s progress. A system that triggers follow-up mailings or phone calls to clients who have not responded by a certain date is necessary. Some of the major administrative tasks for the different modes include the fol- lowing: Self-administered surveys require printing or duplicating questionnaires, preparing cover letters and reminder postcards, selecting or sampling clients, stuffing envelopes, distributing or mailing questionnaires, opening and log- ging returned questionnaires, preparing questionnaires for data entry, entering responses into a data file, coding responses to open-ended questions, and training and supervising staff members. Interviewer-administered surveys require recruiting, training, and supervis- ing the personnel who conduct them. In-person interviews involve scheduling and tracking the process, preparing questionnaires for data entry, and entering the information into a data file. Telephone interviews also involve scheduling and monitoring calls, and tracking completions. Electronic surveys require notification to clients and electronic tracking of progress in addition to the programming involved. Many of these tasks can adversely affect the quality of the results if not handled properly, so someone must be responsible for providing ongoing quality control of the survey process—whether the survey is done in-house or through a contractor. Some quality control tasks are listed in exhibit 10.
36 Administering the Survey 27 EXHIBIT 9 Guidelines on Disclosure of Client Responses Clients should not be, or feel they are, Inform clients that their responses are voluntary. forced to participate in surveys. Informing them they can refuse to answer any of the questions on the questionnaire, or decline to participate entirely without fear of reprisal, is standard procedure. It is certainly acceptable to appeal for client participation, however, stressing that client input is very important for improving service. This a Make responses anonymous, if appropriate. pproach is a low-cost and simple way to dealing with sensitivities and privacy concerns. No identifying information about clients is on the questionnaires. Demographic information may be included, but designed so that re spondents cannot be identified. If clients are comfortable that they will remain anonymous, then they may respond more readily and increase the response rate. However, second or additional mailings of the questionnaire will have to be sent to all clients, since it is not possible to know which clients have returned their questionnaires. Anonymous questionnaires also make it difficult to assess survey bias. Finally, anonymity limits analysis. Fo r example, it is impossible to link survey responses with demographic characteristics and information about clients from other sources, such as organization service records. Such consequences have to be weighed against the benefits of surveying clients anonymously. Promise and maintain confidentiality. ntiality is probably the most effective means Confide sensitivity and privacy concerns against survey quality and analysis needs. to balance De tailed demographic information and detailed service data from client records can be linked individual clients and their answers. However, access to client answers is limited only to to those who process and analyze the survey, and is not available to program staff or casework ers. Individual responses are never made public or, for that matter, used internally other than in ways that been have disclosed to clients prior to their participation. The link be tw een questionnaire responses and other data from clients’ records can be made by name, address, Social Security number, or case file number included on the questionnaire, but another option is to assign a special survey tracking number to each questionnaire and use no other identifiers. Obtain consent where necessary or appropriate. Surveys of children, for example, may require parental notification or formal (informed) consent. It may also be appropriate to obtain the consent of clients where feasible if others, such as caregivers, are to be surveyed. Asking for consent may also enhance the sense of trust and understanding of the survey’s value.
37 28 Surveying Clients about Outcomes EXHIBIT 10 Qua lity Control Tasks staff. Th Train ose involved in survey implementation need to be adequately trained and their work review ed for quality and consistency. Revi Before questionnaires are mailed or interviews are conducted, client ew the client list. lists should be checked for accuracy, completeness, and possible duplication. Check sampling procedures. If samples are used, procedures must be monitored to ensure that they are drawn correctly. Monitor interviewers. Interviewers need to be monitored to ensure that they follow appropriate procedures. The vari ous stages of the survey process—from distribution to Maintain tracking records. fo llow-up, return, and entry into a data system—must be tracked to prevent losing or incorrectly handling questionnaires. If respondents are identified, returned questionnaires need to be checked for duplicates. Enforce confidentiality procedures. Confide ntiality procedures need to be monitored to ensure adherence to appropriate safeguards. P eriodically review the entire process. Survey procedures should be assessed periodically— say, annually. The questionnaire should be reviewed for relevance and utility. Although it is important to use the same questions over time (so that service outcomes can be compared and progress measured), it may be appropriate to make some alterations every now and then. Modification of service mission, service delivery, or outcome objectives may prompt a revi sion.
38 SECTION V Co llecting and Using the Data Step 11: Enter and Tabulate the Data Once surveys are completed, the information should be reviewed for low response and bias, collated or entered into a data file, tabulated, and analyzed. If the survey results are likely to affect important policy or program decisions, yet are based on a low response rate, the demographic characteristics of those who respond should be compared with the characteristics of all clients (using organiza- tional records) to check for systematic differences. An intensive follow-up can contact and survey a small random sample of nonrespondents to determine if their responses to the survey questions are systematically different from those of respon- dents. If substantial differences are observed, the organization will need to be espe- cially cautious about the findings. It may be possible to adjust survey results to reduce nonresponse bias, but assistance from a survey specialist may be helpful in the adjustment process. Data may be tabulated “by hand,” counting the number of responses to each response category for each question—for all respondents or for selected subgroup- ings. This approach is feasible if the number of clients surveyed or responding is small, and the number of questions is limited. It minimizes the use of technical resources. However, hand tabulation restricts possible analyses to simple frequency counts and perhaps a small number of basic cross-tabulations. It also limits report presentation options to basic tables—although specialized computer software can be used for presenting results that are not drawn from a data file. For some organiza- tions and purposes, hand tabulating the information may be adequate, providing basic survey results without file creation or use of a computer. 6 Computer analysis of data enhances the analysis and is usually preferred. The first step is organizing the information into a data file, which most often means enter- ing survey responses into computer software that organizes and helps clean the 6 The pre-programming needed for surveys done by e-mail, the Internet, telephone using Computer Assisted Telephone Interviewing (CATI), or touch-tone entry also produces a data file for analytic purposes. Such files are available immediately following completion of the interviews. Analysis can then follow without the delay of having to enter the data.
39 30 Surveying Clients about Outcomes entries, and performs various tabulations and analyses. Since data entry errors are possible, double entry, or at least double-checking some percentage of each batch of 7 questionnaires entered, is recommended. Other data entry options are also possible. For example, clients may enter responses directly into a computer, eliminating data entry by staff. Or information can be optically scanned when questionnaire responses are placed on special answer sheets. Scanning may be practical if there are a large number of questionnaires and the questions remain constant over time. The investment in creating a special ques- tionnaire and computer capability may be less than entering the data by hand. Clients must also complete questionnaires carefully, since marks that are in the wrong place or are made incorrectly can lead to erroneous results. Here are some ways to ease the data entry process: Create a codebook. Indicate the structure of the data file and the numeric codes that correspond to each response for each question. Review the completed questionnaires. Examine them for irregularities— such as multiple answers to single-response questions, or marks placed between categories as opposed to checking one or the other. Establish rules for how to handle missing information—when clients do not respond to a par- ticular question—and other situations so they are handled consistently. Add a unique survey response number to each Assign a survey number. questionnaire, as well as the date it was returned or the time period of the survey. Determine links. Based on what has been promised to clients, decide how, if at all, to incorporate identifying information or codes to link questionnaire responses to data from client records. Establish a process for open-ended questions. First, read a sample of the answers to a question to establish basic categories into which the remaining answers can be grouped. Then, assign numerical codes to each of the cate- gories. Then, read and code all the questionnaire responses, and enter the codes in the data file that contains responses to the closed-ended questions. Or type responses in a list for review by staff (preferably edited to remove any references to specific staff or clients). Bear in mind information not coded and entered in the data file cannot be analyzed in the same way as the data from closed-ended questions. 7 For surveys involving samples, statistical “weights” may need to be included in the data file. Weights can be used either to account for special sample design features or to compensate for nonresponse. A survey or statistical specialist should be consulted on this issue.
40 Collecting and Using the Data 31 Step 12: Analyze the Results 8 or basic outcome- F Once information is available, it needs to be analyzed. management purposes, analysis is not especially complex or difficult. The most basic form of survey analysis consists of examining responses to each question—one at a time. Starting with the “raw” responses, first calculate the per- centage of clients giving each response to each question. For example, if responses to the question, “How satisfied are you with the quality of care you received?” include “very satisfied,” “somewhat satisfied,” “not very satisfied,” “not at all satis- fied,” and “I don’t know,” the responses can be analyzed in several ways. Dividing the number of clients giving each response (such as for those saying “very satisfied”) by the total number of respondents produces a percentage. All such percentages con- stitute the percentage distribution for that question. Combining categories (for example, adding “very satisfied” and “somewhat sat- isfied” responses and dividing by the total number of respondents) produces a “total satisfied” percentage. This combination provides a single number for use as an out- come indicator. Numerical values can also be applied to each response to produce an average score. For example, a value of 4 can be given to all “very satisfied” responses, a v alue of 3 to “somewhat satisfied” responses, and so on. Multiplying the number of clients responding “very satisfied” by 4, the number responding “somewhat satis- fied” by 3, and so on, then summing the products and dividing by the total number of respondents, results in an average score that can range from 1 to 4. The higher the all number, the higher the level of satisfaction. A score of 4 would indicate that clients are very satisfied. However, information on the distribution of scores is lost in the process. For example, a score of 2.5 could be achieved because half of all client respondents say “very satisfied” and half say “not at all satisfied;” or it could be achieved if one-quarter each of all client respondents say “very satisfied,” “some- what satisfied,” “not very satisfied,” and “not at all satisfied.” These two results are, obviously, very different and therefore such scores should not be the only figures produced. The calculations should be formatted as easy-to-read tables or graphs. Presentations should include the number of respondents on which any figure is based. Percentages based on smaller numbers of respondents are not as reliable as those based on larger numbers. Often, responses include numerical counts—such as how many performances or counseling sessions clients attended. Responses can be grouped into a small number of ranges with a count of the number of clients that fall into each range. Averages may also be useful. Clients may be asked, for example, “Over the past six months, how many times did you come to the center for emergency assistance?” The sum of 8 More guidance on analyzing outcome information is available in another guide in this series.
41 32 Surveying Clients about Outcomes the number of times each client came in divided by the total number of client respon- 9 dents is the average ( ) number of service visits made by clients. Again, the use mean of averages masks wide variations in responses and should be used along with other 10 Other types information, such as the lowest and highest values reported by clients. of comparisons are illustrated in exhibit 11. These kinds of analyses can be done routinely, in house. At least initially, how- ev er, obtaining assistance from those with expertise in survey analysis—perhaps aculty or graduate students at a local college—may be useful. This assistance will f help to establish a basic analytic routine to follow and a recommended format for reporting. Step 13: Prepare and Disseminate Reports Reports are the basic products of the client survey process. They represent an organized, thoughtful way of presenting information, findings, and recommenda- tions for action. At the same time, they provide permanent documentation for later use or reference. How often reports are prepared should reflect an organization’s outcome man- agement requirements. While the interval will vary, reports should be prepared on a regular basis—annually, quarterly, or monthly. Where there may be service results with short-term fluctuations, for example, information should be gathered and reported relatively frequently. The primary purpose of the client survey report is for outcome management. The report will be used by the organization’s program managers and senior executives for reviewing service outcomes, evaluating what needs improvement, and taking appro- priate action. These uses are time-sensitive, so reports should be prepared promptly following survey completion. Where surveys are conducted on a continuing basis, as opposed to in batches, reports should be produced at regular time intervals. If used internally, reports can be less formal than those that may circulate more widely. Even so, they should reflect senior executive and management interests, information needs, level of sophistication with respect to analysis, and preferences regarding report length and level of detail. A standard report might include the following: Summarize and succinctly highlight the survey’s main findings Highlights. and implications. Note any significant caveats related to the reliability or gen- eralizability of the information. 9 Other statistics may also be useful, such as the mode (the most frequently reported response) or the median (the middle response, where half of all responses are above that number and half below). 10 Another measure of dispersion is a standard deviation, which consists of the range from the mean within which approximately one-third of all values fall, assuming the values are distributed in a normal curve.
42 Collecting and Using the Data 33 EXHIBIT 11 Ways to Analyze C lient Survey Data parisons across client subgroups. Th is comparison involves separating clients by key Com demographic or service characteristics, computing percentage distributions or other statistics for each subgroup, and comparing the results across subgroups. For example, the analysis could separate client responses by age, case manager, or frequency of service visit. Such comparisons are likely very useful for identifying service problems. Satisfaction ratings or improvement in condition may vary by client type, the amount or type of service re ceived, the program offering the service, or who provided the service. They can on ly be done if the necessary information is collected in the survey or the questionnaire se can be linked with program records containing the information. respon Com This compar ison shows whether the organization is meeting, parisons against targets. exceeding, or failing to meet its goals. Compar isons with previous client surveys. This comp arison allows for analysis of trends and indicates whether outcomes are improving, worsening, or staying constant over time. isons with previous client surveys on similar programs. arison shows how Compar This comp the organization’s outcomes correspond to those of other providers. Comparability of the data, however, is very important to ensuring that such comparisons are valid. It is also important to determine whether the other organizations serve similar clients and provide similar functions. Com parisons against professional standards. Where th ere are appropriate external benchmarks, this type of comparison also shows how the organization’s outcomes measure up. Com parisons of responses to one question with responses to others. Cros s-question comparisons help to describe patterns and explain why answers vary. There are various analytic strategies for examining responses to multiple questions simultaneously—testing for differences among groups and looking for patterns. For example, overall dissatisfaction with services may be highest for those clients who are dissatisfied with the accessibility of the service facility. Such issues can be examined by cross-tabulating responses to the questions of interest. The comparison can help identify the extent to which service quality characteristics (such as accessibility) are important to overall client satisfaction.
43 Surveying Clients about Outcomes 34 Include a sketch or table profiling survey respondents, Respondent profile. either in demographic or service characteristic terms. Findings. Know the audience’s preference for format and level of detail and make it reader-friendly. Display information in tables with percentage distri- b utions or as line, bar, or pie charts. Display so that comparisons can be eas- ily made between the responses of all respondents and those of significant demographic or service characteristic subgroups. State major findings in text or bullet format. Put more important findings up front, reflecting either the priority of the organization’s outcome indicators or some judgment as to their significance. If samples are used, provide confidence interval information. Attach an appendix with a description of the survey method; the A ppendix. questionnaire; detailed data tables (if not presented in the findings section); response rates; and a discussion of issues or problems that arose in the course of survey administration or analysis. The appendix is useful for answering questions about the survey and as documentation for future reports. Step 14: Encourage Use of the Survey Information Completing a client survey and report is not the end of the outcome management process but, in some respects, the beginning. Those in a position to affect organiza- tional policy and service delivery need to review client survey results as a part of nor- mal policy, program, activity, and service-delivery deliberations. The information can be used in a number of ways, such as identifying where improvement has occurred and where it needed identifying trends, what works, and “successful practices” clarifying organizational and program purposes for staff, motivating them, and serving as a basis for recognizing their good performance encouraging innovation by testing options for alternative practices or new pro- grams before deciding on full implementation informing board members and external stakeholders, for purposes of account- ability, fundraising, and marketing services attracting volunteers and clients Probably the most important of these uses is identifying program weaknesses and shortcomings, and serving as a catalyst for improvement. It is certainly appro- priate to recognize instances where an organization’s outcome expectations have been exceeded—and to point this out as a positive finding. However, from a program improvement perspective, it is likely to be more important to focus on failures to achieve targets or outcome expectations.
44 35 Collecting and Using the Data Where results do not meet organization targets, are trending in the wrong direc- tion, or fail to measure up to the results of other organizations or to industry stan- dards (if known), explanations need to be sought. Sometimes reasons for low or poor ratings can be found within the survey results, through additional analyses. For e xample, differences among service ratings (some better than others) or in client improvement experience by subgroup should be explored. Responses to open-ended questions can also be examined. Often, however, the explanations will need to be sought from contextual information, from other sources of outcome data, by ques- tioning program staff, by holding focus groups with clients, or from more in-depth studies. Recommendations for action or further exploration should be developed from the analyses and explanations of the survey findings. Policy or service delivery methods may need to change, or organization expectations (and targets) may need to be adjusted—or both. To best use the client survey findings, there should be a plan for taking next steps. In a small organization this planning may seem unnecessary, but even small organi- zations should consider how best to use client survey information for improving results. Some suggestions include the following: Discuss results with supervisors, case managers, and others involved in service delivery. This discussion could be done during staff meetings or performance reviews, in newsletters, or via other internal communications. Service shortfalls should be approached in a constructive atmosphere. Top management taking client feedback seriously can go a long way toward rein- forcing the organization’s focus on client results. Discuss results with staff. It is often helpful to have a meeting with staff to discuss the findings. Special attention should be paid to bad or worsening out- comes and to very good outcomes (to see if these activities contain best prac- tices that can be transferred to other parts of the organization). Once needed improvements have been identified, executives and program managers should focus on explanations and consideration of corrective actions. Plan for follow-up. The organization should also plan to examine future find- ings following the implementation of any corrective actions. Subsequent client surveys would be expected to measure improvement if, indeed, that is the result. Beyond consideration of changes to policies, programs, Revisit expectations. or service activities, client survey results may stimulate adjustment or redefi- nition of expectations. Sometimes targets may be too high or too low. Hold discussions with clients. It may also be useful to hold focus group ses- sions with a small number of clients to discuss service shortfalls observed in the survey and obtain their views about solutions.
45 Surveying Clients about Outcomes 36 Reports for external distribution have separate purposes and audiences. Purposes could include fostering community trust, understanding, support, or funding. Audi- ences could include board members, funders, advocates, or the media. Because of these differences, external reports may need to be organized and presented more formally. Good outcomes have public relations value to the organization, and they can be promoted in such a report. But when outcomes do not measure up, negative reaction or publicity is possible. In that case, the organization could include in the report actions already taken or that will be taken to improve the situation. Having correc- tive actions in place demonstrates a proactive approach to recognizing, dealing with, and resolving problems (Hatry et al. 1998, p. 15).
46 SECTION VI Cost and Time ements Requir Unlike surveys for program evaluation or general research, routine client surveys do not usually require elaborate questionnaires or complex survey designs. Nonetheless, they need to be done properly to ensure that useful and credible infor- mation is collected. Ensuring effective information, in turn, calls for commitment of organization resources—staff time, money, or both. How extensive a commitment depends on a number of considerations. While calculations have to be made for each program, this section is intended to help organization personnel think through the cost and time requirements of doing routine client surveys. The discussion is divided into different phases of the client survey process. Initial Decisions Initially, basic client survey parameters need to be established. Establishing parameters is almost always best done by organization personnel as opposed to tech- nical advisors or contractors. Where possible, a lead staff person and, possibly, a orking group should be charged with carrying out the start-up tasks. Several work- w ing days of staff time are likely to be needed to identify the outcomes to be tracked, consult with selected other staff, board members, clients, and representatives of funding organizations, advocacy groups, or public agencies to identify the information needed, possibly through focus groups, review surveys conducted by similar organizations, decide when during service provision clients will be surveyed, decide the number of times clients will be surveyed, and decide how frequently surveys will be administered. If it is determined that surveys are to be developed or conducted entirely in- house, it may be necessary for one or more staff members to acquire basic survey skills—by taking a course in survey research or reviewing a basic text. If surveys are
47 Surveying Clients about Outcomes 38 to be done primarily in-house with the assistance of outside advisors, it may also take some staff time to identify an appropriate consultant and establish a good work- ing relationship. Finally, if an outside firm is to be hired to design or administer the surveys, it will take staff time to prepare the work specifications, identify appropri- ate contractors, review their credentials, check their references, and contract with one of them. The process of hiring a contractor is likely to involve several days of staff effort at the front end in addition to the time needed to work with the contrac- tor during the various stages of survey development and administration. ping the Questionnaire Develo Survey development work tends to be more technical than start-up activities because it involves selecting and designing the mode of survey administration, designing the process for identifying clients to be surveyed, deciding on the number of clients to be surveyed and the method of selecting them—including developing a sampling scheme, where appropriate, writing questions and designing the questionnaire, and pretesting the questionnaire. These tasks are likely to require several working weeks of effort. Also, where sufficient expertise is not available in-house, getting advice and assistance from out- side survey or statistical specialists may be required, by either using volunteers or hiring fee-for-service consultants—the latter constituting an additional survey cost. Even where outside expertise is used, organization personnel should be acquainted with the basic issues involved in survey development, review all development prod- ucts, assess the trade-offs between cost and accuracy considerations, and evaluate all products based on their utility for outcome measurement. These activities require the commitment of staff time, but generally do not have to be repeated for each subse- quent wave or round of client survey. Admin istering the Survey Managing the information-collection process and ensuring its quality is the sin- gle most time-consuming step in surveying clients. It is a recurring process, and the amount and timing of staff involvement depends on how continuously or frequently surveys are conducted, how many clients are surveyed, and whether information is collected by in-house staff or through an outside contractor. In addition to personnel requirements, survey administration generally involves spending money, the amount dependent on which mode of administration is used, wage rates, supply costs, inter-
48 39 Cost and Time Requirements view or questionnaire length, how many clients are surveyed, and whether in-house staff, volunteers, or contractors are used. Survey administration tasks include the following: Checking, organizing, and culling client rosters or lists, where used for sur- ve ying clients, and monitoring selection and sampling procedures; T raining, overseeing, and monitoring staff involved in survey administration, including interviewers if they are used; Considering actions to achieve a high response rate; Mailing or distributing questionnaires or interviewing clients; Maintaining records to track the various stages of the survey process; Putting procedures in place to ensure that surveys do not harm clients, and ensuring that confidentiality procedures are adhered to; and Periodically reviewing the entire process. Mail questionnaires are usually the least expensive survey mode. Personnel costs include the clerical tasks of reviewing client lists, drawing client names, gen- erating address labels, stuffing envelopes, and tracking responses, and the supervi- sory task of directing such activities to ensure adherence to planned procedures and schedule. For non-personnel costs, postage costs need to be estimated (two pages of questions, a cover letter, and a return envelope generally weigh one ounce). The costs of postage-paid return envelopes and of sending clients second questionnaires or pre- or post-survey letters or postcards, if done, also need to be taken into account. Finally, printing costs and the costs of paper and envelopes need to be accounted for. If in-house interviewers are used, they need to be trained and monitored, which generally involves staff time for interviewers and a supervisor. Interview time, whether logged by staff or specially hired personnel, needs to be taken into account in estimating survey costs. Naturally, interview length affects costs: the longer the interview, the more expensive it is. If interviews are done in person, travel time and costs also need to be considered. When calculating the costs of telephone interviews, bear in mind three to four closed-ended questions can generally be asked per minute, b ut this estimate does not include the time it takes to contact clients, which may involve multiple tries. Open-ended questions take longer to administer. If an outside contractor conducts the surveys, there are personnel requirements in addition to contractor costs. In particular, a lead staff person needs to budget time for having ongoing contact with the contractor throughout the process. Co llecting and Using the Data F ollowing each survey wave or round, a variety of post-survey activities need to be undertaken. Some may require assistance from a survey specialist and others are best done by organization personnel. Tasks include
49 Surveying Clients about Outcomes 40 coding open-ended questions, entering data, reviewing results for low response and bias, tabulating and analyzing the data, writing reports, using the information for outcome management and follow-up purposes, and reviewing survey procedures periodically. Coding and data entry are clerical tasks, but reviewing results for low response and bias, tabulating and analyzing the data, and writing reports involve some tech- nical skill. If these skills are not available within the organization, outside assistance may be needed—at least for the first survey round, and the process emulated in sub- sequent rounds. Therefore, some amount of the time of program managers, case- w orkers, and others should be factored into estimates of the effort involved in the client survey process.
50 References American Statistical Association, Section on Survey Research Methods. 1997. “How What is a Survey? to Collect Survey Data.” brochure series. http://www.amstat. org/sections/srms/whatsurvey.html (Accessed March 4, 2003.) ———. 1999. “Designing a Questionnaire.” What is a Survey? brochure series. http://www.amstat.org/sections/srms/whatsurvey.html (Accessed March 11, 2003.) Y outh Development Outcome Measurement Boys & Girls Clubs of America. 1998. ool Kit, 1998. T Dillman, Don A. 2000. Mail and Internet Surveys: The Tailored Design Method. New York: John Wiley & Sons. F olz, David H. 1996. Survey Research for Public Administration. Thousand Oaks, Calif.: Sage Publications. Fo wler, Floyd J. Jr. 2001. Survey Research Methods. Thousand Oaks, Calif.: Sage Publications. Hatry, Harry P., John E. Marcotte, Therese van Houten and Carol H. Weiss. 1998. Customer Surveys for Agency Managers: What Managers Need to Know. W ashington, D.C.: Urban Institute Press. Newcomer, Kathryn E., and Timothy Triplett. 2003. “Planning and Using Surveys to Handbook of Practical Program Collect Useful Program Information.” In Evaluation: Second Edition, edited by Joseph S. Wholey, Harry P. Hatry and Kathryn E. Newcomer. San Francisco: Jossey-Bass. Salant, Priscilla, and Don Dillman. 1994. How to Conduct Your Own Survey. New Y ork: John Wiley & Sons. Schuman, Howard, and Stanley Presser. 1996. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. Thousand Oaks, Calif.: Sage Publications. Sudman, Seymour, and Norman Bradburn. 1992. Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass.
52 APPENDIX A Sample Client Satisfaction Survey Thinking about your experience with this Red Cross Service, on a scale from 1t (with 1 being poor and 6 excellent), please rate each of the following by fill- o6 ing the bubble that best represents your thinking. Very Does Not Excellent Poor Good Moderate Apply Adequate Good 12 3456 0 On-time performance- o timeliness o o o o o o Worker ’s o o o o courtesy o o o Compet ence of driver or worker o o o o o o o Cleanliness of products or facility o o o o o o (van, shelter, food, etc.) o Ease of enrolling in o this service o o o o o o Treating y ou with dignity and respect o o o o o o o Overall satisfaction with o o o o o o this Red Cross service o Source: Excerpted from Guide for Conducting Service Quality and Effectiveness Surveys: Indicators of Chapter Performance and Potential , Red Cross, May 2002.
53 APPENDIX B Sa mple Client Survey about and Benefits Program Outcom es In order to improve mental health services to people like you, we need to know what you think about the treatment you received, the people who provided it, and the results of this treatment. Please indicate your agreement/disagreement with each of the following state- ments by circling the number that best represents your opinion. If the question is about something you have not experienced, circle the number 9, to indicate that this item is not applicable to you. Strongly Strongly Not Neutral Disagree Disagree Applicable Agree Agree 1234 5 9 AS A DIRECT RESULT OF SERVICES I RECEIVED I deal more effectively with daily problems 1 2 3 4 5 9 1 2 3 4 I feel better about myself 9 5 I am better able to control my life 2 3 4 5 9 1 I experienced harmful medication side effects 2 3 4 5 9 1 1 2 3 I am better able to deal with crisis 5 9 4 I am getting along better with my family 1 2 3 4 5 9 I do better in social situations 1 2 3 4 5 9 I do better in schoolwork 1 3 4 5 9 2 1 3 4 5 9 I do better with my leisure time 2 1 2 3 4 My housing situation has improved 9 5 My symptoms are not bothering me as much 1 2 3 4 5 9 I have become more independent 1 2 3 4 5 9 The me dications I am taking help me control symptoms that used to bother me 1 2 4 5 9 3 I have become more effective in getting what I need 1 2 3 4 5 9 I can deal better with people and situations that used to be a problem for me 1 2 3 4 5 9 Source: Excerpted from “Proposed Consumer Survey Items,” Mental Health Statistics Improvement Program (MHSIP) Task Force on Mental Health Report Card: Consumer Survey (http://www.mhsip.org/rptcard.html, accessed April 11, 2003).
54 APPENDIX C A Sampling Primer SAMPLE TYPE Samples should be selected to ensure that the information generated from them can be projected reliably to the full client base. Every client should be given an inde- pendent and measurable chance of being selected. Most samples used for surveys involve some form of random or probability selection, where all of an organization’s 11 clients have an equal chance of being included. When client samples are random, unbiased, and representative, survey results can be used with reasonable confidence to draw conclusions about the organization’s total clientele. The following are among the techniques that can be used to draw a sample: Draw a random sample of the total clientele. If a client list contains names n and addresses, but little else, the easiest technique is to select every th client from the list, using a random start point. If, say, every fourth client is to be selected, start by arbitrarily choosing a number between 1 and 4. If that num- ber is 3, then begin with the third name on the list and select every fourth name after that. Draw random samples separately for important subgroups. If organization records contain relevant information for analysis or comparison purposes (such as client demographic information or service experiences), random samples 12 can be drawn for each group. This process has two steps. First, sort the list into appropriate subgroups (say, by age or by one-time versus multiple-time service recipients) that follow from the outcome indicators of interest to the organization. Second, select every n th client within each group, using a random n n can be used for each subgroup. If, start point. Either the same or a different for example, there was reason to survey one-time service recipients at a higher n w ould be smaller for the former sub- rate than multiple-time recipients, the group than for the latter. If a client list is not available, use other random selection techniques. For e xample, days of the week can be selected randomly and, on those days, all or every n th client provided service could be surveyed. If random sampling is 11 Strictly speaking, a true random sample requires drawing names from the full client population in a way that, each time a name is drawn, every client in the population has an equal opportunity to be selected. To meet this requirement, sampling must be done with replacement: Once a name is selected, it is then placed back in the popu- lation before drawing the next name. It is possible, therefore, for the same name to appear more than once in a sample. 12 When subgroups are selected with different selection probabilities, these probabilities need to be taken into account when the information is tabulated. The organization should consult with a statistical specialist on such matters.
55 Surveying Clients about Outcomes 50 not feasible, clients can still be surveyed, but there may be no way to ensure that results apply to the entire clientele. SAMPLE SIZE It is necessary to determine how large a sample to select each time clients are sur- ve yed. A number of factors should be considered in setting sample size, including likely response rates; the extent of variability in the organization’s client base and its service experiences; and the number of client subgroups about which information is needed. Generally, the more precise information an organization needs to collect, the larger the sample size needed. Larger samples are more likely than smaller ones to produce the results that would be obtained if all clients were surveyed—i.e., the mar- gin of error is smaller. Since larger samples cost more (in dollars and effort) than smaller ones, budget becomes a key consideration. Sample size, then, generally reflects the interplay between precision and cost. When using a sample, organization executives need to decide what level of pre- cision is meaningful and important to them. The precision of an estimate is usually stated in terms of a “confidence interval.” If, for example, 30 percent of a client sam- ple report no desired improvement in condition after receiving service, the more technically correct conclusion may be, “95 percent of the time, the true percentage lies somewhere between plus or minus 7 percentage points of 30 percent—or between 23 percent and 37 percent of all clients.” A policy question for organization ecutives, then, is would some change in policy be called for if the answer were ex 37 percent but not 23 percent? In essence, can a range this wide be tolerated? If a confidence interval is too wide, more precision is achievable at a higher cost. How much precision is needed depends on the way information will be used and the risk associated with having any particular level of precision. The precision or accuracy of a sample survey estimate can be calculated for the sample as a whole or for any subgroup . Confidence intervals will be wider for sub- groups than for the total sample. They will also be wider for smaller subgroups than for larger ones. It is therefore important to consider which client subgroups are per- tinent to the organization’s outcome indicators when deciding on sample size. When subgroups of a sample are compared to each other, a statistical calculation can determine the probability that observed differences are real. That probability means that such differences would be observed if all clients were surveyed, as opposed to being an artifact of the sampling process. Since probability also depends on sample size, the need to make such comparisons should be considered in deter- mining size.
56 APPENDIX D Issues to Consider before Questionnaire Developing a The unique inclinations and charac- Consider what can be expected of clients. teristics of an organization’s clients are particularly important for questionnaire development. Thinking about the following kinds of issues may help question writ- ing and questionnaire design: How receptive to surveying will clients be? How much do they know about the services and service-delivery issues of interest? Are they likely to accurately remember their experiences? Do they have the educational and language skills to deal with the relevant issues and terminology? Are there special sensitivities associated with certain questions that may be asked? These factors can have a bearing on how survey questions are worded, format- ted, and ordered. For example, sensitive questions are usually placed near the end of a questionnaire, not at the beginning. Or, if some clients are not expected to know certain terms, they may need to be explained in the survey. Also, if some clients do not have sufficient English language skills, the questionnaire may need to be trans- lated into other languages. Consider precisely what outcomes are to be measured. Questions asked in a client survey should follow directly from the organization’s outcome indicators. They should pertain to client behavior, condition, and perception of improvement, as well as client evaluation of (and satisfaction with) service delivery. Look to see if there are pre-existing survey questions that can be used or adapted. Early in the questionnaire-development process, it is a good idea to search for client surveys and questions that have been used by others. Questions are likely to be available for many services. Sometimes these can be found in formal reports b ut otherwise they may be located through informal networking, web searches, reviews of professional journals, or by asking those in peer organizations and pro- fessional associations. When found, such questions may not precisely fit the bill. If so, they can be modified to make them more appropriate for the organization’s use. One advantage to using pre-existing questions as-is: They allow for comparisons with other entities, offering a benchmark that may prove valuable for interpreting results.
57 Surveying Clients about Outcomes 52 Consider what demographic and other interpretative information is needed. In addition to questions measuring outcomes, a client survey questionnaire should contain questions pertinent to understanding and interpreting responses to outcome demographic or service characteristics. Demographic r measures, such as elevant characteristics include age, income, household structure, area of residence, race, eth- nicity, education level, disability, gender, and the like. Service characteristics include type and amount of service provided, service dates, service providers, and so on. It may not be necessary to inquire about such information if it is available from orga- nizational records that can be linked to individual survey responses. However, such linking may be technically difficult or pose confidentiality and response-rate prob- lems. If so, the information should be asked for in the questionnaire. Consider whether any “issue-of-the-moment” information will also be col- lected. Beyond collecting core information used to measure and interpret outcomes, it may be useful to include in a questionnaire a limited number of other, time-specific matters that vary from survey to survey. A small portion of the questionnaire, there- fore, can be reserved for covering special issues, such as those that may assist in reviewing or rethinking service policies or procedures. These questions may change each time the questionnaire is administered. Possibilities include process concerns, service delivery options, or policy or program changes being contemplated by the organization. Asking about such subjects gives the organization an opportunity to systematically obtain client input and insights on a timely basis. Adding one-time- only questions adds some cost and burden, because the questionnaire, data entry rou- tine, and tabulation procedures will need to be modified each time such questions are added.
58 The Urban Institute 2100 M Street, NW W ashington, DC 20037 Phone: 202.833.7200 F ax: 202.429.0687 E-mail: [email protected] http://www.urban.org
Substance and Mental Health Abuse Services Administration Center for Substance Abuse Treatment Enhancing Motivation For Change in Substance Abuse Treatment Treatment Improvement Protocol (TIP) Series ...More info »
A TreATmenT ImprovemenT proTocol Using Technology-Based Therapeutic Tools in Behavioral Health Services TIP 60More info »
Ryan White HIV/AIDS Program Annual Client-Level Data Report Ryan White HIV/AIDS Program Services Report 2017More info »
Dell EMC NetWorker Version 18.1 Updating to NetWorker 18.1 from a Previous NetWorker Release Guide 302-004-420 REV 02More info »