The Holy Grail of Survey Design: Achieving a 90% Response Rate
In this talk, Ms. Keniston, director of data and analytics in the division of hospital medicine at the University of Colorado in Aurora, Colo., provided an overview of survey design. Dr. Michtalik, assistant professor of medicine at the Johns Hopkins University School of Medicine in Baltimore, described practices he uses as head of his health system’s annual Hospitalist Morale Survey.
During the design phase, consult resources to design and deploy an effective survey instrument (see additional resources, below). The questions asked and the answers provided are the heart of your survey: each item should be necessary and solicit useful information. Consider whether you can use a validated survey instrument, rather than designing a survey de novo.
If you design your own questions, use familiar words—avoid jargon or slang. Use concrete and specific wording that all respondents will interpret in the same way. Avoid loaded or “double-barreled” questions, e.g., “How interesting and educational was this session?” Separate concepts.
Questions should be grouped by topic. Earlier questions should build rapport, and sensitive questions (e.g., sex, race, age, income) should be near the end, unless extremely important. Including “prefer not to answer” options help create a sense of psychological safety for respondents.
For rating-type response options, provide enough choices to capture all options for respondents, but not so many that they can’t decide. Be cautious with ranking scales: most people can’t rank more than six things reliably. With “check all that apply” responses, people tend to select responses at the top of the list; this can be countered by randomizing the order of the answers. Forced-choice responses, where users must choose yes or no for each option, are effective but can be tiring for respondents. Use open-ended questions sparingly, and only for non-categorical data you cannot get any other way.
Pretest your survey with experts and people who are like your survey population. Ask them to explain their understanding of questions and responses aloud. Check the instrument’s reliability (would the same person read this the same way if they took it again?), face validity (do these questions make sense to your population?), content validity (do experts think this measures the concept?), and construct validity.
Consider the best administration mode for your survey. If it’s online, test for user friendliness on mobile devices. Don’t forget pen and paper—leaving a stack of paper surveys that nocturnists can grab and complete as they leave the office may be more effective than emails.
Dr. Michtalik described the Hospitalist Morale Survey administered annually to five hospitalist programs in one system. The survey uses the Hospitalist Morale Index created and validated by his late colleague, Dr. Shalini Chandra.
Reach out to stakeholders (leadership and local champions) before the survey launch to get buy-in and accurate contact information. Champions should not be in a leadership role; instead, recruit the individual to whom people go if they have a question. Explain the survey’s technical aspects and logistics (especially dates). For end users, explain what changes resulted from the survey and why they should take the survey. For all groups, demonstrating that previous surveys led to change is a major motivator.
Use cheerleading language to encourage people to finish and explain the purpose of a question or section. Progress bars are helpful so respondents feel the survey is progressing, but they can be discouraging for longer surveys. Strategically intermingle your short and long sections.
Test the survey distribution method before you deploy it. Check that email addresses are correct before you begin. Spam folders catch a lot of surveys; avoid using an email alias. Inside your institution, you can ask champions to remind people and note the survey in all-staff email reminders. If you need to send emails outside your institution, talk to their information-technology departments. Institutional emails are less likely to get blocked than those from addresses ending in @gmail.com, @yahoo.com, etc., or sent en masse.
Consider sending confidential (user-specific URLs not linked to identifiers) versus anonymous survey links (a single link sent to all respondents). Confidential surveys allow for targeted reminders to those who haven’t completed a survey versus email blasts to everyone regardless of completion status.
For hospitalists, effective practices include scheduling no more than three reminders per week for alternating a.m. and p.m. times; one hour before shift change seems to work well, and one reminder should be sent on a weekend day.
Empower champions to decide how to remind their colleagues: examples are customizable flyers, pages, text messages, emails, signs, pens, coffee or bagels, and snacks with “guilt labels” with a survey reminder or QR code. Be seen—it is helpful for the overall project team to physically visit sites at least once.
Once the survey response period is over, debrief with stakeholders to see what was effective and what can be improved during the next iteration. Every survey is a learning experience.
Key Takeaways
- Be respectful of respondents’ time and effort by: starting with the end in mind; designing the survey with and for your users; and using the right tool (custom or validated instruments) for the job. Collect only the information you need and can demonstrably use.
- Make the survey user-friendly with a visually pleasing design, considering where and how respondents will take the survey (on a mobile device, pen, paper, etc.).
- Reach out to stakeholders in advance and identify local champions to gain buy-in. Communicate consistently and respectfully, and bear in mind that technology is constantly evolving, so be prepared to react to snags in real time.
- To build support, plan how you will demonstrate to stakeholders (leadership, champions, and respondents) how the data collected will be understood and used and what specific changes could result. Show that their opinions matter.
Ms. Hall
Ms. Hall is the senior medical writer at Emory University’s division of hospital medicine in Atlanta.