How to Conduct Cognitive Interviews in Times of COVID-19

The outbreak of COVID-19 has created new challenges for face-to-face research activities, including conducting in-person cognitive interviews. The first lockdown in Germany in mid-March 2020 coincided with the preparation phase of a pretest that we planned to conduct face-to-face in our pretest lab in Mannheim. Since we were no longer able to interview participants in person, we quickly had to come up with alternative ways of conducting pretests without simply switching to the online mode and testing the questionnaires via web probing. This blog post tells the story of how we switched to remote cognitive interviewing and reflects the experience we have gathered with this method over the last 12 months.

Der Ausbruch von COVID-19 hat Face-to-Face-Forschungsaktivitäten vor neue Herausforderungen gestellt, einschließlich der Durchführung von persönlichen kognitiven Interviews. Der erste Lockdown in Deutschland Mitte März 2020 fiel direkt in die Vorbereitungsphase eines Pretests, im Zuge dessen wir persönlich-mündliche Interviews in unserem Pretest-Labor in Mannheim durchführen wollten. Da es uns nicht mehr möglich war, Testpersonen persönlich zu befragen, mussten wir kurzfristig auf alternative Wege der Durchführung von Pretests umsteigen, ohne jedoch einfach nur in den Online-Modus zu wechseln und die Fragebögen mittels Web Probing zu testen. Dieser Blog-Beitrag erzählt, wie wir auf Kognitive Remote Interviews umgestiegen sind, und reflektiert unsere Erfahrungen, die wir in den letzten zwölf Monaten mit dieser Methode gesammelt haben.

DOI: 10.34879/gesisblog.2021.34

Why it is important to conduct cognitive interviews

It is generally acknowledged among survey practitioners that newly developed survey questions require some form of pretesting before they are fielded, and cognitive interviewing is arguably the most popular method in a social scientist’s toolbox¹. Cognitive interviews are semi-structured interviews that focus on respondents’ thought processes associated with answering survey questions. To this end, cognitive interviewers ask so-called “probes” about how respondents understood specific terms in a question, how they remembered the information being sought, and how they arrived at their answer. The aim of cognitive interviewing is to find out whether individual questions are difficult to understand or to answer and whether they are interpreted differently by different respondents. If this is the case, they cannot be expected to measure what they are supposed to measure, or they do not measure it properly.

How cognitive interviews are usually conducted (in normal times)

Traditionally, cognitive interviews are conducted face-to-face with small samples of five to 30 respondents². Ideally, they should take place in a quiet, closed room. GESIS has a lab specifically designed for conducting cognitive interviews and equipped with video and audio recording facilities as well as a mobile eye tracker. Cognitive interviews should generally be recorded with a tape recorder or video recorder, as this facilitates and improves both the implementation and the analysis of the interviews.

How to carry out cognitive interviews remotely

The COVID-19 pandemic abruptly put an end to our practice of inviting participants to our lab. Conducting cognitive online pretests via web probing presents itself as an obvious alternative (see, for example, Lenzner et al., 2018). This method of cognitive pretesting implements probing techniques from cognitive interviews into web surveys in the form of open-ended questions³. Research comparing traditional cognitive interviewing and web probing demonstrates the great potential of the latter method. However, web probing inherently does not replace the dialogue situation, social interaction, and dynamics of cognitive interviews and is not very well suited for pretesting interviewer-administered surveys. To this purpose, we expanded our methods portfolio by conducting personal interviews in remote settings. Two main aspects need to be considered when carrying out cognitive interviews remotely: (1) via which channel to communicate with the respondent and (2) how to present the questions to be tested to the respondent.

Communicating with the respondent

Remote cognitive interviews can take place via video conference or telephone⁴. The pandemic has seen a boost in online video conferencing. Many people previously inexperienced with video conferencing now have webcams and speakers for their PC and/or respective apps on their smartphones. The advantage of video conferencing is that the interviewer and respondent see each other, and non-verbal cues can be integrated into the analysis⁵. In addition, some videoconferencing platforms offer the option to record the interview, which can facilitate documentation. A disadvantage of this procedure is that it systematically excludes participants who have no access to Internet-connected devices.

In the course of a cross-cultural pretest, video conferencing allowed us to carry out cognitive interviews with Arabic- and Dari- speaking refugees located in another region of Germany with the support of interpreters, who were located in yet another region (Hadler et al., 2021). While not every participant in this target group had their own computer, many were well accustomed to video conferencing via smartphones. In this pretest, we learned that it is important to keep the technological burden as low as possible. For video conferencing, choose a platform that allows desktop users to participate directly in their browser without downloading a program or app⁶ and that can be easily accessed via stationary devices and smartphones⁷.

Compared to such a setting, carrying out interviews by phone initially appears to be an out-of-date choice of mode, unnecessarily forgoing the opportunities of modern technology. However, the advantage is that a phone call does not impose a technological obstacle to participants. This option is particularly useful for non-tech-savvy populations, but also for participants without a stable Internet connection or flat rate. Finally, respondents with concerns about data protection may be more inclined to participate in a telephone interview than to go online.

In the case of the cognitive pretest with a refugee population mentioned above, some participants did not have an Internet connection at home that allowed for stable video conferencing and instead participated by phone. In other remote pretests that we carried out during the last 12 months, keeping the interview mode identical across participants was preferable, and hence we conducted the interviews by telephone only (Hadler et al., 2020, Lenzner et al., 2020, Nießen et al., 2020). In contrast to previous concerns, building rapport with respondents proved to work well in this interview setting⁸, regardless of whether we recruited from a pool of participants who were familiar with cognitive interviews (albeit not with remote interviews) or whether participants were entirely new to being interviewed.

Presenting the questionnaire to the respondent

Once the channel of communicating with participants is clarified, researchers must consider how to present the questions to be tested to the participants. Interviewer-administered questionnaires do not require adjustment to a remote setting, as the interviewer can read the questions and response options aloud and record the respondent’s answer. Self-administered questionnaires, in contrast, require adaptation.

One option is to program an online questionnaire with the questions to be tested. At the beginning of the cognitive interview, the participant is sent the link to the questionnaire and fills it out during the interview. We have used this method in several pretests (Hadler et al., 2020, Lenzner et al., 2020, Nießen et al., 2020). The benefit is that the questions are presented in real-time. On the downside, respondents must have access to a device with an Internet connection, which was not the case for all of our potential participants.

If the interview takes place via video conference, screen sharing presents itself as another option. In this case, the interviewer shares the questionnaire with the participant during the interview. The advantage is that the interviewer can control that the respondent is presented the right question at the right time. The disadvantage is that this method only works as long as the video connection is of high quality.

In contrast, the advantage of a paper questionnaire is that it is independent of the respondent’s personal technology. On the downside, researchers must mail the questionnaire to respondents in advance, at best several days before the interview. Respondents may read the questions prior to the interview or even lose the questionnaire. For instance, in the cross-cultural pretest mentioned above, envelopes with questionnaires were lost in the mail and had to be re-sent. However, since many participants only had a smartphone but not a large screen, having paper questionnaires was preferable to participants trying to read the questionnaire on the small screen of a smartphone while video conferencing with the interviewer.

In sum, the channel of communicating with participants and the way the questionnaire is presented in a remote cognitive interview can be mixed and matched in many ways. We acknowledge that – even a year into the pandemic – remote cognitive interviewing continues to pose challenges, many of which we have not even mentioned in this post (these include recruiting participants, making interview recordings, and installing appropriate consent procedures and documentation thereof⁹).

Still, remote cognitive interviewing has also proven to have advantages. For one, interviewing respondents from and at home means less effort in terms of time and mobility on the side of interviewers and participants, and generally more flexible interview times. Second, remote cognitive interviews can be carried out with geographically dispersed participants much more easily, without imposing additional costs for travel or accommodation.

It seems likely that in-person interviews may not be possible in the near future, at least not for everyone. Vulnerable populations, such as the elderly or people with chronic diseases, may be more hesitant to participate in in-person interviews¹⁰, even once vaccines are in place. Therefore, establishing best practices for remote interviewing will remain a relevant goal for the future.

References

Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287-311. https://doi.org/10.1093/poq/nfm006
Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. London: Sage.
Behr, D., Meitinger, K., Braun, M., & Kaczmirek, L. (2017). Web probing – Implementing probing techniques from cognitive interviewing in web surveys with the goal to assess the validity of survey questions. Mannheim, GESIS – Leibniz-Institute for the Social Sciences (GESIS – Survey Guidelines). https://doi.org/10.15465/gesis-sg_en_023
Johnson, D. R., Scheitle, C. P., & Ecklund, E. H. (2019). Beyond the In-Person Interview? How Interview Quality Varies Across In-person, Telephone, and Skype Interviews. Social Science Computer Review. Advance online publication. https://doi.org/10.1177/0894439319893612
Howlett M. (2021). Looking at the ‘field’ through a Zoom lens: Methodological reflections on conducting online research during a global pandemic. Qualitative Research, Advance Online Publication. https://doi.org/10.1177/1468794120985691
Schober, M. F., Conrad, F. G., Hupp, A. L., Larsen, K. M., Ong, A. R., West, B. T. (2020). Design Considerations for Live Video Survey Interviews. Survey Practice, 13(1), 1-11. https://doi.org/10.29115/SP-2020-0014
Lobe, B., Morgan, D., & Hoffman, K. A. (2020). Qualitative Data Collection in an Era of Social Distancing. International Journal of Qualitative Methods, 19, 1–8. https://doi.org/10.1177/1609406920937875
Jenner, B. M., & Myers, K. C. (2019). Intimacy, Rapport, and Exceptional Disclosure: A Comparison of In-Person and Mediated Interview Contexts. International Journal of Social Research Methodology, 22(2), 165-177. https://10.1080/13645579.2018.1512694
Lobe, B., Morgan, D., & Hoffman, K. A. (2020). Qualitative Data Collection in an Era of Social Distancing. International Journal of Qualitative Methods, 19, 1–8. https://doi.org/10.1177/1609406920937875
Foley G. (2021). Video-Based Online Interviews for Palliative Care Research: A New Normal in COVID-19? Palliative Medicine. Advance Online Publication. https://doi.org/10.1177/0269216321989571