Trust, but verify: Harmonization with dedicated control variables

In the finale of season 1 of the blog series, I explore the idea to craft control variables that help us mitigate quality and comparability issues in our source surveys. Embedding such control variables into our ex-post harmonized datasets creates a living documentation of our source material and our harmonization choices. It also allows us to model methodological differences explicitly in our analyses. The post is inspired by the Survey Data Recycling project and represents my musings on their interesting control variable approach.

Im Finale der aktuellen Staffel der Blogreihe erkunde ich die Idee, Kontrollvariablen bei der Harmonisierung zu erschaffen. Diese sollen helfen, Qualitäts- und Vergleichbarkeitsprobleme zu verringern. Solche Kontrollvariablen in den harmonisierten Datensatz zu integrieren erschafft eine lebendige Dokumentation des Quellmaterials und unserer Harmonisierungsentscheidungen. Es erlaubt uns zudem, methodische Unterschiede in unseren Quellstudien explizit bei Analysen zu modellieren. Der Beitrag ist dabei inspiriert von der Arbeit des Survey Data Recycling Projekts und fasst meine Gedanken zu diesem interessanten Ansatz zusammen.

DOI: 10.34879/gesisblog.2021.42

This month’s post is inspired by the ambitious work of the people behind the survey data recycling project (SDR project) 1. I hope to do a collaborative blog post on their project in the future! However, here I want to explore a core idea of their project that I feel might benefit many other ex-post harmonization projects: Enriching harmonized datasets with carefully constructed control variables. Just to be clear, the post below is less a comprehensive overview of the SDR project and more my personal notes inspired by their work.

Ex-post harmonization projects often combine data from surveys that inevitably vary in any number of methodological aspects. This may mean different levels of quality (think different levels of total survey error), but it may also just mean quality neutral differences that are still challenging when we want to harmonize those surveys.

The basic idea now is twofold: Firstly, integrating structured information on methodology and harmonization aspects into the dataset itself in the form of control variables. Secondly, using these variables when analyzing the ex-post harmonized dataset to arrive at more valid and robust substantive results.

The SDR project makes a useful distinction between two broad categories of control variables 2. There are quality controls that capture information about quality aspects of the source surveys in a structured and comparable form. And then, there are harmonization controls that capture (quality neutral) differences between source surveys or information on the harmonization process itself. After all, combining such differences without accounting for them properly can introduce biases to our harmonized data. Of course, these definitions are not always clear-cut. Still, it is useful to think of the two controls as assigning “roles” to the control variables they should play in data analyses. To my mind, quality controls help mitigate the impact of survey errors in source surveys, while harmonization controls help us mitigate the impact of comparability issues.

Quality controls

Quality control variables embed structured information on quality aspects of the source surveys directly into our ex-post harmonized dataset. Such quality controls can then be used to assess and ideally mitigate the differences in survey quality 3. Of course, survey quality is multi-faceted, and as data users, we may not have complete access to all quality-related information we desire. Furthermore, we need to ensure that our intended quality controls can be compared fairly for all source surveys. Otherwise, we might end up punishing surveys that make quality-related information more readily available. Below, we will look at some possible sources for survey quality-related information. After that, we will discuss what to do with the quality controls we have constructed.

Sources of quality information

Documentation

One of the first sources of quality information is the documentation of the source surveys 4. The documentation contains both explicit and implicit information on quality. Explicit information is survey methodological information reported by the data producers themselves. To assess the quality implications, we can rely on a steadily growing body of evidence for many survey characteristics and their impact on survey quality. See, for example, the GESIS Survey Methods Evidence Map for a systematic overview.

However, we can also assess the quality of the documentation itself and use it as an implicit indicator of the quality of a survey program 5. For example, the SDR project takes stock of inaccurate or missing methodological information in the source documentation.

With regard to specific measurement instruments, we can also attempt to assess quality issues based on question wording or response format. For some inspiration on what to look for, see the GESIS Survey Guidelines on question wording and rating scales.

Consistency between documentation and source datasets

Similarly, we can compare the documentation to the datasets themselves 6. For example, we might consider the degree of consistency between information in the codebook and the values and labels in the dataset as a quality indicator for the survey program.

Source dataset

Of course, the dataset itself is a rich source of quality information 7. We can evaluate the quality of survey data on several levels here. First, on a general level, we can assess the prevalence of non-unique records. When most of the responses are identical, this is most likely a sign of data-processing errors or fabricated cases. Also, data structure aspects can be evaluated, such as checking for ID errors.

Second, on an instrument level, we can assess the quality of specific instruments 8. One basic approach is to explore non-response patterns. Are there instruments with an uncharacteristically high number of non-substantive responses or missing values compared to other instruments in the survey? We should be especially wary of systematic non-response. For example, are voters of a specific party less likely to answer certain questions? If the instruments have multiple items or if they are answered repeatedly by the same respondents in panel waves, we can assess the instruments’ reliability (e.g., as Omega) as an especially powerful quality indicator.

Third, on a respondent level, we can use non-response patterns and perhaps available paradata (e.g., time taken to answer a survey page in a web survey) to identify potentially problematic respondents.

External information on source surveys and source instruments

Lastly, we can make use of external quality information on our source surveys and their instruments. For some survey programs, there is a body of research on their quality aspects. Furthermore, regarding instruments, we might find papers that validate specific instruments. Such validation papers are the norm for psychometric multi-item instruments 9. And while not all single-item instruments are routinely validated, large survey programs perform methodological experiments for specific instruments (e.g., the ESS multitrait-multimethod experiments).

Use of quality controls

Ideally, quality controls allow users of the ex-post harmonized data to mitigate some of this heterogeneity, leading to more robust and less biased substantive analyses 10. Still, the question remains of how to incorporate that quality information into our research. There is no one clear cut way but instead a wide range of possible approaches.

In the SDR project, as with many other comparative research projects, the multilevel modeling (MLM) logic is very pronounced 11. And some quality controls can, indeed, be introduced into such models as control variables. For example, if we assume that a quality factor biases substantive effects, we could introduce a quality control as a moderator (i.e., interaction terms). This allows us to quantify and mitigate some quality influences in one step. However, in MLM analyses (just like with regression in general), multicollinearity can become an issue. For example, if all surveys from one country share a methodological feature that is present in no other country, then we might no longer be able to disentangle country effects from effects due to that methodological feature.

If we have information on the reliability of specific measurement instruments, we can also apply a correction for attenuation 12. Such procedures account for the fact that lower reliability (i.e., more random error in measurement) leads us to underestimate the correlation of the less reliable measure with other variables. Corrections for attenuation estimate the correlation that might have been found if measurement was free of random error. However, they also widen the confidence interval of the correlation to reflect the uncertainty of such an estimate.

Another approach is to weight data from different surveys, instruments, or respondents based on their respective data quality 13. The logic is similar to meta-analyses, which weight coefficients from different studies by the study’s respective sample size. Even more strictly, users might also use controls to outright exclude some surveys or individual data points based on quality controls.

Harmonization controls

Harmonization controls, meanwhile, are usually created during the recoding and combining of source variables into harmonized target variables. They capture structured information on aspects of the source instruments that are no longer (clearly) reflected in the harmonized target variable. Common examples are substantive differences in instrument wording or the conceptual scope of different questions. The SDR project, for example, harmonizes variables on protest participation. In this context, a harmonization control is formed, which captures if the original question also included related but distinct forms of protest, such as strikes 14. Harmonization controls can also be formed to capture instrument design features, such as whether response scales are uni- or bipolar or how many scale points were offered. Lastly, harmonization controls can also capture structured information on specific decisions during the harmonization process. This might mean describing specific decisions or providing additional information from the source variable to enable users to undo some harmonization decisions.

Use of harmonization controls

Many of the uses and approaches of control variables apply here as well. However, the main goal is not to mitigate quality issues of the separate source surveys. Instead, we want to mitigate quality issues that may arise from comparability issues. Harmonization controls help users to empirically quantify the consequences of harmonization choices. In the protest example above, we might ask ourselves if mentioning other forms of protest inflates the number of people agreeing with the question. If so, we can use this information to mitigate the bias. Furthermore, harmonization controls are an elegant, user-friendly way of making the harmonization process readily available during the analysis process. Harmonization controls are certainly a sensible extension of static documentation separate from the dataset.

However, much like with quality controls, the landscape of available survey data is crucial. If all surveys in a country use an extended definition of protest, then it is hard to isolate a suspected bias. Also, please note that harmonization controls might not be able to fully solve the problem of different numerical scales. As we have seen in a previous post, even the same number of response options can result in different numerical scales. However, the control variable approach presented here and approaches such as equating are not mutually exclusive. Instead, they can easily be used in conjunction. In fact, the meticulous charting of survey and instrument structures necessary for the harmonization controls is already an ideal basis for finding suitable data links for equating.

Critical appraisal

Constructing quality and harmonization controls and then using them in analyses with the harmonized dataset is a promising and flexible approach. The result is a living, structured documentation that can be effortlessly accessed during and integrated into the analysis process. The controls by themselves already increase transparency, especially with harmonization controls making harmonization trade-offs and choices clearly visible from within the dataset and not just the accompanying documentation. Furthermore, integrating information on quality and harmonization into our analyses can certainly help in making our substantive results more robust and mitigating methodological heterogeneity.

Of course, the approach does not solve all issues in ex-post harmonization. For example, the issue of numerical comparability cannot be fully solved and has to be addressed with other approaches. Furthermore, especially the usefulness of the controls for multilevel models can be hampered by multicollinearity. What if all surveys of a country favor a certain method or instrument design choice? With such ties, it becomes hard to isolate method effects from substantive differences, such as country-level differences in this example. Where the data is more varied, we still have to take great care of how to include controls into our analyses. Some methodological aspects might not introduce an additive effect but instead interact with (i.e., moderate) other substantive effects. That is most apparent with instrument quality. Low reliability of an instrument means that substantive correlations are underestimated. Hence, there is a quality × substantive correlation interaction.

Yet, those issues should not let us lose sight of the great potential this scalable approach has. Which controls to craft and how many depends on your source material and your research goals (and resources, of course). For some, an elaborate control variable scheme like the one the SDR project uses is ideal. However, other projects might already gain much by including some crucial harmonization information as structured variables.

The finale of season 1

At this point in a post, I usually teaser the next topic. But this time, we have reached the end of this season: Nine posts in total so far! I hope that the posts were of use to some of you 😊. To recharge my creative batteries and to find coauthors willing to share their expertise, the series will be going on a break until fall. Then, we will dive into season 2. Stay tuned!

References

  1. Slomczynski, K. M., & Tomescu-Dubrow, I. (2018). Basic Principles of Survey Data Recycling. In T. P. Johnson, B.-E. Pennell, I. A. L. Stoop, & B. Dorer (Eds.), Advances in Comparative Survey Methods (pp. 937–962). John Wiley & Sons, Inc. https://doi.org/10.1002/9781118884997.ch43
  2. Slomczynski, K. M., & Tomescu-Dubrow, I. (2018). Basic Principles of Survey Data Recycling. In T. P. Johnson, B.-E. Pennell, I. A. L. Stoop, & B. Dorer (Eds.), Advances in Comparative Survey Methods (pp. 937–962). John Wiley & Sons, Inc. https://doi.org/10.1002/9781118884997.ch43
  3. Slomczynski, K. M., & Tomescu-Dubrow, I. (2018). Basic Principles of Survey Data Recycling. In T. P. Johnson, B.-E. Pennell, I. A. L. Stoop, & B. Dorer (Eds.), Advances in Comparative Survey Methods (pp. 937–962). John Wiley & Sons, Inc. https://doi.org/10.1002/9781118884997.ch43
  4. Slomczynski, K. M., & Tomescu-Dubrow, I. (2018). Basic Principles of Survey Data Recycling. In T. P. Johnson, B.-E. Pennell, I. A. L. Stoop, & B. Dorer (Eds.), Advances in Comparative Survey Methods (pp. 937–962). John Wiley & Sons, Inc. https://doi.org/10.1002/9781118884997.ch43
  5. Slomczynski, K. M., & Tomescu-Dubrow, I. (2018). Basic Principles of Survey Data Recycling. In T. P. Johnson, B.-E. Pennell, I. A. L. Stoop, & B. Dorer (Eds.), Advances in Comparative Survey Methods (pp. 937–962). John Wiley & Sons, Inc. https://doi.org/10.1002/9781118884997.ch43
  6. Slomczynski, K. M., & Tomescu-Dubrow, I. (2018). Basic Principles of Survey Data Recycling. In T. P. Johnson, B.-E. Pennell, I. A. L. Stoop, & B. Dorer (Eds.), Advances in Comparative Survey Methods (pp. 937–962). John Wiley & Sons, Inc. https://doi.org/10.1002/9781118884997.ch43
  7. Slomczynski, K. M., & Tomescu-Dubrow, I. (2018). Basic Principles of Survey Data Recycling. In T. P. Johnson, B.-E. Pennell, I. A. L. Stoop, & B. Dorer (Eds.), Advances in Comparative Survey Methods (pp. 937–962). John Wiley & Sons, Inc. https://doi.org/10.1002/9781118884997.ch43
  8. Kołczyńska, M., & Slomczynski, K. M. (2018). Item Metadata as Controls for Ex Post Harmonization of International Survey Projects. In T. P. Johnson, B.-E. Pennell, I. A. L. Stoop, & B. Dorer (Eds.), Advances in Comparative Survey Methods (pp. 1011–1033). John Wiley & Sons, Inc. https://doi.org/10.1002/9781118884997.ch46
  9. Price, L. R. (2017). Psychometric methods: theory into practice. The Guilford Press.
  10. Kołczyńska, M., & Slomczynski, K. M. (2018). Item Metadata as Controls for Ex Post Harmonization of International Survey Projects. In T. P. Johnson, B.-E. Pennell, I. A. L. Stoop, & B. Dorer (Eds.), Advances in Comparative Survey Methods (pp. 1011–1033). John Wiley & Sons, Inc. https://doi.org/10.1002/9781118884997.ch46
  11. Slomczynski, K. M., & Tomescu-Dubrow, I. (2018). Basic Principles of Survey Data Recycling. In T. P. Johnson, B.-E. Pennell, I. A. L. Stoop, & B. Dorer (Eds.), Advances in Comparative Survey Methods (pp. 937–962). John Wiley & Sons, Inc. https://doi.org/10.1002/9781118884997.ch43
  12. Charles, E. P. (2005). The Correction for Attenuation Due to Measurement Error: Clarifying Concepts and Creating Confidence Sets. Psychological Methods, 10(2), 206–226. https://doi.org/10.1037/1082-989X.10.2.206
  13. Slomczynski, K. M., & Tomescu-Dubrow, I. (2018). Basic Principles of Survey Data Recycling. In T. P. Johnson, B.-E. Pennell, I. A. L. Stoop, & B. Dorer (Eds.), Advances in Comparative Survey Methods (pp. 937–962). John Wiley & Sons, Inc. https://doi.org/10.1002/9781118884997.ch43
  14. Kołczyńska, M., & Slomczynski, K. M. (2018). Item Metadata as Controls for Ex Post Harmonization of International Survey Projects. In T. P. Johnson, B.-E. Pennell, I. A. L. Stoop, & B. Dorer (Eds.), Advances in Comparative Survey Methods (pp. 1011–1033). John Wiley & Sons, Inc. https://doi.org/10.1002/9781118884997.ch46
Methods rock Leave a comment

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.