Respondent quality beyond a single survey

Respondent quality decisions are usually made using what researchers can see within a survey. Completion speed, open-end responses, consistency, device information and other checks can all tell researchers something about the person taking part.

But respondents often have a much longer history than any one project can capture. Our CEO Bob Fawson explores what researchers can learn from that history in a recent article for Quirk’s.

“When participation history is available, researchers can evaluate patterns that develop over months rather than minutes,” Bob writes.

Someone who appears unremarkable in one study may have successfully completed dozens of others. Another respondent may repeatedly trigger quality concerns that are difficult to recognize when each survey is considered separately.

“Looking across a larger body of activity makes it easier to distinguish isolated incidents from recurring behavior.”

Individual checks often see different things

Our own research-on-research combined technical fraud indicators, in-survey behavior and source-level quality signals. The measures didn’t simply identify the same respondents over and over.

As Bob explains, “No single measure explained respondent quality on its own. Device checks identified some issues, survey behavior identified others and participation history added another layer of context.”

We continue to see organizations use different combinations of identity verification, technical fraud detection, behavioral monitoring and other measures. A respondent can also look different depending on which part of their activity is being examined.

Participation history gives researchers information from outside the current project. A fast completion, for example, can be considered alongside how that person has performed in previous studies. A respondent who passes the checks in the current survey may have a repeated history of problems elsewhere.

Frequent participation can look different with history

Cross-study data has also given us a closer look at people who participate in research frequently.

In our research-on-research, we identified a segment we called professional panelists. They appeared frequently, generally came from reliable panel sources and produced consistent responses in the project.

Bob writes, “That finding was useful precisely because it ran counter to the easy assumption that frequent participation is always a quality problem.”

Frequency by itself says relatively little about how someone participates. Seeing that same person perform well across unrelated projects and suppliers provides evidence that a single participation count cannot.

This is one area where having more history has changed the questions we can ask. DQC can identify respondents who have repeatedly generated quality concerns, but we can also see people who consistently complete research without those concerns.

“As more participation history becomes available, those respondents are easier to identify,” Bob writes.

More respondent history is now available

No individual research organization sees everything a respondent does. A sample provider sees activity within its own sources. An agency sees the studies it fields. A buyer sees the research conducted on its behalf.

DQC brings quality observations from participating organizations together so that previous activity can be considered when a respondent appears again. The record grows as that person participates across studies and suppliers.

Bob ends his article with several questions for research teams, including one we think deserves more attention:

“How much attention is devoted to identifying trustworthy respondents compared with identifying respondents who should be removed?”

Read Bob’s full article in Quirk’s: https://www.quirks.com/articles/some-respondent-quality-patterns-are-easier-to-see

Want to see how cross-study respondent history works in practice? Book a demo with us.