What happens when the fraud detection ensemble gets bigger?

New academic research tested 31 fraud detection methods and found that combinations of signals performed better than individual checks. Our CEO Bob Fawson considers what respondent history could add to the mix in his newest article for Research World.

Researchers have a growing number of ways to evaluate survey respondents. Device intelligence can flag suspicious technical characteristics, while in-survey measures can identify speeding, inconsistent answers or questionable open-ended responses. Other checks look at source quality or patterns associated with known fraud.

An academic paper, “AI-powered fraud and the erosion of online survey integrity: an analysis of 31 fraud detection strategies,” put 31 of these indicators to the test, along with six combinations, or “ensembles,” of them. The combinations performed better.

Bob Fawson writes: “If combining different forms of evidence improves decisions within a project, it is reasonable to ask what those same signals look like across hundreds of projects instead of one.”

One survey can only provide so much information

The fraud detection methods evaluated in the academic research use information available within the surveys being studied. That gives researchers several ways to evaluate a respondent, but the view begins and ends with those projects, but we know that many respondents have much longer histories.

Someone who appears in a survey today may have participated in dozens or even hundreds of studies before it. Those earlier interactions can show whether the behavior researchers are seeing now is unusual for that person or part of a recurring pattern.

As Bob writes, “Some consistently produce thoughtful, reliable data across independent projects, while others accumulate quality concerns that only become apparent when viewed over time.”

Individual quality signals aren’t usually definitive. Speeding is a good example: a fast completion can raise a flag, but researchers have more information to work with if they also know that the respondent has a long history of reliable participation across unrelated studies. The reverse can happen too: someone may make it through the current survey without triggering common checks while having a history of quality concerns elsewhere.

Our own research-on-research found a similar pattern when technical, in-survey and source quality signals were evaluated together. No individual indicator provided the full picture. The academic study adds another body of evidence showing why combinations work better.

History gives the ensemble another input

Most research organizations already create history at the supplier level. Teams compare vendors based on previous projects, service levels, quality performance and other information accumulated over time.

Building history at the respondent level has been harder because an individual organization sees only the portion of that person’s participation that happens within its own research.

We aggregate quality observations across participating organizations, allowing those previous interactions to become part of the information available when a respondent appears again. As Bob explains: “Historical respondent behavior becomes another source of evidence that helps researchers interpret what they are already seeing within the current survey.”

This also means history can tell researchers something about good respondents. Repeated records of thoughtful participation across independent projects provide information just as repeated quality concerns do.

DQC was built to make that broader history available. As an independent data quality clearinghouse, we aggregate quality signals across studies and suppliers, giving participating organizations access to respondent history they could not build from their own projects alone. That information can help identify known poor-quality participants before they enter a study, recognize respondents with a record of reliable participation and add context when other quality signals are inconclusive.

As more organizations contribute data, the shared history grows and each participating organization benefits from observations made elsewhere. The ensemble gets bigger because the information available to evaluate each respondent gets bigger too.

Want to see what cross-study respondent history could add to your existing quality process? Book a demo with us!