Screening as a tool to increase consumer data quality
We developed and tested a strategy to improve data quality in consumer tests by dropping consumers based on their screener responses. A key question was whether test outcomes were improved enough to justify the extra effort of having this screening step. To do this investigation, we made an online test in which respondents answered a short screener followed by a concept test where respondents gave their opinions on three concepts for crackers, where each one emphasized a particular benefit: health, sustainability, or deliciousness. The screener contained category agree/disagree, check-all-that-apply, and open comment questions that were posed in a way that allowed us to determine whether answers were dishonest, contradictory, incomplete, or dubious. We had a penalty system that assigned points to bad answers. A pass/fail threshold was set; respondents with too many points “failed”.
The ballot was completed by naïve respondents in Finnish (n=343) and Turkish (n=342). Respondents who “passed” the screener (259 Finns; 191 Turks) gave better-quality test results and stronger directional differences between concepts without compromising concept discrimination. When we simulated a large number of smaller, fixed-size panels using only consumers who pass the screener vs. all consumers, then again we find the panels composed by screener-passing respondents to be more discriminating and larger differences between concepts. Between-country differences were observed. Respondents who were familiar with sensory evaluation were instructed to give bad data deliberately according to one of these instructions: (i) to please the researcher (n=47), (ii) as fast as possible (n=59), and (iii) inattentively (n=63). Our penalty system was most effective at identifying ”pleasers”. Provisionally, we recommend integrating data quality checks into the consumer selection criteria. Further study is needed to investigate other cross-cultural differences and to develop best practices in screening.
Castura, J.C., Pohjanheimo, T., McEwan, J.A., & Varela, P. (2022). Screening as a tool to increase consumer data quality. 12th European Conference on Sensory and Consumer Research, 13-16 September. Turku, Finland. (Poster).