Best practice recommendations for attribute order in Check-All-That-Apply (CATA) and related test methodologies
It is well documented that the position of attributes in a Check-All-That-Apply (CATA) question can bias responses. As positional biases cannot be eliminated, they are balanced across products via experimental designs, ensuring each attribute appears with equal frequency in each position for each product. But what is the best way to allocate attribute list orders?
One approach involves allocating attribute orders to assessors, such that each assessor has a particular fixed attribute list, but different assessors have different list orders. Another approach involves allocating attribute orders to samples, such that a particular assessor gets a different attribute list for each sample. For statistical reasons, the hypothesis was that the first approach should give higher operational power.
These two approaches were evaluated in a consumer bread study (n=93) conducted at Compusense Inc. The ballot included a CATA question with 32 sensory attributes. About half the consumers evaluated 6 breads in a Williams modified Latin square design, all with the “to assessor” allocation, then after a longer delay, evaluated the same 6 breads in a different Williams modified Latin square design, all with the “to sample” allocation. The other half of consumers had the same ballot, but received a “to sample” allocation first, followed by the “to assessor” allocation. Appropriate care was taken to avoid confounding effects by the intersection of two experimental designs.
Data analyses were conducted using Cochran’s Q test and an exact McNemar’s test, with appropriate adjustments made for (slightly) unequal sample sizes for the two groups. Data from the two list order allocations produced very similar sensory profiles, but the “to assessor” allocation indeed had superior operational power. The recommendation to allocate attribute orders to assessors also extends provisionally to TDS and TCATA, which are subject to the same positional effects.
Castura, J. C., & Meyners, M. (2015). Best practice recommendations for attribute order in Check-All-That-Apply (CATA) and related test methodologies. In: 11th Pangborn Sensory Science Symposium. 23-27 August. Gothenburg, Sweden.