A crossover coefficient for quantifying disagreement in product ranks between individual panelists and the panel
To quantify rank order differences between panelist j and a sensory panel, we propose a univariate crossover coefficient (XCj) which takes values between 0 and 100. Higher values indicate greater crossover.
Panel data are analysed. An appropriate multiple comparison test is used to separate k products into homologous subsets. The panel’s product rankings becomes the standard against which each panelist’s product rankings are evaluated. Panelist j evaluates q of k products; 1 ≤ q ≤ k, where q = k in a complete block design without missing data. Dj is the number of statistically different pairs and omits any pair not discriminated by the panel; 0 ≤ Dj ≤ q(q-1)/2. If all Dj = 0 then the panel is non-discriminating at the required level of significance and crossover is not considered meaningful.
Nj incorporates the pair inversions for panellist j, penalizing only crossovers for statistically significant product pairs. Crossover is quantified XCj = 100 * Nj / Dj. XC obtained from random panel rankings contextualize the XCj values.
The procedure is nonparametric if the panel data are analysed nonparametrically, and semiparametric if the panel data are analysed parametrically. Unlike most conventional approaches, XC isolates ranking disagreement from scale usage differences. The XC procedure can be extended to incorporate penalties based on the sum of squares of deviances for relevant pairs rather than the sum of absolute deviances. XC can be evaluated on a leave-one-out basis, i.e. contrasting j with the other panellists. Panel-to-panel or other comparisons are also possible. Differences with other rank agreement statistics, e.g. Kendall’s W, are noted.
We illustrate the procedure using results from a trained descriptive panel (n=12) which developed a lexicon of 29 sensory attributes to evaluate 12 noodle products which either sold in or being developed for the Chinese market.
Castura, J. C., Truong, M., & Findlay, C. J. (2016). A crossover coefficient for quantifying disagreement in product ranks between individual panelists and the panel. In: SenseAsia 2016: 2nd Asian Sensory and Consumer Research Symposium. 15-17 May. Shanghai, China.