Stimulus-specific random effects inflate false-positive classification accuracy in multivariate-voxel-pattern-analysis: A solution with generalized mixed-effects modelling

When conducting multivariate-voxel pattern analysis (MVPA), researchers typically compute the average accuracy for each subject and statistically test if the average accuracy is different from the chance level across subjects (by-subject analysis). We argue that this traditional by-subject analysis leads to inflated Type-1 error rates, regardless of the type of machine learning method used (e.g., support vector machine). This is because by-subject analysis does not consider the variance attributed to the idiosyncratic features of the stimuli that have a common influence on all subjects (i.e., the random stimulus effect). As a solution, we proposed the use of generalized linear mixed-effects modelling to evaluate average accuracy. This method only requires post-classification data (i.e., it does not consider the type of classification methods used) and is easily implemented in the analysis pipeline with common statistical software (SPSS, R, Python, etc.). Using both statistical simulation and real fMRI data analysis, we demonstrated that the traditional by-subject method indeed increases Type-1 error rates to a considerable degree, while generalized mixed-effects modelling that incorporates random stimulus effects can indeed maintain the nominal Type-1 error rates.