Learning and implementing Voxelwise Encoding Models: A guide to reviews, tutorials, and software

Voxelwise Encoding Models (VEMs) are one of the most sensitive methods available for modeling and decoding functional brain activity. In the VEM framework, features are extracted from the stimulus (or task) and used in an encoding model to predict brain activity. If the encoding model is able to predict brain activity in some part of the brain, then one may conclude that some information represented in the features is also represented in the brain.

In VEM, a separate encoding model is fitted on each spatial sample (i.e., each voxel). VEM has many benefits compared to other methods for analyzing and modeling neuroimaging data. Most importantly, VEM can use large numbers of features simultaneously, which enables the analysis of complex naturalistic stimuli and tasks. Therefore, VEM can produce high-dimensional functional maps that reflect the selectivity of each voxel to large numbers of features. Moreover, because model performance is estimated on a separate test dataset not used during fitting, VEM minimizes overfitting and inflated Type I error confounds that plague other approaches, and the results of VEM generalize to new subjects and new stimuli.

Because the VEM framework is based on relatively advanced methods from statistics and data science, VEM has not yet been adopted broadly in the field. Several years ago we set out to change that by creating open source software that implements the framework, along with tutorials and reviews that guide researchers in its use an implementation. We’ve now completed this effort, as summarized below. The first section of this blog summarizes the foundational papers that give an overview of VEM and its history. The second section summarizes the VEM tutorial. The third section describes the various pieces of software that implement VEM.

Note that detailed information on all of this software and accompanying tutorials and papers, as well as appropriate links, can be found on the Learn and Code pages of our web site.

Background papers

Review of Voxelwise Encoding Models: This review paper provides the first comprehensive guide to the Voxelwise Encoding Model (VEM) framework. The VEM framework is a complete pipeline for fitting encoding models to fMRI data. This framework is currently the most sensitive and powerful approach available for modeling and decoding fMRI data. It can be used to fit dozens of distinct models simultaneously, each model having up to several thousand distinct features. The Voxelwise Encoding Model framework also conforms to all best practices in data science, which maximizes sensitivity, reliability and generalizability of the resulting models. Although VEM is optimized for fMRI, its roots go back to the system identification approach in neurophysiology, and so the methods can also be applied to encoding models fit to other brain measurements. (Visconti di Oleggio Castello, M., Deniz, F., Dupré la Tour, T., & Gallant, J. L. Encoding models in functional magnetic resonance imaging: the Voxelwise Encoding Model framework. Currently in review as of Dec. 2025 but available as a preprint.)

Older review of system identification in neurophysiology: The VEM framework grew out of a prior approach used in neurophysiology called System Identification. That method aimed to fit encoding models to neurophysiology data, particularly sensory data. System identification provides a clear set of guidelines for combining experimental data with other knowledge about sensory function to obtain a description that optimally predicts the way that neurons process sensory information. This prediction paradigm provides an objective method for evaluating and comparing computational models. This paper provides a review of many of the system identification algorithms that have been used in sensory neurophysiology, and shows how they can be viewed as variants of a single statistical inference problem. The paper also reviews many of the practical issues that arise when applying these methods to neurophysiological experiments: stimulus selection, behavioral control, model visualization, and validation. Finally, the paper discusses several problems to which system identification has been applied recently, including one important long-term goal of sensory neuroscience: developing models of sensory systems that accurately predict neuronal responses under completely natural conditions. Reading this older paper will give you a stronger foundation in VEM and its proper historical context in neurophysiology. (Wu, M. C. K., David, S. V., & Gallant, J. L. (2006). Complete functional characterization of sensory neurons by system identification. Annu. Rev. Neurosci., 29(1), 477-505.)

Explanation of the mathematical algorithms and software for fitting encoding models in VEM: This paper describes the quantitative and theoretical foundation of Himalaya, the banded ridge regression software that forms the computational foundation for VEM. Himalaya is routinely used in our lab to fit voxelwise encoding models to very large fMRI data sets. (Dupré la Tour, T., Eickenberg, M., Nunez-Elizalde, A. O., & Gallant, J. L. (2022). Feature-space selection with banded ridge regression. NeuroImage, 264, 119728.)

Theoretical basis of the banded ridge (a.k.a. Tikhonov) regression algorithm that underpins VEM: Predictive models for neural or fMRI data are often fit using regression methods that employ priors on the model parameters. One widely used method is ridge regression, which employs a spherical multivariate normal prior that assumes equal and independent variance for all parameters. However, a spherical prior is not always optimal. Expert knowledge or hypotheses about the structure of the model parameters can often be used to construct a better prior. In these cases, non-spherical multivariate normal priors can be employed using a generalized form of ridge regression known as Tikhonov regression. This paper explains the theoretical basis for Tikhonov regression, demonstrate a computationally efficient method for its application, and show several examples of how Tikhonov regression can improve predictive models for fMRI data. (Nunez-Elizalde, A. O., Huth, A. G., & Gallant, J. L. (2019). Voxelwise encoding models with non-spherical multivariate normal priors. Neuroimage, 197, 482-492.)

Tutorials

Voxelwise Encoding Model tutorials: To demystify the VEM framework and ease its dissemination, this paper presents a series of hands-on tutorials accessible to novice practitioners. The tutorials walk the reader through the VEM pipeline, step-by-step. Readers begin by defining features, then fit encoding models, and finally visualize the data on flattened cortical maps. The VEM tutorials are based on free open-source tools and public datasets, and reproduce the analysis presented in previously published work. If you are new to VEM and you want to get going quickly, this tutorial is the place to start. (Dupré la Tour, T., Visconti di Oleggio Castello, M., & Gallant, J. L. (2025). The Voxelwise Encoding Model framework: a tutorial introduction to fitting encoding models to fMRI data. Imaging Neuroscience, 3, imag_a_00575.)

Software

Himalaya software package: Himalaya is the computational core of the VEM framework. This Python package implements machine learning linear(ized) models, focusing on computational efficiency for large numbers of targets. Himalaya efficiently estimates encoding models on large numbers of targets (for example, thousands of voxels in an fMRI experiment), it runs on both CPU and GPU hardware, and it provides estimators that are compatible with scikit-learn’s API. Himalaya is routinely used in our lab to fit voxelwise encoding models to very large fMRI data sets. (Dupré la Tour, T., Eickenberg, M., Nunez-Elizalde, A. O., & Gallant, J. L. (2022). Feature-space selection with banded ridge regression. NeuroImage, 264, 119728.)

Pycortex: When VEM is used to analyze the results of complex, naturalistic fMRI experiments, it produces very rich, high-dimensional data sets that cannot be visualized or interpreted easily using conventional methods. To address this problem we borrowed a method that was first developed by neuroanatomists Jaime Oliveira and David Van Essen: cortical flattening. The Pycortex software package is a python-based toolkit for surface visualization of fMRI data that deals correctly with the complications that inevitably arise when converting from the volumetric fMRI signals to a surface-based representation. The Pycortex software underlies the many brain viewers that you can find on our web site, and it is also used in other systems (e.g., Neurovault) and by other groups. (Gao, J. S., Huth, A. G., Lescroart, M. D., & Gallant, J. L. (2015). Pycortex: an interactive surface visualizer for fMRI. Frontiers in neuroinformatics, 9, 23.)

Autoflatten: One of the most difficult parts of cortical flattening is to move from the original volumetric anatomical brain to a flattened surface. This procedure proceeds in several steps. The brain is computationally removed from the skull. Then a few cuts are placed strategically around the edges of the brain and the whole hemisphere is flattened in a way that minimizes distortion. Until just a few weeks ago this process had to be performed largely by hand. However, our postdoc Dr. Matteo Visconti di Oleggio Castello has now created a new software package that automates this procedure!

None of this work would have been possible without the selfless contributions of various graduate students and postdocs in our lab. In particular I want to highlight the contributions of the following people: Dr. James Gao and Prof. Alex Huth for development of the Pycortex software; Dr. Tom Dupré la Tour for the development of Himalaya; Dr. Matteo Visconti di Oleggio Castello for the development of autoflatten; Dr. Anwar Nunez-Elizalde and Prof. Alex Huth for the theoretical development of Tikhonov regression for VEM; Dr. Tom Dupré la Tour and Dr. Matteo Visconti di Oleggio Castello for the VEM tutorials; and Prof. Fatma Deniz and Dr. Matteo Visconti di Oleggio Castello for the VEM review.