Our aim and perspective

The mammalian cerebral cortex is a multi-scale biological computing device consisting of billions of neurons, arranged in layered, local circuits. These local circuits are arranged into columns, and groups of columns form an area. Connections between neurons within a local circuit, column and area are both convergent (projections within structures) and divergent (projections between structures). Both feed-forward and feed-back connections are typical, and information may flow over multiple routes to reach the same target. The result is a hierarchical, parallel, highly interconnected network of areas that tile the cerebral cortex.

The information that is represented in each cortical area reflects a nonlinear sum of the information represented in all areas that provide feed-forward or feed-back input to the area. Thus, each area represents information explicitly that is only implicit in the input. This property suggests that it should be possible to understand brain computation by first discovering cortical areas, and then determining what specific information is mapped across each area. The central goal of our research program is to discover how the mammalian brain represents information about the world and about its own mental states, by identifying and characterizing these cortical maps.

To address this problem, our laboratory makes heavy use of an inductive scientific approach called system identification. System identification is a systematic approach for discovering the computational principles of an unknown system such as the brain. Most of the data collected in our lab comes from functional magnetic resonance imaging (fMRI) studies of the human brain. In a typical experiment these data are collected under very general conditions: while subjects watch movies, listen to natural sounds and so on. When necessary, we supplement this approach with targeted experiments that are optimized to test very specific hypotheses about brain function.

Once the data are collected, we use classical statistical tools, Bayesian analysis and machine learning approaches to fit computational models to the brain data. (These are usually called “encoding models”, because they describe how information about the world is encoded in the brain.) We usually evaluate several different, competing models in order to find the one model that most accurately predicts responses, using a separate data set that was not used to estimate the model. The resulting encoding models describe what specific information is represented explicitly at each location across the cerebral cortex.

Computational encoding models that accurately predict brain activity are the gold standard of systems and cognitive neuroscience. However, these models also have many practical uses. They provide a new tool for neurological evaluation and diagnosis. They also provide a critical foundation for developing therapies to repair brain damage (after all, one needs to understand how a system functions before one can hope to repair it). Finally, computational encoding models can be inverted by means of a Bayesian framework, in order to decode brain activity. This provides a direct and principled way to do “brain decoding”, and to build brain-machine interfaces (BMI) and neural prosthetics.


The experiments in our laboratory generate large, rich data sets that cannot be summarized in a single number. Therefore, we use and develop many statistical tools for “big data” problems, and we develop software optimized for visualizing our results. Most notably, we developed the open source pycortex software, the most sophisticated platform for visualizing functional brain data. (You can find the paper describing pycortex here.) The online brain viewer provides several demonstrations of this software.

Brain Decoding

Some of our side projects focus on brain decoding. Many people are interested in brain decoding, so we answer common questions about this topic here.

What is the goal of brain decoding?

There are two main reasons that scientists conduct research on brain decoding. First, many scientists and engineers seek to develop brain-machine or brain-computer interfaces that can be used to solve problems in the real world. For example, many different labs are trying to develop brain-machine interfaces that can be used to control prosthetic limbs. However, practical brain decoding devices would have wide applicability in many other fields, including in communication, medical diagnosis and entertainment. The second motivation for brain decoding research is that it can serve as a useful adjunct to basic research in cognitive neuroscience. One way to test whether some part of the brain contains a specific type of information is to see whether that information can be decoded from brain activity.

What types of brain decoding are currently possible?

It is useful to distinguish between three types of brain decoding: classification, identification and reconstruction. In classification we try to determine which one of several possible brain states a person is in. For example, one can try to classify whether a person has seen a horizontal or a vertical grating. In identification we try to determine which specific stimulus a subject has seen, when the stimulus has been drawn from a list of known stimuli. For example, imagine that someone has drawn a photograph at random from a large stack of photos. If we record this person’s brain activity while they view the photo, can we search through the entire stack of photos and guess which one they saw? Reconstruction is the most ambitious form of brain decoding. In reconstruction we try to reconstruct the stimulus that a person saw while they were being scanned, even if the stimulus is completely novel and has never been seen before. For example, if someone watched a movie while in the scanner, we could try to recreate the movie that they were watching. All three of these types of brain decoding have been demonstrated in using visual stimuli such as gratings, geometric patterns and natural images.

What equipment is required for brain decoding?

At this time non-invasive approaches to brain decoding include functional MRI (fMRI), magneto-encephelography (MEG), electro-encephelography (EEG) or near infra-red spectroscopy (NIRS). However, these techniques vary substantially in the quality of results that can be achieved. Functional MRI can recover a wide range of information from the brain, MEG somewhat less, EEG relatively little and NIRS still less. Although fMRI provides the best non-invasive measures of brain activity (and so the best decoding), it is an extremely expensive technique that requires a very large magnet housed in a shielded room. Therefore, future brain decoding devices that are both portable and non-invasive will be based on other technologies, such as MEG. Note that in theory brain decoding could be performed invasively by placing electrodes between the skull and the surface of the brain, or by inserting them directly into the brain. However, these invasive techniques are unlikely to be used except in rare medical situations where an invasive surgery might provide significant improvement in life quality. Therefore, much current work focuses on non-invasive techniques for brain decoding that are likely to have wider applicability.

How does a typical brain decoding experiment work?

As noted elsewhere in this FAQ there are several different approaches to brain decoding, and the procedures differ. Our laboratory typically uses a two step procedure. The first step of the process is to construct a set of encoding models that describes how visual stimuli are represented in the pattern of activity across visual cortex. The assumption here is that the activity in visual cortex is systematically related to the particular visual stimulus that is being viewed at any point in time. The goal is to build encoding models that can take any visual stimulus and predict the pattern of activity that the image would produce in visual cortex. The second step of the process is to use these encoding models to solve the image reconstruction problem. One way to describe how we approach this problem is to think of all possible images as potential reconstructions. To determine if a particular candidate is an accurate reconstruction, we use the encoding models to determine the likelihood that it would evoke the brain activity we actually measured. We also use prior information about what natural images look like to determine if the candidate is an accurate reconstruction. Both of these sources of information are used to converge on a reconstruction that is both structurally and semantically accurate.

What are the limitations on brain decoding?

Decoding performance depends on the quality of brain activity measurements. At this time the best way to measure brain activity is with functional MRI (fMRI). However, fMRI does not actually measure the activity of neurons. Instead, it measures blood flow consequent to neural activity. Many studies have shown that the blood flow signals measured using fMRI are generally correlated with neural activity. However, fMRI has relatively modest spatial and temporal resolution, so much of the information contained in the underlying neural activity is lost when using this technique. fMRI measurements are also quite variable from trial to trial. Both of these factors limit the amount of information that can be decoded from fMRI measurements. Decoding also depends critically on our understanding of how the brain represents information, because this will determine the quality of the computational model. If the encoding model is poor (i.e., if it does a poor job of prediction) then the decoder will be inaccurate. While our computational models of some cortical visual areas perform well, they do not perform well when used to decode activity in other parts of the brain. A better understanding of the processing that occurs in parts of the brain that mediate more abstract cognitive processes will be required before it will be possible to decode other aspects of human experience.

What are the future applications of this technology?

There are many potential applications of devices that can decode brain activity. In addition to its value as a basic research tool, it could be used to aid in diagnosis of diseases (e.g., stroke, dementia); to assess the effects of therapeutic interventions (drug therapy, stem cell therapy); or as the computational heart of a neural prosthesis. It could also be used to build a brain-machine interface, as described below.

At some later date when the technology is developed further, will it be possible to decode dreams, memory, and visual imagery?

Neuroscientists generally assume that all mental processes have a concrete neurobiological basis. Under this assumption, as long as we have good measurements of brain activity and good computational models of the brain, it should be possible in principle to decode the visual content of mental processes like dreams, memory, and imagery. However, current computational models of visual processing have been developed to account for visual perception of natural scenes. The accuracy of these models for decoding subjective states such as dreaming and imagination will depend on how similar those processes are to normal visual perception. This is an active topic of research in our lab and in many other labs.

At some later date when brain decoding technology is developed further, will it be possible to use this technology in detective work, court cases, trials, etc?

The potential use of this technology in the legal system is problematic. Many psychology studies have demonstrated that eyewitness testimony is notoriously unreliable. Witnesses often have poor memory but often do not realize that their memory is poor; their recollections tend to be biased by intervening events, inadvertent coaching, and rehearsal (prior recall); and they often confabulate stories to make logical sense of events that they cannot recall well. These errors are thought to stem from several factors: poor initial storage of information in memory; changes to stored memories over time; and faulty recall. Any brain decoding device that aims to decode stored memories will inevitably be limited not only by the technology itself, but also by the quality of the stored information (an accurate read-out of a faulty memory only provides misleading information). Therefore, any future application of this technology in the legal system will have to be considered with extreme caution.

Will we ever be able to use this technology to insert images (or movies) directly into the brain?

Not in the foreseeable future. In our experiments brain activity was measured using fMRI, which is only indirectly coupled to neural activity. There is no known technology that can remotely send signals to the brain in a way that would be organized enough to elicit a meaningful mental state.

Can brain decoding be performed remotely or covertly, without an individual’s knowledge?

No, brain decoding cannot be performed remotely, and it cannot be performed without the knowledge of the subject. Brain decoding requires that brain activity be measured at fairly high spatial and temporal resolution. At this time there are no brain decoding technologies that can record brain activity remotely at a resolution sufficient to decode mental activity. All successful brain decoding experiments use devices such as fMRI machines, MEG machines or EEG recording arrays that must be placed in close proximity to the brain. These devices are quite large (for example, an fMRI machine weighs several tons), and they would certainly be noticed if they were being used. Furthermore, any small movements such as head motion, blinking or coughing will dramatically reduce the quality of the brain activity measurements. Therefore, all these technologies require the cooperation of the individual whose brain activity is being measured.

What are the ethical/privacy issues that stem from brain decoding?

Current methods for decoding brain activity are relatively primitive. Current computational models are immature and in order to construct a model of someone’s visual system they must spend many hours in a large, stationary magnetic resonance scanner. For this reason it is unlikely that brain decoding technology could be used in practical applications any time soon. That said, both the technology for measuring brain activity and the computational models are improving continuously. It is possible that decoding brain activity could have serious ethical and privacy implications downstream in, say, the 30- to 50-year time frame. As an analogy, consider the current debates regarding availability of genetic information. Genetic sequencing is becoming cheaper by the year, and it will soon be possible for everyone to have their own genome sequenced. This raises many issues regarding privacy and the accessibility of individual genetic information. We believe strongly that no one should ever be subjected to any form of brain-decoding process involuntarily, covertly, or without complete informed consent.