Natural speech reveals the semantic maps that tile human cerebral cortex

Alexander G. Huth, Wendy A. de Heer, Thomas L. Griffiths, Frederic E. Theunissen & Jack L. Gallant (Nature, 2016)

You can find a link to this paper at the Nature web site  here.

Unfortunately this paper is currently paywalled at Nature. To receive a copy, please send an email to: <huthreprint@gmail.com>.

The meaning of language is represented in regions of the cerebral cortex known collectively as the “semantic system”. However, little of the semantic system has been mapped comprehensively, and the semantic selectivity of most regions is still unknown. Here we systematically map semantic selectivity across the cerebral cortex using voxel-wise modeling of fMRI data collected while subjects listened to several hours of natural narrative stories. We show that the semantic system is organized into intricate patterns that appear highly consistent across individuals. We then use a novel Bayesian generative model to map these patterns and create a detailed semantic atlas. Our results suggest that most areas within the semantic system represent information about specific semantic domains and our atlas shows which domains are represented in each area.


Video explanation of the paper

This nice, brief video overview was a collaboration between the folks at Nature and the first author, Alex Huth. To receive a copy of the paper, please send an email to: <huthreprint@gmail.com>.


Frequently asked questions about this work

What was the purpose of this study?

Our goal in this study was to map how the brain represents the meaning (or “semantic content”) of language. Most earlier studies of language in the brain have used isolated words or sentences. We used natural, narrative story stimuli because we wanted to map the full range of semantic concepts in a single study. This made it possible for us to construct a semantic map for each individual, which shows which brain areas respond to words with similar meaning or semantic content. Another aim of this study was to create a semantic atlas by combining data from multiple subjects, showing which brain areas represented similar information across subjects.

What were the most innovative aspects of this experiment?

  • We took a data driven approach rather than the conventional method of point-null hypothesis testing.
  • We used natural narrative language rather than simple words presented in isolation.
  • We used cross-validation on a separate dat a set to test model prediction performance and generalization.
  • We used voxel-wise modeling to construct a separate semantic tuning curve for each voxel in each participant.
  • We used a probabilistic and generative model of areas tiling cortex (PrAGMATiC)  to construct the semantic atlas.

What were the main conclusions of the study?

Our study did not set out to test a single hypothesis or to ask a simple question. Instead, we sought to exhaustively map the representation of the meaning, or semantic information, in narrative language, across the entire cerebral cortex. The resulting maps show that semantic information is represented in rich patterns that are distributed across several broad regions of cortex. Furthermore, each of these regions contains many distinct areas that are selective for particular types of semantic information, such as people, numbers, visual properties, or places. We also found that these cortical maps are quite similar across people, even down to relatively small details.

How was the experiment conducted?

Study participants listened passively to several stories selected from The Moth Radio Hour while brain activity was monitored using functional magnetic resonance imaging (fMRI). The stories were then transcribed and annotated with the time each word was spoken. Then we used the fMRI data and story transcripts to build computational models that predict brain activity as a function of which words the subject heard. To validate these models, they were used to predict fMRI responses to a new story that had not been used before. We found that the models were able to predict responses relatively well throughout several broad regions of the cerebral cortex.

Next, we aimed to discover what types of semantic information were represented at each point in cortex. In order to visualize the very high-dimensional semantic models, we used a dimensionality reduction technique called principal components analysis (PCA). PCA finds the most important dimensions in a dataset, which allowed us to reduce the 985-dimensional models to only three dimensions, while preserving as much information as possible. We used these three dimensions to visualize roughly which types of semantic information were represented at every location in the cortex, revealing complex semantic maps that tile the brain.

Finally, to discover which aspects of these maps are shared across subjects we developed and applied a new computational approach called PrAGMATiC, a probabilistic and generative model of areas tiling the cortex. This approach finds functional areas that are shared across subjects, while also allowing for individual variability in the anatomical location of each area.

How does this study change the way that we think about language and the brain?

These semantic maps give us, for the first time, a detailed map of how meaning is represented across the human cortex. Rather than being limited to a few brain areas, we find that language engages very broad regions of the brain. We also find that these representations are highly bilateral: responses in the right cerebral hemisphere are about as large and as varied as responses in the left hemisphere. This challenges the current dogma (inherited from studies of language production, as opposed to language comprehension as studied here) holding that language involves only the left hemisphere.

How similar or different are the semantic maps across different individuals?

Our study found that different individuals have remarkably similar semantic maps. For example, in the lateral parietal cortex, all seven of our subjects showed an area selective for words related to people. That area is surrounded by areas selective for visual words, tactile words, and number words. Our PrAGMATiC atlas was designed to capture these similarities. While there are small differences in the exact anatomical location of these semantic areas across individuals, their relative positions are very similar. In future studies we hope to study how experience, native language, and culture may relate to more fine-grained individual differences in these maps.

How will this study further the progress of science or medicine?

This work provides a first comprehensive look at how the meaning, or semantic content, of language is represented across the cortex. These results suggest that selectivity for particular semantic domains—groups of related concepts such as people or numbers—is a basic organizing principle for how the brain represents the meaning of language.

Another important contribution of our study is the development of data-driven methods sufficiently sensitive to recover detailed semantic maps. This approach allowed us to efficiently map the representations of many different types of semantic information. Our approach can also be used to map other kinds of language-related information, such as phonemic information, syntactic information and narrative.

Our work also has implications for studying how people recover from brain injuries including stroke, which may affect language related areas. For example, unaffected semantic areas may provide compensatory mechanisms for the brain to rewire after injury. Future studies will clarify whether and how these maps change in language disorders such as dyslexia or delayed language learning, or in neurological conditions related to social language processing such as autism.

What does fMRI measure and how can the signals be interpreted?

Functional MRI (fMRI) does not measure neural activity directly. It measures changes in blood oxygenation, blood flow and blood volume that are caused by neural activity. Most of the useful fMRI signal comes from veins about the size of the sampling lattice (in this study, volumes of about about 2x2x4 mm). Given this, the only safe assumption is that fMRI signals are monotonically related to the integrated synaptic activity of the local neuropil upstream from the site of measurement. Thus, fMRI signals likely reflect both excitatory and inhibitory synaptic activity of both feed-forward and feed-back connections. That said, many studies have validated fMRI by showing that fMRI signals are closely correlated with local field potentials (LFPs) and local multi-unit activity (MUA), which are themselves closely associated with the activity of excitatory neurons. (Perhaps this is because local inhibitory influences in cortex appear to be relatively untuned, when compared to excitatory signals.) Furthermore, studies of the primate analog of the fusiform face area (FFA) have shown that tuning of single excitatory neurons and of voxels are similar. That said, it is prudent to remain agnostic about the precise relationship between fMRI signals and neural signals.

Is it possible to determine in what order different regions of cortex were activated?

Not from these results. Functional MRI only measures slow signals related to blood oxygenation, blood volume and blood flow. These blood-oxygen-level-dependent signals signals are much slower than the underlying neural signals. So it is very very difficult to try to determine the order in which areas are activated using fMRI.

Why are data shown on the surface of a flattened cortex?

The cerebral cortex is a sheet of tissue that overlies older mid-brain structures. In the normal human brain the cortex is highly folded. (This ensures that the head remains small even as the brain grows larger.) Unfortunately, these folds make it difficult to understand how information is mapped across the cortical sheet. Therefore, to better visualize semantic maps we computationally inflate the cortex and flatten it out. The functional data are then projected onto the surface of the flattened cortex, where it can be visualized easily. (Note that most fMRI studies show results in the 3D brain space. They do not flatten the cortex and they do not show flat maps. This makes the data produced in other studies difficult to see and interpret. Given that most fMRI studies are focused on cortical activity, we believe that the results of these other studies would be easier to understand if they made flat maps as we did here.)

This study only used a few subjects. What are the advantages and disadvantages of this?

Most fMRI studies group the data from different subjects together, and only make inferences at the group level. Grouping this way averages out much of the detailed information available in the cortical maps and it ignores individual variability. To maximize sensitivity our voxel-wise modeling approach analyzes the data recorded from each subject individually. We then use a generative statistical model, PrAGMATiC (Probabilistic and Generative Model of Areas Tiling Cortex), to recover a semantic atlas that describes brain areas that have common semantic selectivity across all individual subjects.

Is it appropriate to generalize the results of this experiment given that there were only seven subjects?

The methods used in most fMRI studies are rooted in experimental psychology. These studies average over large numbers of people in order to minimize potential biases due to individual variability. However, this approach comes at a cost. The need to collect data from many people means that relatively little data is acquired from each individual person. And averaging the results across individuals tends to smooth out the data and erase fine details that may be important. As a result, the smooth functional maps shown in most fMRI studies look nothing like the data from any individual person. Our experiment is rooted in an older approach to behavioral science called psychophysics. A typical psychophysics experiment focuses on collecting large data sets from a few individuals. The results obtained from each person are then compared and contrasted (rather than averaged) in order to identify important features in the data. One important innovation of our experiment is the development and application of the PrAGMATiC algorithm, which provides a quantifiable and principled statistical method for extracting these common features without the need for averaging the data across individuals.

Would the semantic maps be different in individuals from another culture?

It is possible that the semantic maps would vary between individuals that grew up in very different cultures, or who have a different native language. Further research will be required to determine what if any aspects of these maps depend on culture or language.

Why are results measured in terms of predictions rather than significance?

Most fMRI experiments focus on statistical significance. Statistical significance is important because it tells scientists that the thing that is being measured is not just random noise. However, a more rigorous and important criterion for any theory in science is its ability to predict the future. Because we seek to create a model that provides accurate predictions of brain activity under all possible conditions, we focus on predictive power rather than mere significance. To generate and test predictions we collect two different sets of data. The first set is used to estimate a semantic model for each voxel. The second set is used to validate the models by testing predictions of activity for each voxel.

Where is this research going?

Human language is a complex signal that contains many different types of information: spectral information, phonemes, morphemes, syntax, semantics and narrative. During natural comprehension of narrative speech the brain must process all of these different aspects of language. Therefore, each of these different aspects of language must be represented somewhere in the brain. In this specific study we focused on mapping the representation of semantic information across the cerebral cortex. In future studies we hope to use a similar approach to map other aspects of language, such phonemes, syntax and so on.

These data were collected while subjects were listening to narrative, autobiographical stories. These stories were engaging and interesting to the subjects, but there are other types of stories and other modalities for receiving them. To explore these issues we therefore plan further studies using different stories, different modalities and different languages.