Closed-loop complex experiments: A paradigm shift¶
The traditional design of experiments is as follows: the experimenter decides on a set of stimuli to present the participants, and records their responses to the stimuli. These responses can be both behavioral, such as a button press, or brain activity, as measured by fMRI or ECoG. This paradigm has two key characteristics: it is parametric and largely open-loop. First, this paradigm is parametric in that the entire space of possible stimuli is known before the experiment is run. Second, this paradigm is open-loop in that the response space is also carefully constrained to a set of limited actions, and participant actions do not meaningfully affect the stimulus. For example, although trial timing may be based on participant responses, the sequences or sets of trials or stimuli are nevertheless fixed. These characteristics stem from the fact that statistical analyses must be performed in order to reach conclusions, and we must know the experiment parameters in order to perform such analyses. In a traditional experiment, we fix these parameters beforehand such that they are all known, or at least easily determined from logs, e.g. button presses.
However, such experiments don’t match up with real-world behavior. Real-world experiences are a continuous perception-cognition-action loop. We see the world, make decisions based on these inputs, and we then act. These actions then drastically alter our perceptual inputs, and the loop continues. The implicit assumption of cognitive neuroscience is that we want to understand how the brain functions in the real world. However, the brain is nonlinear, and traditional experiments don’t allow the brain to operate in the same regime as it does in the real world. This constraint is a limited issue in the study of perceptual systems: Perceptual systems in the brain can be reasonably approximated as an input-driven system, but the systems that mediate more complex behaviors cannot. For example, visual perception can be largely studied with open-loop stimuli, but spatial navigation or social interaction are by definition closed-loop processes.
A new paradigm of experimental design is needed for understanding the human brain systems that underlie complex behavior. The key characteristics for experiments to enable real-world-like behavior are two-fold: they must be closed-loop and capable of emergent complexity. First, the paradigm is closed-loop in that the participants’ actions can meaningfully influence the experimental state. However, closed-loop is necessary but not sufficient. The critical characteristic is the capability for emergent complexity. This complexity is in the same vein as mathematical chaos: the temporal evolution of the system is heavily dependent on its initial state and inputs. For example, allowing the participant to move along a virtual track is closed-loop, but parametric. On the other hand, driving through a virtual city requires the participant to deal with traffic and jaywalkers, and offers a practically unlimited number of ways to get to the destination; doing so is not only closed-loop, but also complex. Having both enables naturalistic behavior in the experiment.
Sidebar: chaotic versus naturalistic. Experiments can be naturalistic without being complex, for example movie-watching or story-listening. Here the stimuli are naturalistic, but there is no emergent chaos in the experiment. Contrast this with driving or a two-way conversation: the visual scene in driving and the semantic content of the conversation are wholly unpredictable before the experiment. Yet chaotic is not necessarily naturalistic: random noise can also be generated via chaos, but it is not naturalistic. We assume that we are building chaotic and naturalistic experiments.
The apparently uncrossable open waters of chaotic experiments¶
How does one go about implementing emergent complexity in an experiment? Software designed for traditional experiments, such as PsychoPy or Psychtoolbox are ill-suited for closed-loop, complex experiments. These imperative programming packages operate linearly, make it impractical to program the logic needed to support emergent complexity, they don’t provide the graphics capabilities for rendering complex stimuli, and they are not capable of the performance required to run complex systems in real time.
Even after we find the software to implement chaotic experiments, there is another apparently insurmountable challenge to chaotic experiments. Namely, how does the investigator acquire the parameters for the statistical analysis downstream of data collection? The answer is, of course, to record everything. But the details of an emergent chaotic system grow exponentially with its complexity, and it rapidly becomes impractical to record features of interest.
Game engines as a solution to complexity¶
Game engines are architected for complexity¶
Game engines provide a practical solution to both challenges. They abstract away many of the graphics, world-building, input handling, state updating, physical simulation, and other functionalities so that the user can focus on implementing their own logic. Crucially, unlike classical experimental software platforms, they are truly object-oriented. Object-oriented programming is vital for implementing complex emergent systems by decomposing a complex system into its component parts: doing so logically compartmentalizes the system and makes the code tractable to understand, implement, and maintain.
Sidebar: a Copernican shift. In traditional experimental software, the experimental logic is central and defines the program that you write. When programming with a game engine, the experimental logic is only a small part of the whole system that makes up the game. Creating experiments in game engines requires making a Copernican shift from writing scripts that revolve around the experiment logic to architecting software in which the experimental logic is one of many actors in a dynamic world.
Game engines can extract any information¶
Game engines not only make building complex experiments tractable, but also enable extracting parameters from these experiments for statistical analysis. Because the experimenter built the game, the experimenter has backstage access to all state variables in the game at any point in time, and can implement functionality to extract whatever parameters they wish. At first glance, however, doing so may only seem like a theoretical possibility rather than a practical possibility: many parameters, such as semantic segmentation, are computationally intensive to calculate and can interfere with the real-time operation of the experiment.
Fortunately, this problem has been addressed by the replication systems used to support networked multiplayer games. Briefly, replication systems transmit and synchronize world state between multiple computers. To make the most efficient use of network bandwidth, replication systems rely on all computers having the same game executables and assets. The replication system then only has to transmit compact semantic descriptions of world state changes between computers. For example, when one player moves, the replication system does not render the movement and transmit pixel values; rather, it transmits the message that a certain player has moved from point A to B, and the receiving machine updates the world state in its game instance and renders the appropriate images. Doing so compresses the state of large, complex worlds down to extremely small file sizes.
Replication systems naturally lend themselves to recording: one can capture the network packets emitted by a replication system, store them to disk, and play them back later in time. Here, it is important to understand that the playback of gameplay recording through the replication system is conceptually not the same as a video recording. In playing a video recording, we see the pixel values as shown, but a video recording has no semantic understanding of the events that occurred. In playing a game recording, we see a faithful re-enactment of the events that occurred. We can step around stage and view the actors from a different angle, we can ask the actors about the roles that they are playing, and we can even add new objects into the play. For experiments, this replication system enables the experimenter to extract parameters offline, regardless of the computational requirements of extracting the parameters. These two key attributes: the capability for implementing chaotic experiments and the replication system that decouples parameter extraction from the experiment, make game engines especially suited for designing naturalistic, closed-loop, and complex experiments.
For further reading, see the Unreal article on the replay system, the Valve Networking article, the Unreal Engine wiki on the classic replication system, and the new Unreal Iris replication system.
Other options to game engines¶
Implementing an experiment in a game engine is time-consuming and labor-intensive. Here we briefly note two alternative approaches and how they differ from implementations in game engines. First are 3D graphics systems such as Blender or 3Ds Max. These platforms give the user fine-grain control over rendering and stimulus presentation. In addition to their 3D graphics capabilities, they also include scripting systems that allow programmatic interaction with the scene. However, these platforms are graphics-oriented and do not have the capability for complex interactions, such as simulations of AI agents, UI scaffolding for end-user display, or, crucially, a replication system that can faithfully replicate program state. 3D graphics systems are extremely powerful for building realistic scenes, but require additional infrastructure to support complex interactions. They occupy a different niche than game engines: they produce the 3D assets that are then used downstream by game engines.
A second option is to use pre-existing games. Many games, such as Garry’s Mod, Minecraft, or Stardew Valley, provide modding systems that allow for extensive user customization. Often, these games have built-in replication systems that enable the critical offline reconstruction needed for analysis. In many cases, such games can suffice for experiments. However, working at the mod level of a game restricts the experimenter to the affordances provided by the game itself. For example, off-the-shelf games do not have semantic segmentation capabilities, or customizable AI agents, and parameter access is limited to what is exposed by the game’s API. Building experiments from the ground up in a game engine allows the experimenter to retain complete control over all aspects of the software platform.