Our paper is now published online , and the source code for the VR engine and real-time tracking system is available.

Stowers JR*, Hofbauer M*, Bastien R, Griessner J, Higgins P, Farooqui S, Fischer RM, Nowikovsky K, Haubensak W, Couzin ID, Tessmar-Raible K, Straw AD (* equal contribution) (2017) Virtual reality for freely moving animals. Nature Methods. doi 10.1038/nmeth.4399

FreemoVR tracks an animal’s position in real time and redraws a perspective-correct view of a virtual world around it. The update is fast enough for the animal to respond to virtual objects as if they were real. We built it in the Straw lab and tested it with freely flying Drosophila, freely swimming zebrafish, and freely walking mice.

The science is summarised on our laboratory website , in the University of Freiburg’s “A Holodeck for Flies, Fish and Mice” press release, in The Verge , on the BBC World Service , and in the video below. Marco Hofbauer and I share first authorship. The work was done with the labs of Kristin Tessmar-Raible and Iain Couzin, under the direction of my former boss Andrew Straw . The full author list is above.

Until now, showing an animal a controlled visual world usually meant holding it still. A fly might be glued to a tether above an air-supported ball, or a mouse might walk on a treadmill under a fixed dome. An animal does not work out where it is from vision alone. It also uses signals from movement and balance, along with the movement of the world across its eyes. Tethering breaks that connection. The animal can still see the pictures, but the experiment has changed its behaviour.

We removed the tether. The system tracks the animal in three dimensions and redraws the scene from its point of view. It does this fast enough for the animal to move through the virtual world while its normal movement and visual feedback remain connected.

Here are some of the decisions behind the system. The diagram shows how it works.

General Architecture

There are two jobs, with only a few milliseconds between them: find the animal, then decide what it should see. Tracking supplies the position and rendering uses it to draw the scene. The loop has to close quickly. As with FlyMAD , the processes use the Robot Operating System (ROS) to pass position estimates, control messages and configuration around. They often run on several computers, one for each projector. Most of the higher-level code is python; the rendering engine is C++.

Real time tracking

flydra , the lab’s multi-camera tracker, supplies the animal’s position. Calibrated, synchronised cameras watch the arena. Each camera finds candidate 2D points, and triangulation combines them into one 3D position. flydra publishes that position over ROS as fast as the cameras run. The delay is low enough for the renderer to use it as the animal’s current position. The lab had already used flydra for years, so it was the obvious base.

Perspective correct rendering

Rendering was harder. Drawing the scene straight onto the display only works for a flat screen and a fixed viewpoint. Our arenas are not flat. The fly arena is a projection cylinder, and the fish swim above a hemispherical bowl. The animal moves inside the display, so every pixel must show what it would see along that line of sight from its current position.

We separated the scene from the screen. The animal has a pose in a global coordinate frame. From that pose we render six views, up, down and the four sides, into a cube map. This records the virtual world around the animal without referring to the physical display. We project the cube map onto a calibrated 3D model of the display surface, then warp it for each projector. Several projectors can therefore show one perspective-correct world around an animal that moves inside it.

The engine uses OpenSceneGraph , with per-pixel work on the GPU, optionally through CUDA. A display_server node runs on each display computer. It loads the experiment as a plugin, reads the animal’s pose over ROS, and computes the image for its projector from the calibrated layout. The display geometry and the position of each projector must be measured and stored carefully. Everything else relies on those values.

Three animals, three arenas

We tested FreemoVR on three species with very different visual systems.

For Drosophila we used a one metre cylindrical flight arena, the “FlyCave”, with the virtual world projected onto the walls. The flies responded to virtual objects as if they were real, and visual motion stimuli changed their flight paths.

For zebrafish we built the “FishVR” rig, where the fish swim in a shallow dish and the virtual environment is projected from below. With the Couzin lab, we put real fish among virtual fish and measured their responses. A fish was more influential when it balanced its own preferred direction with its response to the others. The virtual fish let us control the behaviour of every other fish in the scene.

For mice we built an elevated maze in which the drop below the walkways was virtual. Mice avoid heights, so we tested whether they would avoid a virtual drop as they avoid a real one. They did. This was the clearest test that the virtual scene affected behaviour.

Open source

As with FlyMAD, we are releasing the whole system under an open source licence, including the VR engine and the flydra tracker. Building a VR rig for a new animal still takes work. The tracking, perspective-correct rendering onto an awkward display, and calibration do not have to start from scratch.

For those still reading, the supplementary video below shows the system running with a freely flying fly.

Comments, suggestions or corrections can be emailed to me .