Today I’m proud to announce the advanced online publication of our paper, and the availability of all source code for the VR engine and the real-time tracking system behind it.

Stowers JR*, Hofbauer M*, Bastien R, Griessner J, Higgins P, Farooqui S, Fischer RM, Nowikovsky K, Haubensak W, Couzin ID, Tessmar-Raible K, Straw AD (* equal contribution) (2017) Virtual reality for freely moving animals. Nature Methods. doi 10.1038/nmeth.4399

FreemoVR is a virtual reality system for freely moving animals. It tracks an animal’s position in real time and redraws a perspective-correct view of a virtual world around it, fast enough that the animal responds to the virtual objects as though they were real. We built it in the Straw lab , and tested it on three quite different animals; freely flying Drosophila, freely swimming zebrafish, and freely walking mice.

The scientific aspects of the publication are better summarised on our laboratory website , in the University of Freiburg’s “A Holodeck for Flies, Fish and Mice” press release, in The Verge and on the BBC World Service , or in the video at the bottom of this post. This was very much a team effort. I share first authorship with Marco Hofbauer; the work was done in collaboration with the labs of Kristin Tessmar-Raible and Iain Couzin, and under the direction of my former boss Andrew Straw . The full author list is above.

Briefly however; virtual reality for animals usually comes with a catch. To show an animal a controlled visual world you have traditionally had to hold it still, for example a fly glued to a tether above an air-supported ball, or a mouse on a treadmill under a fixed dome. The difficulty is that an animal does not work out where it is from vision alone. Movement and sensation are bound together; the vestibular and proprioceptive cues, and the optic flow of a world that shifts because the animal moved, are all part of the same loop. Tether the animal and you break that loop. You can still show it pictures, but you have changed the thing you set out to study.

Our approach removes the tether. By tracking the animal in three dimensions in real time, and recomputing the scene from its own point of view quickly enough to keep up with it, the animal can move freely through a virtual world while its sensorimotor loop stays intact.

This blog post will cover a few of the technical and architectural decisions behind the system. Perhaps it is easiest to start with the principle of operation.

General Architecture

There are two problems to solve, and a deadline between them; first, where is the animal, and second, what should it see. Position is a tracking problem and the scene is a rendering problem, and the system only works if the loop between them closes in a few milliseconds. As with FlyMAD , everything is built on top of the Robot Operating System (ROS), which carries the position estimates, control messages and configuration between processes. In FreemoVR these processes are often spread across several computers, one driving each projector. The great majority of the high level code is python; the rendering engine itself is C++.

Real time tracking

The animal’s position comes from flydra , the lab’s multi-camera real-time tracker. A set of calibrated, synchronised cameras view the arena. Each camera extracts candidate 2D points, and these are combined by triangulation into a single 3D position which is published over ROS as fast as the cameras run. The latency is low enough that the rendering engine can treat the estimate as the current position of the animal. flydra had been in use in the lab for years and was the natural thing to build on.

Perspective correct rendering

The rendering was the harder of the two, and it is where most of the work went. The obvious approach, deciding what the animal should see and drawing it straight onto the display, stops working as soon as the display is not a flat screen in front of a stationary eye. Our arenas are not flat; the fly arena is a projection cylinder, and the fish swim above a hemispherical bowl. The animal is somewhere inside and it is moving, so a given pixel has to show whatever the animal would have seen along that line of sight, from wherever the animal currently is.

So we separate the scene from the screen. The animal has a pose in a global coordinate frame. From that pose we render six views (up, down and the four sides) into a cube map, which captures the whole virtual world around the animal without any reference to where the physical display happens to be. The cube map is then projected onto a calibrated 3D model of the display surface, and finally warped to the output of each projector. The result is that a display of almost any shape, driven by several projectors, can show one seamless, perspective-correct world to an animal that is free to move around inside it.

The engine is built on OpenSceneGraph , with the per-pixel work done on the GPU (optionally through CUDA). A display_server node runs on each display computer; it loads the experiment as a plugin, takes the animal’s pose over ROS, and computes the image for its own projector from the calibrated layout. Calibration matters a great deal here. The geometry of the display surfaces and the pose of each projector have to be measured and stored carefully, because the rest of the pipeline trusts them completely.

Three animals, three arenas

A VR system is only as good as the behaviour it can reproduce, so we tested FreemoVR on three species whose visual worlds are very different.

For Drosophila we used a one metre cylindrical flight arena, the “FlyCave”, with the virtual world projected onto the walls. Free flying flies treat the virtual objects as real, and we could steer their flight paths with visual motion stimuli.

For zebrafish we built the “FishVR” rig, where the fish swim in a shallow dish and the virtual environment is projected from beneath. Working with the Couzin lab on collective behaviour, we put real fish among photorealistic virtual ones and asked what makes a leader. We found that effective social influence depends on a would-be leader trading off its own preferred direction against how strongly it responds to the others, which is the kind of question that is hard to ask unless you control what every other “fish” in the scene does.

For mice we built an elevated maze in which the apparent drop below the walkways was virtual. Mice avoid heights, which is a well studied rodent behaviour, so the test of immersion was simple; would a mouse avoid a virtual drop the way it avoids a real one. It does. A mouse carrying an innate response across into the virtual world is, I think, the most convincing single result in the paper.

Open source

As with FlyMAD, we are releasing the whole system under an open source license, both the VR engine and the flydra tracker. Building a VR rig for a new animal is still real work, but the hard parts, the tracking, the perspective-correct rendering onto an awkward display, and the calibration, no longer have to be written from scratch.

For those still reading, the supplementary video below shows the system running with a freely flying fly.

Comments, suggestions or corrections can be emailed to me .