FreemooVR and LocustVR

A few years ago I wrote about FreemoVR , the virtual reality system for freely moving animals that we built in the Straw lab and published in Nature Methods. Since then I have left academia and co-founded loopbio . We kept working on FreemoVR there. This post is about FreemooVR , our fork of the original engine, and LocustVR , the first large system we built with it.

FreemooVR is the successor to the original FreemoVR. The original system depended on the Robot Operating System (ROS). That was useful in the lab, but it made the software harder to use elsewhere. We removed ROS and replaced its inter-process communication with ZMQ. We also cleaned up and extended the scene handling and OpenSceneGraph support. The result is a smaller base for building a custom VR rig. The main rendering method is still the same: a perspective-correct cube map projected onto an arbitrary calibrated display surface.

The first real test was a system we built for the collective behaviour groups in Konstanz. The Department of Collective Behavior at the Max Planck Institute of Animal Behavior and the Centre for the Advanced Study of Collective Behaviour wanted a VR system for freely walking locusts. They planned to use it to study how one locust makes decisions in a crowd. The work formed part of Bianca Schell’s master’s thesis on locust decision-making.

A locust will not sit on a tether like a fly, so we put it on a spherical treadmill. It walks on a 600mm sphere with a roughened surface. Two servo motors with rotary encoders turn the sphere under the locust to cancel its walking and keep it near the centre of the arena. This works in any direction. We built this part with PhenoSys , following the approach of Kaupert and colleagues . The locust can keep walking without reaching the edge of the rig.

LocustVR has three parts: the treadmill, the tracking and control system, and the VR projection.

Motif , our video recording and realtime processing software, handles the tracking. One colour camera records the animal, and a 100fps near-infrared camera tracks it. A contrast-based method finds the locust’s centre of mass and orientation. A Kalman filter smooths that estimate, and a control law sets the motor speeds to keep the locust in place. The VR system uses the tracked position and the measured rotation of the sphere to place the locust in its virtual world.

Three 120fps projectors point down at an 800mm cylindrical screen and the flat floor around the sphere. Their images overlap, so we blend them to keep the brightness even. FreemooVR handles this display. The virtual world is designed in Blender , and the experiment logic is written in python. The engine warps the images into one perspective-correct scene around the locust.

The experiments are simple. Locusts are drawn to the edges of objects, so one virtual world contains two identical vertical posts on an infinite chequerboard. As the locust walks towards them, it must choose one post. The researchers can move the posts apart or add more posts, then record where and when the locust chooses. This is the question Bianca’s thesis set out to answer.

LocustVR has appeared in television documentaries about locust swarms: a Dutch NTR documentary , an ARTE Xenius episode, and PM Wissen on Servus TV. Our short video is below.

Tom Scott also visited the lab and made a video about LocustVR. It starts at the section about this system.

FreemooVR is on GitHub. The LocustVR case study has more detail about the build. The collective behaviour groups in Konstanz and Bianca Schell did the science. loopbio built the rig and the engine.

Comments, suggestions or corrections can be emailed to me .