FreemooVR and LocustVR

A few years ago I wrote about FreemoVR , the virtual reality system for freely moving animals that we built in the Straw lab and published in Nature Methods. A lot has happened since; I left academia and co-founded loopbio , and one of the things we did there was continue FreemoVR. Today I want to introduce FreemooVR , our friendly fork of the original engine, and the first big system we built on top of it, LocustVR .

FreemooVR is a fork of, and the successor to, the original FreemoVR. The science it came out of was tied tightly to the Robot Operating System (ROS), which made good sense in the lab but is a heavy dependency to ask of everyone else. So the main job of the fork was to take ROS out; the inter-process communication is now done over ZMQ, the scene handling and OpenSceneGraph support have been cleaned up and extended, and what is left is a smaller, more hackable base to build a custom VR rig on. The hard parts are unchanged from the paper; the perspective-correct cube-map rendering onto an arbitrary, calibrated display surface is still the heart of it.

The first real test of the fork was a system we were asked to build for the collective behaviour groups in Konstanz. The Department of Collective Behavior at the Max Planck Institute of Animal Behavior and the Centre for the Advanced Study of Collective Behaviour came to us wanting the world’s first virtual reality system for freely walking locusts, to study how an individual locust makes decisions in a crowd. The work supported Bianca Schell’s master’s thesis on locust decision-making.

A locust will not sit on a tether the way a fly will, so for LocustVR we put the animal on a spherical treadmill. The locust walks on top of a 600mm sphere whose surface is treated for grip; two servo motors with rotary encoders spin the sphere underneath it to cancel out its walking and hold it near the centre of the arena, whichever way it chooses to go. This part was built in conjunction with PhenoSys , and follows the approach of Kaupert and colleagues . The locust can walk as far as it likes and never leaves the rig.

LocustVR has three parts; the locomotion compensator above, the tracking and closed-loop control, and the virtual reality projection.

The tracking is done by Motif , our video recording and realtime processing software. There are two cameras; a colour camera for observation, and a 100fps near-infrared camera for tracking. A contrast-based method finds the locust’s centre of mass and orientation, the estimate is smoothed with a Kalman filter, and a control law drives the motor velocities to keep the animal in place. That same position, together with the measured rotation of the sphere, is what tells the VR where the locust actually is in its virtual world.

The visual world is drawn by three 120fps projectors mounted above and pointing down, onto an 800mm diameter cylindrical screen and the flat floor around the sphere. Where the projectors overlap, the images are blended so the brightness is even. This is the part FreemooVR handles: the virtual world is designed in Blender and the experiment logic is written in python, and the engine warps it into one seamless, perspective-correct scene for an animal walking around in the middle.

The experiments themselves are simple. Locusts are strongly drawn to the edges of objects, so one virtual world is just two identical vertical posts standing on an infinite chequerboard. As the locust walks towards them it reaches a point where it has to commit to one post or the other. By moving the posts apart, or adding more of them, and watching where and when the animal commits, the researchers can pick apart how a single decision is actually made. That is the question Bianca’s thesis set out to answer.

LocustVR has been featured in several television documentaries about the locust swarms: a Dutch NTR documentary on collective behaviour, ARTE’s Xenius episode on locusts, and PM Wissen on Servus TV. Our own short video of the system is below.

FreemooVR is on GitHub, and there is more about the build on the LocustVR case study page. The science was done by the collective behaviour groups in Konstanz and by Bianca Schell; loopbio built the rig and the engine that runs it.

Comments, suggestions or corrections can be emailed to me .