FreemooVR and LocustVR

A few years ago I wrote about FreemoVR , the virtual reality system for freely moving animals that we built in the Straw lab and published in Nature Methods. A lot has happened since; I left academia and co-founded loopbio , and one of the things we have done there is give FreemoVR a second life. Today I want to introduce FreemooVR , our friendly fork of the original engine, and the first big system we built on top of it, LocustVR .

FreemooVR is a fork of, and the successor to, the original FreemoVR. The science it came out of was tied tightly to the Robot Operating System (ROS), which made good sense in the lab but is a heavy dependency to ask of everyone else. So the main job of the fork was to take ROS out; the inter-process communication is now done over ZMQ, the scene handling and OpenSceneGraph support have been cleaned up and extended, and what is left is a smaller, more hackable base to build a custom VR rig on. We also put the second “o” back into the name, which exists mostly so we could have a cow logo. The hard parts are unchanged from the paper; the perspective-correct cube-map rendering onto an arbitrary, calibrated display surface is still the heart of it.

The first real test of the fork was a system we were asked to build for the collective behaviour groups in Konstanz. The Department of Collective Behavior at the Max Planck Institute of Animal Behavior and the Centre for the Advanced Study of Collective Behaviour came to us wanting the world’s first virtual reality system for freely walking locusts, to study how an individual locust makes decisions in a crowd. The work supported Bianca Schell’s master’s thesis on locust decision-making.

A locust will not sit on a tether the way a fly will, so for LocustVR we put the animal on a spherical treadmill. The locust walks on top of a 600mm sphere whose surface is treated for grip; two servo motors with rotary encoders spin the sphere underneath it to cancel out its walking and hold it near the centre of the arena, whichever way it chooses to go. This part was built in conjunction with PhenoSys , and follows the approach of Kaupert and colleagues . The locust can walk as far as it likes and never leaves the rig.

LocustVR has three parts; the locomotion compensator above, the tracking and closed-loop control, and the virtual reality projection.

The tracking is done by Motif , our video recording and realtime processing software. There are two cameras; a colour camera for observation, and a 100fps near-infrared camera for tracking. A contrast-based method finds the locust’s centre of mass and orientation, the estimate is smoothed with a Kalman filter, and a control law drives the motor velocities to keep the animal in place. That same position, together with the measured rotation of the sphere, is what tells the VR where the locust actually is in its virtual world.

The visual world is drawn by three 120fps projectors mounted above and pointing down, onto an 800mm diameter cylindrical screen and the flat floor around the sphere. Where the projectors overlap, the images are blended so the brightness is even. This is where FreemooVR earns its keep; the virtual world is designed in Blender and the experiment logic is written in python, and the engine warps it all into one seamless, perspective-correct scene for an animal walking around in the middle.

The experiments themselves are deceptively simple. Locusts are strongly drawn to the edges of objects, so one virtual world is just two identical vertical posts standing on an infinite chequerboard. As the locust walks towards them it reaches a point where it has to commit to one post or the other. By moving the posts apart, or adding more of them, and watching where and when the animal commits, the researchers can pick apart how a single decision is actually made. That is the question Bianca’s thesis set out to answer.

LocustVR has had a fair amount of attention since, mostly because swarming locusts make for striking television. The system turns up in a Dutch NTR documentary on collective behaviour, in ARTE’s Xenius episode on locusts, and on PM Wissen on Servus TV. Our own short video of the system is below.

FreemooVR is on GitHub, and there is more about the build on the LocustVR case study page. None of the science here is mine; that belongs to the collective behaviour groups in Konstanz and to Bianca Schell. What we contributed was the rig and the engine to run it, which is exactly the sort of thing loopbio exists to do.

Comments, suggestions or corrections can be emailed to me .