On Abstractions, Simplicity, and Positioning

Over the last few years I have spent a lot of time writing scientific software for research. In science the primary role of software is to enable discovery - software here does not exist to admire its own beauty (this point is often made ).

Prior to that I was completing my PhD, teaching various engineering and embedded software classes. I have hacked on and around the periphery of GNOME for a while, and mentored a few GSOC students.

During this time I have come to believe that the most important design principle in software engineering, and the one from which all the others follow, is minimising the cost of the inevitable failure of your abstraction.

How often have you watched someone hack together a limping prototype in no time at all on top of some 50,000ft, buzzword-laden framework, only to pay those time savings back exponentially later? Typically it happens the moment they have to navigate down through the levels of abstraction, armed with knowledge they do not possess, to implement something only marginally outside the scope the framework anticipated.

This post is a collection of musings on that topic: on defensive design to minimise such costs, on proactive education so that people do not fall into the trap quite so often, and finally on positioning your project not so that the shiny tasks are trivial, but so that the complex tasks are predictably difficult.

I would like to discuss this in the context of a few projects I am involved with. It should also be read with the knowledge that three years ago I left the engineering field and switched to behavioural neuroscience. I did this willingly - choosing to focus on knowledge, and not to fall into the trap of building the shiny, waste-of-time widgets I saw far too often in engineering.

Python

One of the reasons we use Python so extensively is that it offers a whole hierarchy of options for addressing the failure of an abstraction. If a pure-Python approach is too slow, or you need to reach a library that has no binding, the escape hatches are numerous and, crucially, well signposted. One can access external libraries through a number of strategies (ctypes, CFFI, Cython). One can incrementally port the hot path to another language and call back into it. One can lean on the specialist numerical and vector libraries (numpy, scipy and friends) that already live much closer to the metal. One can incrementally accelerate code in place with JIT techniques (numba and similar), or move the whole program to another interpreter (PyPy) entirely.

What matters here is not merely that these strategies exist, but that they are well known, openly advertised within the scientific community, and - tellingly - regarded as a benefit rather than an admission of failure. Python tells you, up front, how to leave Python. That is good positioning.

OpenCV

I would like to contrast that positioning with OpenCV. For as long as I can remember OpenCV has been a victim of its own success - or, as I prefer to think of it, a victim of the level of abstraction it provides, combined with the documentation and positioning of the project, and the absence of any clear pathway back down through its layers.

Consider an API that offers a single function call to detect faces in an image. It is a wonderful demo. It is also a near-guarantee that the moment a user’s problem strays outside the narrow case that function was built for, they are stranded - with no obvious next step, no map of what sits beneath the call, and a forum full of identical, unanswerable questions. The abstraction is not wrong; it is simply presented as though it has no underside. When it fails, and it always eventually fails, the cost of that failure is enormous, precisely because the project never showed you the way down.

Arduino and the maker community

Arduinos have been both a blessing and a curse to the maker community. While I think making engineering attractive and accessible to the masses is a great thing, and I have no wish to be the person who sneers at it, it is worth considering whether the positioning - and the options it leaves for de-abstraction - actually serve the goal we claim to have.

My mantra is not knowledge for its own sake, but knowledge appropriate to what the user is ultimately trying to do. A traditional engineering education would have a person build their own microcontroller board, attach their own hardware, and write their own drivers. This might appear tedious, and much of it feels like make-work, but having walked once up the stairs of complexity you implicitly learn two things: the path back down when you eventually need it, and a realistic sense of how long these things actually take.

I have not seen much evidence that the ability for thousands of people to build a flashing LED has increased the number of talented engineers building anything substantial. I am not advocating starting every software course with assembly, or every hardware course with logic gates. I am arguing that if you want to see a diversity of complexity downstream, you should not sell shiny and reward ignorance. That path starts with novelty and ends in noise.

GNOME Shell

It is no secret that I was completely dumbstruck when GNOME chose JavaScript as its preferred development language. Rationale such as ’to help bootstrap the platform and drive development’, and similarly the counting of the very existence of the Python standard library as a negative, are in complete opposition to the philosophy and thesis I have explained here. To borrow a line: we built a wasteland and called it peace.

And so, while we might see a bunch of trivial shell extensions for changing the colour of the activities button, I am not seeing much evidence of widespread JavaScript adoption driving genuinely complex application development in GNOME. Might that be, in part, because we sold the shiny without ever drawing a clear path back down for the people who would have had to build the difficult things?

Robot Operating System

ROS is the project that prompted much of this thinking, and it is the one I have the most sympathy for. ROS is, at heart, an honest abstraction. You build a system out of independent processes (nodes) that communicate by publishing and subscribing to strongly-typed message “topics”, and almost everything about that arrangement is inspectable. When a node misbehaves you can watch the messages on the wire, swap the node for another implementation in a different language, drop down to the raw transport, or rewrite a hot path as a nodelet to claw back the performance you lost crossing a process boundary. The seams are visible, and the path down is short and well documented. By the standards of this post, that is close to ideal.

Where ROS reproduces the trap is at the top, in the large meta-packages - the navigation stacks, the manipulation frameworks - that promise a great deal out of the box. They are enormously useful, and they are also the easiest place to assemble something impressive without understanding it, and therefore the place where the eventual failure of the abstraction is most expensive. The lesson is not that the high-level pieces are bad. It is that a framework earns trust by making the descent from them obvious, and loses it the moment a stranded user goes looking for the stairs and cannot find them.

In closing

None of this is an argument against abstraction. Abstraction is the only reason any of us get anything done. It is an argument about where the bodies are buried, and about being honest with your users concerning the day they will have to dig.

So design defensively, on the assumption that your abstraction will fail and that the cost of that failure should be small and survivable. Educate proactively, so that the people downstream have walked the stairs at least once and know the way back down. And position your project deliberately - not so that the shiny tasks are trivial, because everyone manages that, but so that the genuinely complex tasks are predictably difficult. Predictable difficulty is a feature. It is, in fact, the whole game.

This post, appropriately, took rather longer than my initial abstraction of it suggested it would.