Conduit Architecture and Updates
A lot has been happening on the Conduit front of late, and I probably should have blogged about it earlier. Unfortunately, it has seemed as though I have made 5 steps forward and 4 steps back with every hour of work on the project, so why, and whats up?
Background
I will be dropping a 0.3 release to coincide with the release of GNOME 2.18. This will be a developer release designed to get things working well before a stable 0.4 release some time after.
All of Conduits synchronization capability is determined at runtime. Conduit scans the users' and the system wide dataproviders directories in the same way that Deskbar loads handlers when it starts. The core synchronization logic is decoupled from the dataproviders, and only those relevant dataproviders are shown to the user (i.e. only show iPod Notes if an iPod is connected, etc).
Remember that I want Conduit to be a GNOME wide synchronization service that other applications can hook into, so I need to not only decouple the core sync logic from the application and usage specific dataprovider back ends, but also allow conduit to be extended by third parties.
Extension by 3rd Party Applications
I mentioned that all synchronization capability is determined at runtime. The other half of this story is that a third party may wish to synchronize something that Conduit knows nothing about (I'll use a Jokosher project as an example, although in my next post I will address these issues in the Tomboy case). Lets assume that a Jokosher project cannot be described in terms of the basic types of data that Conduit knows about (email, contact, file, etc). In order to allow Jokosher projects to be synchronized between different computers the Jokosher developers would need to do the following
-
Define a Jokosher datatype that describes what it is to be synchronized.
-
(optionally) Define some conversions between the Jokosher datatype and the basic Conduit datatypes (File, Text, etc)
-
Define a dataprovider that encapsulates Jokosher invocation specific information such as the Jokosher user, and how does one go about extracting/modifying Jokosher specific data when Jokosher may be running.
The Conduit framework will do a lot of heavy lifting here, such as caching modification times, providing a way for the user to configure a Jokosher sync partnership, providing a way for a user to resolve conflicts, and even allowing the whole sync process to be controlled from outside of the application over DBus. Most interestingly however, depending on the fidelity and number of conversions specified in #2, Conduit will facilitate a number of other useful sync scenarios including,
-
Sync Jokosher projects via any gnomevfs compatible location (ftp, etc)
-
Sync via Amazon S3 (coming soon)
-
Sync via USB key
-
Sync directly over a local network using Avahi. (next release)
The point here is that this decoupling of the dataproviders, from the core and the datatypes (sans the conversion functions) lets the Jokosher dataprovider take advantage of any/all additional $FUTURE dataproviders (e.g. Amazon S3) and all improvements to the Conduit core (such as direct sync using pickle'd python over network via Avahi).
Testing
As I get closer to release I get more nervous about releasing a tool which will could eat peoples data. To mitigate this risk I spent a week or two implementing an extensive testing framework for Conduit, a task I should have tackled much earlier in the project.
Unfortunately the testing is made more complicated when the applications capabilities can change between invocations (dynamic loading of dataproviders and conversions explained earlier). Aside from the usual unit level tests (does this throw the correct exceptions, etc) I am also required to test the fidelity and accuracy of the conversions between datatypes (round trip comparisons), and the fidelity of synchronization from dataprovider $FOO to dataprovider $BAR via any of these datatypes.
The testing framework uses a mixture of python and shell script, and is even web 2.0 compliant ;-). It currently tests available conversions, saving and getting data from various dataprovider back ends (Flickr, gnomevfs, Tomboy, etc). It even does code coverage and pushes the results on line.
What's Next
I apologize for this post being mostly academic in nature, lacking in screen shots. I will post again in a few days with some more relevant information (from a users perspective). I just wanted to highlight how and why I believe a desktop sync service is useful, and how I am trying to satisfy this in a sustainable way for the GNOME platform as a whole.
I anticipate finishing up the two way Tomboy support (via iPod notes, gnomevfs location, and backpackit.com) in the next few days. There are some bugs preventing running Conduit on Feisty (goocanvas 0.6.0 is ABI incompatible with 0.4.0). There are also some other lose ends (like some GUI blocking calls in conflict resolution, and some difficulties saving program configuration). Unfortunately, like a lot of FOSS, the best is always just around the corner....