Jun 12, 2007
Excuse the shameless quoting of fellow conduit contributor John Carr's blog post but this video is pretty cool;
[googlevideo]http://video.google.co.uk/videoplay?docid=-2623798477165899506&pr=goog-sl[/googlevideo]
Video showing how you can use Conduit to synchronise Evolution Contacts and Tomboy Notes between two machines running the GNOME desktop environment.
To start with, laptop 1 has 2 contacts and 1 tomboy note. Laptop 2 just has 1 tomboy note. After syncing, both will have 2 contacts and 2 notes. Not too exciting.. but then... (on laptop 2) 1 note and 1 contact are deleted. The remaining note and contact are modified as well. When the camera flips back to laptop 1 and the sync happens you will see the windows update in real time - including the an open tomboy note getting updated.
Apologies for the poor quality
Jun 7, 2007
Conduit
For all those that are interested I just released Conduit 0.3.1, the second development release in the series. The release includes a heap of bug fixes and some interesting new features including;
Goals for the next release include
The thing that keeps me interested in conduit is the speed at which we are progressing. Each new dataprovider added has a benefit proportional to the number of already included dataproviders; for instance adding evolution support opened up the ability to do tomboy notes <--> evolution memos, as well as evolution memos <--> ipod notes, etc.
Gnome Link Drop
Here are some interesting things on the horizon that made me excited about GNOME and gtk. Delegated to people with more free time and ability than me!
Anyway, Im still on holiday, and loving it!
Jun 1, 2007
The reaction to my previous post was extremely positive, with a lot of interest in the bindings. Ive had a bit more free time and done some more hacking on both the bindings, and integrating evolution support into Conduit. On the first point;
Work on the 0.3.1 bugfix release of Conduit is coming along nicely, it should be out sometime over the weekend. In between fixing bugs however I been able to introduce some new dataproviders, which in turn will open up a heap of new synchronization possibilities.
-
Thanks to new conduit contributor Thomas Van Machelen we now support Smugmug and Picasaweb photo sharing sites.
-
Support for Facebook photos (uploading yours, and downloading your friends) is nearing completion thats to the excellent API complete pyfacebook bindings.
-
By supporting Evolution Memos and TODO items we now support some pretty cool sync combinations;
-
Evolution Memos <--> Backpackit.com and Tomboy Notes
-
Evolution Memos and Contacts <--> iPod
-
gnome-about-me (evolution) <--> Facebook profile
I recognise not all of these sync pathways are immediately useful or new, but they are providing a good test of the framework, checking if the design is correct and providing useful test cases. Its also exciting to think that when network support lands next cycle, direct sync between two computers over the web/LAN will just work, and will support any dataprovider that is available to conduit.
May 7, 2007
Finally, and ?only? 3 months overdue I have released Conduit 0.3.0. This release finally marks the end of the sync engine rewrite from the previous release. This also signifies the first time that conduit is simultaneously useful to end users, and application developers as a desktop sync service.
[
](http://www.conduit-project.org/wiki/Screenshots)
What can it do?
From an end user perspective Conduit has reached the level of being useful. I am currently travelling around Europe for a few months and using Conduit on a daily basis, at least for the task of Synchronization/backup of my photos to Flickr and my home server, Sync/Export of my Tomboy notes to iPod. Other than that Conduit can currently perform the following sync partnerships;
-
Two way file/folder sync on gnomevfs volumes
-
Two way Tomboy note sync via gnomevfs volumes
-
Two way Tomboy note sync via ipod notes
-
One way sync of files/folders of photos to Flickr
-
One way sync of FSpot tagged photos to Flickr
In the one way sync/export case Conduit is smart in the sense that if a piece of data has not been modified then it will not be synchronized/exported again, it will replace the existing data.
I have also added conflict detection, and a UI for resolving conflicts, including the ability to compare the conflicting data (using gnome-open on the relevant URI).
Desktop Sync Service
I have been talking about the merits of Sync as a desktop sync service for a while, and this release finally marks the point where I can start to add export and sync capabilities to GNOME apps using conduit. In this situation Conduit is exclusively controlled via DBus (independent of the UI). By sharing the same DB we can avoid duplicate data in the synchronization process.
To demonstrate this I modified to Uploadr to call conduit over DBus (and called the resulting app Yaput - yet another photo upload tool) (screenshot). This means that duplicate photos will not be uploaded, updated photos will replace old ones, and it doesn't matter whether you upload the photos from the Conduit GUI, the DBus interface or $(YOUR_APP_HERE). All that using 10 DBus calls!
Anyway, that's the plan, and apart from a few annoying known issues, I finally feel like I am on track.
Mar 29, 2007
Its been a while since I blogged about Conduit, and a lot has been going on. My main task has been working on the conflict resolution UI and on tweaking the conversions between datatypes so as to maintain maximum fidelity. John Carr, the other main Conduit contributor has also been working on some awesome stuff.
Conflict Resolution
During the synchronization process a conflict can occur in any of the following scenarios
-
In a one way synchronization, if the destination data is newer that the source data (the exported data shoud not clobber user updated data case)
-
In a two way sync when the destination data has been modified since the last synchronization (the two computers sync to the same location but both have had source changes case)
-
In any situation when (for backend specific reasons) a comparison between data is unable to determine which is newer than the other.
-
A piece of source data has been deleted and the user has selected a synchronization policy in effect which states they want to be certain before deleting the last remaining copy of that data at the destination. (I handle delete as a special case of a conflict)
Dataprovider backends communicate conflicts to the UI via signals. The UI has access to the conflicting data so can offer the user the ability to compare which is newer. Conflicts use the same arrow metaphor that the main UI uses and can either be resolved in the main window, or be shown with more detail in their own window.
If the data has set a (gnome-open'able) URI then the user can view the data at this location and decide which is newer. For example, a user is synchronizing his Tomboy notes via an iPod. If there is a conflict in this scenario then clicking Compare will cause Tomboy to show the local note and will launch a text editor to inspect the conflicting note off the iPod. (Lazyweb request: Any idea on how I might arrange the windows side-by-side via libwnck for example).
Fidelity
The main difficulty with synchronizing different datatypes, or the same datatypes but via a middleman is maintaining fidelity (that is not losing any information) during the whole process. In the case of contact or calendar data then the default would be to support just standard datatypes (vcard, ical, etc).
-
Opensync 0.30 is looking to define their own XML schema for basic datatypes, and I would support this idea and likely adopt the same schema.
-
Apple .mac uses strict datatypes and pushes everything to a central server. One alternative to maintaining fidelity is to keep data in all formats but maintain a single canonical source of modification times, and perform conversions (even if they lose fidelity) only when needed from this central store.
This leads onto my next suggestion Having an optional central repository (like .mac) not only addresses some of the difficulties of maintaining fidelity through conversion but also opens up some other use-cases which the open source world has not addressed.
-
Online editing of tomboy notes while also being able to sync them on multiple computers (I can currently do this through backpackit.com but the conversions lose fidelity/information)
-
Synchronize desktop settings (wallpaper, theme, nautilus prefs, etc between two computers)
-
One click file sync with no configuration.
So lazywab hackers. Who here is a Django hacker that wants to write me a web service to do this. I already have a documented API that you could write against. I have the spec, the idea, but not the time. I had considered writing this myself, and setting it up as a paid service (I would pay a few dollars a month to be able to do this) but dont have the time, nor the web development experience, nor an inclination of artistic ability to make it look nice! Any takers, please email me!
Miscellany
-
John has finished up all the remaining pieces to allow iPods to be hotplugged/removed in conduit, so that synchronization settings and state are preserved.
-
He is also hacking on direct computer-computer sync over local network. Using avahi for discovery and pickling data over the wire. Should be awesome when this lands!
-
Unfortunately I was not eligible for summer of code this year, I am on holiday from university for six months and wont be starting PhD till after the summer. I was pleased to see mention of Conduit in the Ubuntu SOC ideas page, so I am glad to see distros recognizing the importance of making synchronization easy!
-
Releasing 0.3.0 is blocking on my having a working Ubuntu Fesity install to test against. (Lazyweb request: Does suspend,resume and hotkeys for panasonic sub notebooks such as the CF-R4 work on Feisty out of the box - it has not done so in previous releases)
Update: Google just released python GData bindings. This will make excellent support for Google Calendar, Google Notebook and a few other ideas I have up my sleeve a reality in the near future!. My blog is also back up, sorry for the trouble.
Mar 10, 2007
A lot has been happening on the Conduit front of late, and I probably should have blogged about it earlier. Unfortunately, it has seemed as though I have made 5 steps forward and 4 steps back with every hour of work on the project, so why, and whats up?
Background
I will be dropping a 0.3 release to coincide with the release of GNOME 2.18. This will be a developer release designed to get things working well before a stable 0.4 release some time after.
All of Conduits synchronization capability is determined at runtime. Conduit scans the users' and the system wide dataproviders directories in the same way that Deskbar loads handlers when it starts. The core synchronization logic is decoupled from the dataproviders, and only those relevant dataproviders are shown to the user (i.e. only show iPod Notes if an iPod is connected, etc).
Remember that I want Conduit to be a GNOME wide synchronization service that other applications can hook into, so I need to not only decouple the core sync logic from the application and usage specific dataprovider back ends, but also allow conduit to be extended by third parties.
Extension by 3rd Party Applications
I mentioned that all synchronization capability is determined at runtime. The other half of this story is that a third party may wish to synchronize something that Conduit knows nothing about (I'll use a Jokosher project as an example, although in my next post I will address these issues in the Tomboy case). Lets assume that a Jokosher project cannot be described in terms of the basic types of data that Conduit knows about (email, contact, file, etc). In order to allow Jokosher projects to be synchronized between different computers the Jokosher developers would need to do the following
-
Define a Jokosher datatype that describes what it is to be synchronized.
-
(optionally) Define some conversions between the Jokosher datatype and the basic Conduit datatypes (File, Text, etc)
-
Define a dataprovider that encapsulates Jokosher invocation specific information such as the Jokosher user, and how does one go about extracting/modifying Jokosher specific data when Jokosher may be running.
The Conduit framework will do a lot of heavy lifting here, such as caching modification times, providing a way for the user to configure a Jokosher sync partnership, providing a way for a user to resolve conflicts, and even allowing the whole sync process to be controlled from outside of the application over DBus. Most interestingly however, depending on the fidelity and number of conversions specified in #2, Conduit will facilitate a number of other useful sync scenarios including,
-
Sync Jokosher projects via any gnomevfs compatible location (ftp, etc)
-
Sync via Amazon S3 (coming soon)
-
Sync via USB key
-
Sync directly over a local network using Avahi. (next release)
The point here is that this decoupling of the dataproviders, from the core and the datatypes (sans the conversion functions) lets the Jokosher dataprovider take advantage of any/all additional $FUTURE dataproviders (e.g. Amazon S3) and all improvements to the Conduit core (such as direct sync using pickle'd python over network via Avahi).
Testing
As I get closer to release I get more nervous about releasing a tool which will could eat peoples data. To mitigate this risk I spent a week or two implementing an extensive testing framework for Conduit, a task I should have tackled much earlier in the project.
Unfortunately the testing is made more complicated when the applications capabilities can change between invocations (dynamic loading of dataproviders and conversions explained earlier). Aside from the usual unit level tests (does this throw the correct exceptions, etc) I am also required to test the fidelity and accuracy of the conversions between datatypes (round trip comparisons), and the fidelity of synchronization from dataprovider $FOO to dataprovider $BAR via any of these datatypes.
The testing framework uses a mixture of python and shell script, and is even web 2.0 compliant ;-). It currently tests available conversions, saving and getting data from various dataprovider back ends (Flickr, gnomevfs, Tomboy, etc). It even does code coverage and pushes the results on line.
What's Next
I apologize for this post being mostly academic in nature, lacking in screen shots. I will post again in a few days with some more relevant information (from a users perspective). I just wanted to highlight how and why I believe a desktop sync service is useful, and how I am trying to satisfy this in a sustainable way for the GNOME platform as a whole.
I anticipate finishing up the two way Tomboy support (via iPod notes, gnomevfs location, and backpackit.com) in the next few days. There are some bugs preventing running Conduit on Feisty (goocanvas 0.6.0 is ABI incompatible with 0.4.0). There are also some other lose ends (like some GUI blocking calls in conflict resolution, and some difficulties saving program configuration). Unfortunately, like a lot of FOSS, the best is always just around the corner....
Feb 19, 2007
Hey Planet GNOME'rs
Thanks Jeff for putting me on Planet GNOME. My name is John Stowers and i'm currently splitting my time between a few things that might be of interest to people here.
-
Synchronization and the GNOME Desktop
I probably spend most of my free time hacking on Conduit, which is a synchronization application for GNOME. I hope to provide a (DBus) service where application authors can use Conduit for their individual sync and export capabilities, and don't have to keep reimplementing them in their own applications. Furthermore as a stand alone application I aspire to the ease of use of Apple's .Mac while supporting core GNOME technologies.
More information
-
Conduit Website
-
Slides and Video from a talk at Linux.conf.au
-
The next release is scheduled for around the same time as the release of GNOME 2.18 and will support that favorite requested feature coughtomboy synchronizationcough. Thanks Boyd.
-
A Metadata Enabled GNOME
Im pretty excited about Tracker and the ability to move away from a folder centric GNOME. By embracing metadata, tagging, and a desktop indexer I think that we will leap past our competitors in terms of integration between applications. To this end I am hacking on;
-
Nautilus + Tracker integration
Injecting some love into the nautilus emblem functionality. Support tagging and attaching emblems (emblem = a tag with an image) from either nautilus or Tracker.
-
Nautilus metadata integration
Working with neilj on a nautilus-on-steroids + libtracker-gtk. Lets make metadata visible (in nautilus) and easy for application authors to add to their apps (libtracker-gtk)
-
GtkFileChooser + Tracker
Attach tags and notes at save time (whats the state of the filechooser extension spec?)
Other than that Im 23 years old, just finished a masters and electrical engineering at the University of Canterbury. Im really an electronics nerd so im also hacking on Albatross Unpiloted Aerial Vehicle. Im doing some traveling around Europe for the next few months and then probably starting a phD.
Jan 25, 2007
Well I had a fantastically great time at linux.conf.au. I was fortunate enough to present two talks at the conference;
But I think the coolest thing was meeting heaps of interesting (and impressive) people including Jono, Eugene, Manish, Andy, Davyd, Karla, Nigel, Rafael, etc, etc. I will put some more photos and some of my new BHAGS for Conduit and Albatross up soon.
Jan 6, 2007
So its been a while since I have posted but there is some interesting news to report;
-
Conduit has a new developer - confusingly also named John
-
I am speaking at Linux.conf.au both at the GNOME Miniconf and the Main conference
-
I have some interesting Topaz style demos to show soon
-
Conduit 0.3 is going to rock
-
Opensync will be supported (to some degree)
-
The UI is starting to really ROCK, and improved treeview to display dataproviders, and improved canvas performance.
-
I have been working on a python binding to evolution. It is starting to reach the state of being useful and will be supported in the next release. I hope that this library could one day be included in GNOME.
Sep 25, 2006
So I finally got around to releasing Conduit 0.2.0, which, while having many limitations, is relatively useful and safe to use on a daily basis. I have been swaped by university work at the moment, as this is the last week, but I hope to be back developing again soon.
Other Things on the hack-agenda over the next few weeks (months)
