Johns Blog

More Conduit GSOC Ideas

I see that Google has extended the SOC application deadline. Here are some Conduit related SOC ideas for GNOME.

• Port Tomboy sync to use Conduit (and get free support for $WEBSERVICES) Use Conduits DBus interface and our C# bindings to said interface to be able to configure and initiate synchronization from Tomboy. This means that peer-to-peer Tomboy sync will get easier (no more ssh fuse), and that support for additional websites/mobile devices will become available.

• Port F-Spot photo export to use Conduit With the exception of mentaloo gallery, we support most/all of what F-spot can export to. I can count a number of times where F-spot has hung/crashed during photo export. Using Conduit to do the sync will help prevent this, and ensure that if something untoward does happen, when you restart the sync duplicate photos will not be uploaded.

• Mobile phone support

  • Integration with gnome-phone-manager for phone discovery and/or pim data Get/Set

  • Use of python-gammu for fetching PIM data, and photos from those devices that do not support obex-ftp

• Gstreamer based media transcoding We currently call FFMPEG or mencoder via command line to convert/scale video and audio files. I would like to use gstreamer. A possible solution to this would be to create a gstremer transcoding utility (GUI and command line), or the use of the python gstreamer bindings from within Conduit. While I prefer the latter, the former would be a useful addition to the GNOME desktop.

• Port cheese to use Conduit for photo and video site upload This would be based upon out glib dbus bindings.

• More GNOME plugins using out DBus interface

  • (finish) Eog plugin for photo site upload

  • Add video upload to Youtube and Vimeo from Totem

  • Better nautilus integration. Removable volume support has improved in Conduit. It would be good to expose this from nautilus, although I am not sure the role this would take, for example

• Support windows mobile devices. We have preliminary support for SyncCE, but this was never completed. There are capable python bindings to SyncCE, so it would take a hacker with a WM5/WM6 device to finish this.

• Support palm pilots. Once again, not something I can work on as I do not have a device. There are some python bindings for getting data from Palm devices, and there is also the possibility of wrapping the GNOME pilot code to enable it to be used from Python.

New Conduit Features

Last week (or so..) I released Conduit 0.3.9, the notable features of this release were the addition of documentation, and dramatically improved support for removable devices, like USB keys and portable hard drives.

Conduit has always supported removable disks, but the UI for working with them has been dramatically improved. In the screenshot below you will see Conduit automatically suggesting pre-configured dataproviders to sync with folders on the removable device. The use case for this is

1. Joe synchronizes stuff from one PC to his USB key.

2. When Joe plugs that USB key into another computer also running Conduit he is presented with a pre-configured method to synchronize information from the usb key to a location on his local machine.

The functionality is not all that different to before (it shares all the same code as the traditional folder <--> folder sync, meaning you get conflict detection etc) but the way it is now presented is more sensible and discoverable.

In the screenshot you will also see the first commit of mobile phone support in Conduit. Phone discovery is done using bluetooth. Picture sync is implemented atop gnomevfs-obex-ftp. Data is currently fetched from the phone using gammu, but discussion are underway on how this might tie into gnome-phone-manager (which uses gnokii).

This work is (mildly) separate, but necessary for a sensible SyncML (using libsyncml) implementation. In order to provide smart suggestions on what sync options to offer for a particular phone, phones in range (via bluetooth or USB connection) should be enumerated and their capabilities queried. Currently I am using bluez for the former, and gammu for the latter, however, at least in the bluetooth case, the latter could probbably be accomplished using pure AT commands over a serial socket to get the manufacturer and model number and them comparing this with a database of capabilities.

If its not clear, this is still in the experimental stage, but the current approach seems to demonstrate that Conduits architecture, and the approach to implementation are approximately correct.

There have also been a few other fixes go into SVN, including

• The ability to create tags in F-spot from within the Conduit UI

• Improved configuration for picasa web galleries

Finally I will finish with a dear lazyweb.....

Dear Lazyweb; The current implementation of network sync doesnt really scale to pushing large files around. Its using the python pickle module, which has a tendency to read everything into memory, base64 encode it, run out of memory, and then explode. I have considered may ways to solve this

• Use sftp - but that requires the user setup a ssh keypair with the other computer. Seahorse, can you give me a dbus interface to make this happen, easily?

• Re-implement (because the license is CPL), some of the ingenious performance optimizations in conary WRT decreasing XMLRPC memory usage.

• Wait for the GNOME desktop to offer some sort of GiveFileToThisUser() dbus api call. Could perhaps be implemented atop Giver/Telepathy (if... it gets accepted to the desktop....)

• _Custom python module written in C to do this.... _

• Continue doing what I am doing, and wait for people to complain louder....

Summer of Code

Work on Conduit is progressing nicely, squashing bugs that have appeared when interacting with the new versions of GNOME applications. I have kind of been tracking the GNOME release schedule (congratulations on the latest release BTW), so I expect to put out another Conduit release this week.

I intend to keep making stable bug fix releases until there are Python GIO bindings. At that time it would probably make sense to branch so that I can begin targeting 2.24, and have the opportunity to land some more invasive changes.

The current bugs that are being hacked on (i.e. have patches), depending on how the implementation turns out, may appear in $NEXT_STABLE_RELEASE, or in $FUTURE_RELEASE.

• #521196 Sync FSpot photos with changed captions/tags

• #511691 Improve removable volume/folder support

• #520668 Mobile phone support

• #520699 Support Google contacts using gdata

Summer of Code Projects I create this page on the GNOME wiki which includes some summer of code ideas for Conduit (and by extension, synchronization in the GNOME desktop).

Disclaimers:

1. I am eligible, and will be applying to participate in summer of code this year.

2. I wont be offended if people apply to do the same tasks that I am going to be proposing. The more the merrier!

3. Im not sure how the whole mentoring regulations work. Can I mentor and be a student?, are there mentors in the GNOME community who would like to champion Conduit integration in their application?

Coming Soon to an Internet Tablet Near You

• Multiple site photo upload

• PIM sync to your desktop (including over the air sync using our network sync support)

• File/Folder sync to your desktop and to online services such as box.net

• The same codebase as Conduit for the desktop

• The awesome work of Thomas Van Machelen

• Did I say shiny?

They come in pairs

Following the tradition of the last Conduit release, I realized that I shipped Conduit 0.3.7 with a stupid build issue that broke all the Gmail/Google/Picasa support for people. Following in that tradition here is Conduit 0.3.8 which fixes that, and also features updated icons contributed by mejogid.

For more information:

• Release Notes

• Download

Conduit developer John Carr has also setup a Ubuntu PPA for Conduit. Check it out if you are interested in running the Latest and Greatest for Ubuntu Gutsy and Hardy.

Conduit 0.3.7: Many Small Updates

I just released Conduit 0.3.7. This was another regular release which contains the usual array of bug fixes and a few small new features and tidy ups. This was also the first release made automatically using maintainer. Aside from a few quirks, the tool worked amazingly well.

The next release will feature rewritten user documentation, more conflict UI improvements, and fix whatever bugs crop up in testing its compatibility with new GNOME. Highlights of this release are;

• Update to use the newer Tomboy Set/GetNoteComplete API. This finally allows lossless sync of Tomboy notes including metadata such as tags and notebooks.

• Update to the latest versions of our shipped static libraries

  • flickrapi (fixes many Flickr bugs)

  • pyfacebook

  • python-gdata (and use this for Picasa support)

  • libgmail (make emails work again)

• The Eye of gnome plugin for uploading photos to multiple sites now shows status while synchronizing

• Add one-way support for syncing Gmail contacts (with evolution, ipod, etc)

• Add a 'developer' sub-menu to the help menu if the user is running a development version. This sub-menu contains items which link to the newly improved developer documentation. The linked pages are displayed in conduit's built in web browser.

• Make the Conflict preferences per Conduit add descriptive icons to the preferences menu

Update: I completely forgot. Thomas Van Machelen recently added C# bindings for our DBus API to SVN. If any of you enterprising Novell Hack Week hackers wants to play with them, I have some ideas (use Conduit to export photos from FSpot, use Conduit as the sync engine for Tomboy, for both synchronization of notes between two computers and/or export to ipod, box.net, etc). Please email me if you are interested. I know Conduit is ready, there are many examples of the Conduit DBus API in SVN.

Linux.conf.au Talk

Just a quick note to say that I'm giving a talk on Conduit at 9:20am Tuesday 29th of January at GNOME.conf.au. Come along if you are interested. I will be going through some demos showing the many ways you can use Conduit including;

• As a multiple site photo uploader

• To keep your Tomboy notes in sync

• As a file/folder sync tool

• Keep your GConf preferences in sync over multiple computers

• Keeping two computers in sync over the local network

• Sycing with removable devices like Nokia n800 and iPods

• As a PIM sync tool supporting a number of online services

I will also be demonstrating how to write your own dataprovider, and hopefully I will have time to discuss more difficult issues such as

• Furthur integration with GNOME (Roadmap to 2.24)

• SynCE/Opensync/SyncML

• Progress on the Maemo/Hildon port

Shape of things to come

I just committed the first version of a working GObject binding to the Conduit DBus API. You can check out an example test program here.

Contrived Example-like Code

   /* The running conduit app/daemon */
   ConduitGApplication *application = conduit_g_application_new ();

    /* Get two dataproviders, a datasource and datasink */
    ConduitGDataprovider *source = conduit_g_application_get_dataprovider(application, "TestSource");
    ConduitGDataprovider *sink = conduit_g_application_get_dataprovider(application, "TestSink");

    /* Put them in a conduit so we can sync them */
    ConduitGConduit *conduit = conduit_g_application_build_conduit(application,source,sink);
    g_signal_connect (conduit, "sync-progress", (GCallback) _sync_progress_cb, NULL);
    g_signal_connect (conduit, "sync-completed", (GCallback) _sync_completed_cb, NULL);
    g_signal_connect (conduit, "sync-conflict", (GCallback) _sync_conflict_cb, NULL);

    /* Add it to the GUI so the user can see it (optional) */
    ConduitGSyncset *gui_syncset = conduit_g_syncset_new(application, "/syncset/gui");
    conduit_g_syncset_add_conduit(gui_syncset, conduit);

    /* Perform the sync (boom!) */
    conduit_g_conduit_sync(conduit);
    g_main_loop_run (mainloop);

Conclusion

• A GObject binding to the Conduit DBus API means you can synchronize your application data, from within your application, in very few lines of code;

  • Your application is not going to crash

  • Your application is not going to hang or block the UI

  • The difficult sync logic for interfacing with various other places can be written in Python (faster, more productive, most web-services get python bindings first), live in the Conduit source tree, and immediately be available to all other applications which use Conduit to sync stuff.

• In SVN NOW!

• Hat-tip to Mikkel Kamstrup Erlandsen whose Xesam Glib bindings provided a useful starting point, and saved me much autotools pain.

• Yes, the names are a bit long.Who has a better idea for the prefix, gcond_ perhaps?

• Who wants to help me integrate this into;

  • Cheese

  • Evolution

  • GNOME Phone Manager

  • $YOUR_APP_HERE

Conduit 0.3.6: I can has sleep please?

Quote courtesy of John Carr, who I had cook me up a patch at 1:30AM UK time. Ahhh the wonder of a 12h time difference.

I just released Conduit 0.3.6, a quick follow up to the last release. It fixes a number of bugs which were preventing some common sync scenarios.

• Download

• Release Notes

• Report Bugs

• Mailing List

• Test Results

• Bugs Fixed

  • Fix two-way Tomboy <--> folder sync

  • Fix two-way GConf <--> folder sync

  • Dont recreate the backpack API everytime we refresh

  • Fix Shutterfly to support image uploads and delete again (Jeremy Slater)

  • Fix attribution of Banshee dataprovider. It was contributed by Don Smith

  • Many fixes to image dataproviders

OK I lied, there is one new feature..

Back To Reality

Conduit 0.3.5: Enough Excuses

I am a terrible project maintainer. It has been a full 3 months since the last Conduit release, and I have run out of excuses. First I got distracted by online desktop shenanigans, then by Opensync and Ubuntu things. I got caught up in moving Conduit to GNOME SVN. Then I got distracted by JHBuild on windows. Finally I lost my nerve and went on holiday, not touching a computer for 3 weeks. Enough is enough!

The good news however is that there is no doubt in my mind that this is the best Conduit release ever. I'm using Conduit on a daily basis for a variety of tasks, and its looking really solid. Yes there are still some bugs, but thats life. So, no screenshots this time, just a list of shiny new features. Over the next few weeks I will be doing a series of "look what you can do with conduit" type posts, as this is probably the best way to demonstrate to folks what they can accomplish with a generic sync+conversion framework, with a focus on integration with the desktop environment and great support of online services!

• Release Notes

• Download

• Report Bugs

• Whats New

  • Move to GNOME SVN and Bugzilla

  • In memory mapping DB has been replace with sqlite based one. Should reduce memory usage during large sync's.

  • Restructure code layout to allow Conduit to be used as a library by other python programs.

  • Removed Gtk dependancy in core. Conduit command line can now be run without Gtk installed.

  • Added 'always up to date' infrastructure for automatic syncing.

  • Improved n800 support for syncing audio, video and photos to the device.

  • New AudioVideoConverter for transcoding media types (uses FFMPEG and/or mencoder)

  • Include Banshee support in configure.ac - from lool

  • Support data hashes in addition to mtimes as a means for detecting changes

  • Remove twisted dependency for network sync. Uses the built in python xmlrpc module

  • Use python logging module for better logging granularity

  • Make HAL easier to extend in dataprovider factories

  • Files that did not load correctly are shown in the preferences window to help debugging

  • Experimental maemo/hildon UI (Thomas)

  • Added iPod photos support

  • Write support in f-spot using its new dbus interface (Thomas)

  • Support tags in all datatypes (useful for images)

  • Incredible improvements in test coverage. 75% lines of core code tested (including close to 100% of the core sync engine)

  • RSS Feed source now supports video enclosures

  • Many many bug fixes as a result of test focus

As more people start to use conduit I am becoming more conservative and focusing a lot more on testing. One aspect of this is a machine I have set up that runs the test suite every time there is a commit. The other aspect is more regular releases. For the former look here, for the latter please test and give me feedback!

Team conduit is well poised for a 0.4.0 release to coincide with GNOME 2.22.

Holidays

It feels strange to be back at university after a solid 3 weeks of doing very little. I had the best summer holiday I have had in a long time. Considerable thanks must go to the weather in New Zealand, its great to finally have the good weather in December. Anyway here are some holiday snaps for those that are curious about what a summer Christmas/New Year in New Zealand looks like.

My Christmas Present

Relaxing on the Jetty

Sunset at the Bay Of Many Coves

Hunter Gatherer