Johns Blog

Back To Reality

Conduit 0.3.5: Enough Excuses

I am a terrible project maintainer. It has been a full 3 months since the last Conduit release, and I have run out of excuses. First I got distracted by online desktop shenanigans, then by Opensync and Ubuntu things. I got caught up in moving Conduit to GNOME SVN. Then I got distracted by JHBuild on windows. Finally I lost my nerve and went on holiday, not touching a computer for 3 weeks. Enough is enough!

The good news however is that there is no doubt in my mind that this is the best Conduit release ever. I'm using Conduit on a daily basis for a variety of tasks, and its looking really solid. Yes there are still some bugs, but thats life. So, no screenshots this time, just a list of shiny new features. Over the next few weeks I will be doing a series of "look what you can do with conduit" type posts, as this is probably the best way to demonstrate to folks what they can accomplish with a generic sync+conversion framework, with a focus on integration with the desktop environment and great support of online services!

• Release Notes

• Download

• Report Bugs

• Whats New

  • Move to GNOME SVN and Bugzilla

  • In memory mapping DB has been replace with sqlite based one. Should reduce memory usage during large sync's.

  • Restructure code layout to allow Conduit to be used as a library by other python programs.

  • Removed Gtk dependancy in core. Conduit command line can now be run without Gtk installed.

  • Added 'always up to date' infrastructure for automatic syncing.

  • Improved n800 support for syncing audio, video and photos to the device.

  • New AudioVideoConverter for transcoding media types (uses FFMPEG and/or mencoder)

  • Include Banshee support in configure.ac - from lool

  • Support data hashes in addition to mtimes as a means for detecting changes

  • Remove twisted dependency for network sync. Uses the built in python xmlrpc module

  • Use python logging module for better logging granularity

  • Make HAL easier to extend in dataprovider factories

  • Files that did not load correctly are shown in the preferences window to help debugging

  • Experimental maemo/hildon UI (Thomas)

  • Added iPod photos support

  • Write support in f-spot using its new dbus interface (Thomas)

  • Support tags in all datatypes (useful for images)

  • Incredible improvements in test coverage. 75% lines of core code tested (including close to 100% of the core sync engine)

  • RSS Feed source now supports video enclosures

  • Many many bug fixes as a result of test focus

As more people start to use conduit I am becoming more conservative and focusing a lot more on testing. One aspect of this is a machine I have set up that runs the test suite every time there is a commit. The other aspect is more regular releases. For the former look here, for the latter please test and give me feedback!

Team conduit is well poised for a 0.4.0 release to coincide with GNOME 2.22.

Holidays

It feels strange to be back at university after a solid 3 weeks of doing very little. I had the best summer holiday I have had in a long time. Considerable thanks must go to the weather in New Zealand, its great to finally have the good weather in December. Anyway here are some holiday snaps for those that are curious about what a summer Christmas/New Year in New Zealand looks like.

My Christmas Present

Relaxing on the Jetty

Sunset at the Bay Of Many Coves

Hunter Gatherer

Jhbuild Anything on Windows - In 12 Steps

Part 1: The Complete Gtk+ Stack from Source

Its been a long week spent watching things compile [0], but I'm happy to report that Jhbuild is now able to build the complete [1] stable Gtk+ stack on windows using the msys/mingw tool chain. That literally means that you can now build you Gtk app, or Gtk itself, for windows, in 12 steps. You still getting all the normal benefits that Jhbuild allows. So, the steps are;

1. Install MinGW combined installer (MinGW-5.1.3) to C:\mingw

2. Untar msysCORE-1.0.11-2007.01.11 to C:\msys

3. Run C:\msys\postinstall\pi.bat, answer yes when it asks whether mingw is already installed and point it to C:\ming

4. Install Python 2.5.1 to default location (C:\Python25)

5. Install ActivePerl to C:\Perl. You can probably also install a MingW-provided perl package, but the latest version of this was some 5.6 at the time of this writing, while ActivePerl is 5.8.

6. Install bzr for windows to C:\Bazaar

OK thats all the preparation steps. Launch C:\msys\msys.bat now to start a minimal unix-like shell. Now onto the meaty part;

1. bzr branch http://www.gnome.org/~jstowers/bzr/jhbuild-win32

2. cp jhbuild-win32/win32-hacks/* . && mv profile .profile && mv perl.sh /bin/perl

3. edit pkg-config.sh to reflect the prefix you are going to install (or have installed) pkg-config to

4. edit jhbuild-win32/configs/gtk-win32.jhbuild so that full unix paths reflect the correct location of pkg-config.sh. Make sure /bin/perl and pkg-config.sh are executable

5. Close the shell and restart it so that it picks up your new .profile. Check that you can run all needed executables perl --version && python --version && jhbuild --help && ./pkg-config.sh --version

Oh, I seem to be forgetting one more thing.... The following command will build gtk+,atk,cairo,pango,zlib,libpng, and glib from source. No really, that is it [0].

1. jhbuild -f jhbuild-win32/configs/gtk-win32.jhbuild build

The Hacks (aka things I don't really understand)

There are a few parts of this setup that still confuse me.

• /bin/perl is a thin wrapper around perl that quotes args and escapes path slashes. Its necessity is based around the interaction of the msys path escaping and what happens when autotools invokes glib-mkenums. It was based upon what I read here. It also writes a debug log to ~/perllog

• ~/pkg-config.sh is a wrapper around pkg-config.sh that quotes args. It is as described here but with a full prefix to pkg-config (so that you may use one that is installed in your jhbuild prefix). It also writes a debug log to ~/pkglog

Jhbuild Core Modifications

I am currently hosting a bzr branch with my modifications to Jhbuild. I'm still cleaning some things up, but I hope that this is suitable for committing to SVN in the future. My modifications to Jhbuild are basically;

• Cleaning up all path handling code. That means using os.path.abspath and os.path.join internally. The msys does shell does some shiny things to convert unix paths to windows paths but it works best if the path it is passed is coherent and not a mix of both systems path separators.

• Some path and environment variable fudging that checks all paths that get passed out to a shell, and converts them to unix format. Python on windows calls the windows CreateProcess function (that expects a windows executable). Therefor we starts a new msys/bin/sh.exe with the first arg the script we originally wished to start, followed by all the inferior args.

• Added a binary module type. This is basically like a tarball but instead of executing configure, make, make install you can define arbitary shell commands to execute post download and unpack. This is used to provide glib-bin and gtk-bin metapackages (which get binaries from Tor's gtk win32 site). You can depend on the metapackages if you dont feel like compiling the gtk+/glib stack yourself and just want to build your app. You can also insert these binary modules in the dependency chain and use them to execute arbitrary scripts to fix (read: hack) some aspects of packages.

• New Gtk+ and Glib modules files for binary installation, and source installation.

• Use pythons build in functions to download files (instead of wget) and to untar/unzip archives (instead of tar and unzip command line programs). Note that this is not strictly necessary as there are builds of tar/unzip and wget for msy - but its easier from a developer perspective.

Remaining Work, Caveats (aka the full story)

• libjpeg builds ok but doesn't seem able to generate a shared library. Hence libtiff fails and gtk builds without jpeg support.

• glib-mkenums seems to insert arbiraty data (a directory listing) at the end of .h files. This results in many compiler warning about data after #endif.

• Read the gtk-win32.jhbuildrc file.

• Gtk doesn't set the correct paths to the pixbuf loaders and input modules. You will need to run (from the prefix to which Jhbuild installed everything)

  • $JHBUILDPREFIX/bin/gtk-query-immodules-2.0.exe > $JHBUILDPREFIX/etc/gtk-2.0/gtk.immodules

  • $JHBUILDPREFIX/bin/gdk-pixbuf-query-loaders.exe > $JHBUILDPREFIX/etc/gtk-2.0/gdk-pixbuf.loaders

• You can change gtk to use the default theme by doing

  • echo 'gtk-theme-name = "MS-Windows"' > $JHBUILDPREFIX/etc/gtk-2.0/gtkrc

• I have not tested this with SVN modules on windows, only tarball releases. I wasn't prepared to discover the compatibility of the msys provided autotools stack (yet).

• --disable-docs on which/all modules?

• Hookup to, and submit build reports to JhAutobuild

• Python or msys or something buffers all the shell output. This means that there can be long periods (minutes) of nothing getting printed to the shell. If it looks like something has failed to build (no CPU/disk activity) the push 1,3 or 4 to retry, cancel, or drop into a shell respectively (i.e. the normal Jhbuild menu presented upon failure).

Part 2: I Just Want to Build my Application (not Gtk)

It should be noted that it is not necessary to build Gtk (from source) if you just want to build a single application. Using the new binary module type, I have prepared a gtk-binary metamodule that downloads all the needed files from Tor's site, and installs them for you. That sounds confusing, so lets take a randomly chosen example program [2]. Lets install medit, from source, using Jhbuild.

Starting with the example configs/gtk-win32.jhbuild file change the module and modulesets lines to

moduleset = [
    os.path.join(thisdir,'gtk-win32-bin.modules'),
    os.path.join(thisdir,'meddit.modules')
]
modules = ['meddit']

Add a module_autogen line for meddit

#Module specific configure arguments passed in addition to autogenargs
module_autogenargs['meddit'] =  ' --without-xml --disable-help --with-python=no'

And the final step....

jhbuild -f jhbuild-win32/configs/gtk-win32.jhbuild build

Conclusion

I got into this because I wanted to be able to build pygoocanvas on windows, so I could hack on Conduit there. I still have not done that, instead I have;

• gotten royally distracted from the original goal,

• hacked on Jhbuild,

• learned a lot about msys/mingw and autotools on windows,

• hopefully been able to contribute something useful back to the community that makes it easy for joe normal and jane developer to test their app on windows

• Oh and its not yet finished (but I have tested it a few times on fresh installs, and it should just work) yet

• Pretty normal summary for a FOSS hackers week, correct?

I would really like help from some people in testing this further, so go ahead, test it today. Ive made it 12 easy steps for you to follow.......

References (thanks)

• Building Gtk+/Gimp on Win32 (May 2005, Tor Lillqvist)

• Building Evolution on Windows (Feb 2007)

• An attempt at Win32 build instructions (Jan 2005, Owen Taylor)

• Compiling Glom on Windows (Dec 2007, Armin Burgmeier)

• pkg-config.sh (Jun 2007, Cody Russel)

• Circular deps of pkg-config (Tor Lillqvist)

[0] A full Gtk+ build from source takes about 6 hours on Windows XP (running in VMware getting both 2.4Ghz Pentium 4 cores)

[1] Ok, not the complete stack. To get around some circular dependency issues I start with binary versions of pkg-config iconv and gettext. With pkg-config present I then build Glib from source. I also use a binary version of poppler because it depends on Gdk while Gtk depends on poppler.

[2] Well not entirely random. It was already advertised as being able to builad on windows. Also it failed during make install, complaining about missing share/mime/packages/freedesktop.org.xml and bin/update-mime-database. I created both as empty files and it was successfully installed.

Jhbuild and Windows - We Meet Again

The Good News

Following on from my last post I have continued work on getting Jhbuild to run on windows (within msys). The basic problem which I identified in my last post was the struggle between Python (for Windows) expectations about paths, and the hoops that msys and cygwin jump through to allow unix style paths to be used. Basically os.path.join and friends do the correct thing providing the subsequent path is used within python alone. Things get all difficult when that path is then passed out to a configure script, or as an argument to a shell command. With that in mind however, a picture speaks a thousand words....

1. It works...

2. On windows...

3. Through magic....

The Status

Jhbuild is successfully able to build a test moduleset, and even some real modules such as zlib and iconv on windows using the mingw compiler. Things are looking promising and I will continue to play with this tomorrow. Its a bit of a pain to debug because of the inconsistancies between how subprocess.Popen behaves on windows and on linux. So far my changes can be summarised as follows

1. Add Makefile.win32 and patch install-check to just call /usr/bin/install

2. Replace wget/curl with pythons built in urllib.urlretrieve

3. Replace tar/unzip with pythons built in tarfile and zipfile modules

4. Misc fixes to get Jhbuild to start including disabling the tray icon etc

5. Refactor buildscript.execute (the subprocess.Popen) wrapper to clean it up and to special case local script commands from system commands (because of differences in how win32 treats executing executables in the current dir). Best summarised as the following

   def execute_script(self, command, args_str='', cwd=None, extra_env=None): # THIS IS THE MAIN WIN32 HACK if sys.platform.startswith('win') and command[0].startswith('./'): command = ['sh', command[0].replace('./','')] + command[1:]

Jhbuild still works on Linux just as it used to (more or less, I need to port over some more things to the new buildscript.execute() functions). Wget/tar/curl/unzip etc are still used on Linux. In general the changes so far have been self contained and should not break regular operation

The Install Procedure

How can you try this pre-alpha code out?. Follow these steps on windows and check out from my bzr branch (or use the patch against SVN)

1. Install MinGW combined installer (MinGW-5.1.3) to C:\mingw

2. Install msysCORE-1.0.11-2007.01.11 to C:\msys

3. Run C:\msys\postinstall\pi.bat, answer yes when it asks whether mingw is already installed and point it to C:/mingw. Launching C:\msys\msys.bat now provides us with a minimal unix-like shell.

4. Install Python 2.5.1 to default location (C:\Python25)

5. Install bzr for windows to the (C:\Bazaar)

6. Start msys.bat (in C:\msys)

7. Install Jhbuild

   1. bzr co http://www.gnome.org/~jstowers/bzr/jhbuild-win32 jhbuild-win32

   2. cd jhbuild-win32

   3. make -f Makefile.win32 install

8. Add python to path. Other path manipulation is done within JHBuild. Add the following line to ~/.profile

   export PATH=$PATH:/c/Python25:/c/Python25/bin

You should now have a minimal msys install with JHBUILD. Test by performing

• cd ~/bin

• python jhbuild --help

Finally, you can now install the test modules with

python jhbuild -f jhbuild-win32/test.jhbuildrc build test

Standard disclaimers apply, alpha quality code, help appreciated, yada yada yada. Now back to Conduit hacking....

Jhbuild Adventures on Windows

Background: I'm going home to my parents house for Christmas, and because I don't have a laptop, I thought it would be the perfect opportunity to use their computer to get Conduit working on windows. However I could not find windows builds of (py)goocanvas anywhere. Never minding a challenge I thought it would be a good experiment to see if I could build pygoocanvas for windows via cross-compilation or natively - both using jhbuild.

I like to think that F/OSS works because people keep re-inventing the wheel in new and obscure ways everytime they do something, so what follows is my rambling tale of cross-compiling-gtk-for-windows wheel reinvention.

Cross compiling Gtk+ for Windows using Jhbuild

Starting with the work of Alberto Ruiz and Piotr Gaczkowski I thought I would setup a jhbuild environment to cross compile the Gtk+ stack. After a bit of fiddling I managed to get everything to build fine using mingw32 and jhbuild. The following files work successfully on Ubuntu gutsy and can build the latest stable releases of Gtk+/cairo/glib/libglade.

• gtk+-win32-john.jhbuildrc

• gtk+-win32-john.moduleset

At this point I moved onto trying to build pygobject. This is where things got difficult. I tried a few different ways to get the cross compilation environment to see python. With some inspiration from this page on compiling glom for windows, I copied across the entire Python25 directory from a windows box and added it to the appropriate LDFLAGS and CFLAGS in jhbuild. This allowed pygobject to pass configure. However it now failed to build;

In file included from /usr/include/python2.5/Python.h:57,
    from pygobject.h:5,
    from pygobject-private.h:9,
    from gobjectmodule.c:27:
    /usr/include/python2.5/pyport.h:246:24: sys/select.h: No such file or directory

I went back to the windows machine and checked to see if sys/select.h was present in mingw. No luck. OK at this point I was stuck. Compiling wasn't looking likely, and I still wasnt even confident that if it did compile it would manange to actually link and run (due to different compiler versions and platforms, etc). I googled for a bit and saw that its now possible to build python extensions using mingw32 and that people had managed to cross compile python using mingw. Hmm, I pondered..... 'lets cross compile Python25 using mingw and then build pygobject using that.' Furthur googling revealed this python bug+patch for cross compiling using mingw. This is going to be easy.......

I added a new section to my modules file that would apply the patch to the Python 2.5.1 before building it. YOU FAIL AGAIN JOHN.

checking for %zd printf() format support... configure: error: cannot run test program while cross compiling

Hmm. I tried several strategies to get around this (including supplying a config.cache file with ac_cv_printf_zd_format=yes) but with no luck. You can see my plea for help on the bug report.......

At this point an entire day of my life had been irrevocably taken from me. I retired to bed. I would not let windows beat me....

Native compilation of Gtk+ for Windows using Jhbuild

OK. At this point I thought I might as well start again, but compile natively on windows using mingw and msys. In hindsight it probbably would have been easier to follow most of the instructions for glom verbaim, but I decided to continue my quest at wheel re-invention.

I read that Alberto was thinking about ways to make Jhbuild work on windows - a good goal. Starting with his patch on that bug report I modified it slightly so that tar, gunzip and friends wouldn't need to be installed. Furthur to that I set about hacking up some patches to allow jhbuild to at least run on windows. A few hours later and I was sure that this was going to take a few more hours :-). I now had a working (ish) minimal easy to set up windows build environment according to these instructions..... or so I thought.

Jhbuild managed to successfully parse my minimal jhbuildrc and module files, download and untar iconv but then it failed. Basically there is considerable work and thought needed to see how this should operate in msys/cygwin. The problem is already explained in part here, but I will summarise.

Python's os.path and friends do a good job at converting paths to the native system format. Unfortunately the msys and cygwin shells also convert paths. For instance msys automagically converts /foo/bar to C:\msys\foo\bar. Basically it means that python converts paths to windows format, but msys likes them in unix format (because its a unix shell environment). This results in files not getting found, and gross composite paths like --prefix=C:\foo\bar/baz/bob getting passed to configure.

So, I had a bit of a look at how this could be solved in Jhbuild. I guess there are a few strategies

1. Create a whole bunch of utility functions join_path(), prepend_path(), etc. And call these instead of the system ones and only escape to windows format if you are running in windows, and not running in windows just kidding you are really running in cygwin haha.

2. Remove all os.path and friends calls. Be really naive and hard code all path operations to use unix semantics. Then explictly note that Jhbuild on windows must be run from within a unix shell like msys/bash or cygwin/bash.

3. Magic. Come up with some sort of paranormal event that would win GNOME $1 million dollars which could then pay a developer to fix all this for me.

Conclusion

Well things didn't work out as well as I hoped. I will probbably go and compile everything without jhbuild and be happy. I did learn that;

• Jhbuild on windows is a worthwhile cause. It would really help in automatically (jhautobuild) testing things like glib and GIO (at least giving devs a compile yes/no signal)

• It could make generating the latest tarballs really easy, perhaps even sharing the same modulesets as those maintained by the release team

• For extra bling it could even automatically create installers by tracking installed files using a strategy similar to checkinstall.

Help, thoughts or advice appreciated. For all the luck I have had the last few days I would not be surprised if I have missed something fundamentally obvious to this whole process. Heck if some fairy was to drop a pygoocanvas installer for windows in my mail box then I would be super happy. Oh and did I mention that im really looking forward to gio!

The Big Move

The short news is that Conduit now lives in GNOME SVN. The move took a little longer than I would have hoped, but thats largely my fault. I would like to express my thanks to Olav Vitters for importing the SVN repository, his excellent response time to my emails, and the awesome mango system he created.

Whats Good

Conduit trunk is now pretty much feature frozen. It feels really good to finally cross off the TODO list and fix some of the bugs that I created (due to some poor choices when I started the project). While I will blog again soon about the shiny new features for the moment I will talk about one useful piece of code that I am proud of, and that may be useful to others.

Initially I used an in-memory python dictionary based database to store the object mappings which represent a sync relationship between two pieces of data (the mapping DB). I pickled this to disk when conduit closed. Unfortunately this was not only horribly gross (memory usage was huge), it was embarrassingly short sighted, as the one place I care most about types is this code path. We use dictionaries and object hashes extensively in the sync logic and its really hard to debug sync problems when you (for example) don't notice that a dataprovider returns the wrong type until you unpickle it a few days later.

To remedy this I needed (wanted) to use a database that had few or no external dependencies, that was a bit tougher on type safety, and that was usable from multiple threads. After a while I decided that because sqlite was now part of python2.5 this would be a good choice, and that I could get around the threading issues some other way.

Not wanting to re-invent the wheel completely I started with a few existing pieces of code

• Christian Hergert had written a nice Gtk.TreeModel wrapper around a sqlite DB http://vwdude.com/dropbox/pystore/

• Louis RIVIERE had written a threadsafe wrapper around sqlite that allows the DB to be accessed from more than one thread http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/526618

I glued these together and created a really nifty (yet another) DB abstraction layer that can be used in either single or multiple threaded applications, and then displayed in a Gtk.TreeView with NO additional effort. Whats more, the DB also has a LRU cache that dramatically reduces the number of calls into the DB when displayed in a Gtk.TreeView. For more information check out the following

• The Database abstraction (single and multiple thread compatible)

• The Gtk.TreeModel wrapper

• An Example of how to use all these parts together

All of this has been really heavily tested and everything seems to work as expected*........ except for the LRU cache, which I broke somewhere - hmm. I should really fix that.

Another n810 Post...

Thanks to the awesome work of Thomas Van Machelen, Conduit has the beginnings of a port to the hildon ui. Whats more impressive, is that unlike me (who owns a n800), Thomas has been doing all this in an emulator. So anyone who reads this, consider this an endorsement of Thomas request for a developer discount code!

Conduit on the n8xx is something I have been looking forward to for a while. The first and most tangible benefits will be giving n8xx users the ability to use Conduits photo sync facilities to upload pictures to all the webservices we support. Beyond this however, there is much more to offer;

• PIM sync using the evolution-python bindings on both the host PC and the n8xx

• File sync to the n8xx without having to plug it into USB and transfer things via the memory card. By using conduits built in network sync (over python XMLRPC), sync is as easy as letting the PC and n8xx be on the same network.

One Thousand

Conduit recently reached 1000 commits, and with it came together some functionality that has been brewing for a while.

• Far far far improved nokia n800 support

  • You can now sync files, photos, music, and videos to your nokia n800 in an intelligent way (i.e. its not just a dumb file/folder sync).

  • The ability to transcode said data to the most appropriate format for the device. FFmpeg/Mencoder is used to transcode audio/video, and gtk.Pixbuf is used to convert photos. The converters use conduits built in dynamic modular plugin system, so if someone wants to write a audio video converter using GStreamer then please contact me.

  • Also, those that are familiar with FFmpeg/Mencoder/Transcoding please feel free to contribute preset encoding profiles (video codec, size, etc). Presets are stored here (video), here (audio) and here (photo).

Another one of those conduit screenshots that always look the same

• Looser dependence on gtk. This means that conduit can be run from a console (-c command line switch) without gtk/goocanvas installed. This reduces the memory usage by about 10Mb!

• Massive refactoring effort to aid more code sharing between dataproviders and the complete decoupling of the GUI from the core application. This allows other guis to written very independently of the rest of the application [1]. Stay tuned for a native hildon gui for nokian800 and/or ubuntu mobile.

• Rhythmbox support. A simple dataprovider has been added that is able to sync rhythmbox static playlists with things. Initially this is used to test sync music with n800 (with the ability to transcode audio formats), but in my ideal world, sync with rhythmbox would involve

  • Improving rhythmboxes DBus interface to support adding/removing songs to the library and querying playlists (including smart playlists).

• Remember because we are gnomevfs aware some cool things are possible. For instance, one can sync + transcode videos/music to/from their phone which is mounted using gnomevfs-obex.

• Work is currently proceeding furiously on

  • Integrating with opensync

  • Removing the twisted dependence, and using pythons built in xml-rpc server for network synchronization

  • Replacing our current in memory database with one based on sqlite

  • Support for iPod photos

  • Test coverage andbug fixing.

Anyway, svn HEAD is pretty rocky at the moment, but please experiment. Once the database changeover has taken place things should stabilise for a 0.3.5 release. This will be a release candidate and then we will focus on bug fixing only.

Ideally we will release a stable 0.4.0 release at or before gnome.conf.au. As an aside, I would be interested in talking to any of the RH online desktop folks at linux.conf.au as I am unable to make it to any of the northern hemisphere conferences on account of me being poor and in New Zealand.

[1] Excluding a dependence on gnomevfs (yuck). Hopefully gio/gvfs will make all my dreams come true.

Google Online Desktop

Stumbled upon this. Discuss..

Unintended Uses

I just returned to my computer to find that nautilus crashed somewhere in the midst of copying my 50GB (1000 odd holiday videos) from my external hard disk to my NAS via FTP. Not wanting to work out where exactly the crash happened and what files were incomplete or still needing to be copied I thought 'hey, I could use Conduit to do a one way sync'. So I did, and it worked.

Its always cool when software you have written manages to scratch an itch that you didn't design it for!

Conduit 0.3.4 - 37% more integrated

Wahoo!, after mad busy weeks of development, I just released Conduit 0.3.4. Check out the screencast (download here). This release is fresh with new features, and heaps of polish and integration work. These new features include;

• Google calendar support (Paul Novotny).

• Youtube dataprovider added (Renato Araujo).

• Photo sites are now two way, you data is no longer locked with one provider. Also see #480754 for improvements to Fspots DBus interface allowing two way photo sync with FSpot.

• Nautilus extension. (disabled by default).

• EOG plugin added. (disabled by default).

• Conduit now supports resizing and converting photos before uploading them to photo sites. This infrastructure means improved support for other types of conversions, such as transcoding music and video, will be added in the future.

• Progress indicator shows percent complete during synchronization.

• Add support for Facebook photo upload.

• Add included web browser for authenticating against web sites.

Website Authentication

The last point deserves some explanation. Currently, to authenticate against some sites the user must log in either once (Flickr), or once per session (box.net). This is difficult when using the system default web browser, if the browser is currently running on a different virtual desktop to conduit, the user often doesnt notice the web browser go to the login website. This is compounded by the different APIs of some sites. Flickr is good, I can just call am_i_authenticated(token) in a loop, and wait for success, or time out. Facebook is not good. Any call involving the facebook token before it has been validated (i.e. user logs in) invalidates the pending login. Because there is no guarenteed way to block in all cases (i.e. until the user logs in and closes the window), things like gnome.open_url() or python.webbrowser() are unsuitable. This leaves me with two options. I can wait a random amount of time, and hope its long enough for the user to log in, or I can write my own web browser that will block until the user closes it. Suggestions for option 3 are appreciated....

I know this is not ideal. I hope something like web-login-browser can solve my needs. Also, I know my gtkmozembed based browser has some (crasher..) issues (maybe related to this). I hope to fix them and/or move to a pywebkitgtk based browser, depending on what GNOME favors.

Here is a screencast showing the latest version and its features. You may download the original here. Apologies for the google video quality.

[googlevideo]http://video.google.com/videoplay?docid=-262844619875208739&hl=en[/googlevideo]

Boston Summit

Unfortunately, owing to my proximity to the opposite side of the world as Boston, I will not be attending the summit. I understand that the GNOME online desktop is on the agenda, and will no doubt be discussed at some length. I hope those involved in the discussion try out conduit before the summit, as it would be a shame if the future of the GNOME online desktop was set without due consideration. This is not a criticism, sour grapes, or anything like that. Its just a reflection of what appeared to happen after GUADEC - which I also did not attend (New Zealand = isolated and distant).

Have fun with Conduit. Please report bugs, join the mailing list, and abuse me at will. We are currently moving to GNOME servers, so if my server goes down from all the traffic I promise it will be the last time.