Jan 17, 2008
Quote courtesy of John Carr, who I had cook me up a patch at 1:30AM UK time. Ahhh the wonder of a 12h time difference.
I just released Conduit 0.3.6, a quick follow up to the last release. It fixes a number of bugs which were preventing some common sync scenarios.
-
Download
-
Release Notes
-
Report Bugs
-
Mailing List
-
Test Results
-
Bugs Fixed
-
Fix two-way Tomboy <--> folder sync
-
Fix two-way GConf <--> folder sync
-
Dont recreate the backpack API everytime we refresh
-
Fix Shutterfly to support image uploads and delete again (Jeremy Slater)
-
Fix attribution of Banshee dataprovider. It was contributed by Don Smith
-
Many fixes to image dataproviders
OK I lied, there is one new feature..
Jan 16, 2008
Conduit 0.3.5: Enough Excuses
I am a terrible project maintainer. It has been a full 3 months since the last Conduit release, and I have run out of excuses. First I got distracted by online desktop shenanigans, then by Opensync and Ubuntu things. I got caught up in moving Conduit to GNOME SVN. Then I got distracted by JHBuild on windows. Finally I lost my nerve and went on holiday, not touching a computer for 3 weeks. Enough is enough!
The good news however is that there is no doubt in my mind that this is the best Conduit release ever. I'm using Conduit on a daily basis for a variety of tasks, and its looking really solid. Yes there are still some bugs, but thats life. So, no screenshots this time, just a list of shiny new features. Over the next few weeks I will be doing a series of "look what you can do with conduit" type posts, as this is probably the best way to demonstrate to folks what they can accomplish with a generic sync+conversion framework, with a focus on integration with the desktop environment and great support of online services!
-
Release Notes
-
Download
-
Report Bugs
-
Whats New
-
Move to GNOME SVN and Bugzilla
-
In memory mapping DB has been replace with sqlite based one. Should reduce memory usage during large sync's.
-
Restructure code layout to allow Conduit to be used as a library by other python programs.
-
Removed Gtk dependancy in core. Conduit command line can now be run without Gtk installed.
-
Added 'always up to date' infrastructure for automatic syncing.
-
Improved n800 support for syncing audio, video and photos to the device.
-
New AudioVideoConverter for transcoding media types (uses FFMPEG and/or mencoder)
-
Include Banshee support in configure.ac - from lool
-
Support data hashes in addition to mtimes as a means for detecting changes
-
Remove twisted dependency for network sync. Uses the built in python xmlrpc module
-
Use python logging module for better logging granularity
-
Make HAL easier to extend in dataprovider factories
-
Files that did not load correctly are shown in the preferences window to help debugging
-
Experimental maemo/hildon UI (Thomas)
-
Added iPod photos support
-
Write support in f-spot using its new dbus interface (Thomas)
-
Support tags in all datatypes (useful for images)
-
Incredible improvements in test coverage. 75% lines of core code tested (including close to 100% of the core sync engine)
-
RSS Feed source now supports video enclosures
-
Many many bug fixes as a result of test focus
As more people start to use conduit I am becoming more conservative and focusing a lot more on testing. One aspect of this is a machine I have set up that runs the test suite every time there is a commit. The other aspect is more regular releases. For the former look here, for the latter please test and give me feedback!
Team conduit is well poised for a 0.4.0 release to coincide with GNOME 2.22.
Holidays
It feels strange to be back at university after a solid 3 weeks of doing very little. I had the best summer holiday I have had in a long time. Considerable thanks must go to the weather in New Zealand, its great to finally have the good weather in December. Anyway here are some holiday snaps for those that are curious about what a summer Christmas/New Year in New Zealand looks like.
My Christmas Present
Relaxing on the Jetty
Sunset at the Bay Of Many Coves
Hunter Gatherer
Dec 16, 2007
Part 1: The Complete Gtk+ Stack from Source
Its been a long week spent watching things compile [0], but I'm happy to report that Jhbuild is now able to build the complete [1] stable Gtk+ stack on windows using the msys/mingw tool chain. That literally means that you can now build you Gtk app, or Gtk itself, for windows, in 12 steps. You still getting all the normal benefits that Jhbuild allows. So, the steps are;
-
Install MinGW combined installer (MinGW-5.1.3) to C:\mingw
-
Untar msysCORE-1.0.11-2007.01.11 to C:\msys
-
Run C:\msys\postinstall\pi.bat, answer yes when it asks whether mingw is already installed and point it to C:\ming
-
Install Python 2.5.1 to default location (C:\Python25)
-
Install ActivePerl to C:\Perl. You can probably also install a MingW-provided perl package, but the latest version of this was some 5.6 at the time of this writing, while ActivePerl is 5.8.
-
Install bzr for windows to C:\Bazaar
OK thats all the preparation steps. Launch C:\msys\msys.bat now to start a minimal unix-like shell. Now onto the meaty part;
-
bzr branch http://www.gnome.org/~jstowers/bzr/jhbuild-win32
-
cp jhbuild-win32/win32-hacks/* . && mv profile .profile && mv perl.sh /bin/perl
-
edit pkg-config.sh to reflect the prefix you are going to install (or have installed) pkg-config to
-
edit jhbuild-win32/configs/gtk-win32.jhbuild so that full unix paths reflect the correct location of pkg-config.sh. Make sure /bin/perl and pkg-config.sh are executable
-
Close the shell and restart it so that it picks up your new .profile. Check that you can run all needed executables
perl --version && python --version && jhbuild --help && ./pkg-config.sh --version
Oh, I seem to be forgetting one more thing.... The following command will build gtk+,atk,cairo,pango,zlib,libpng, and glib from source. No really, that is it [0].
- jhbuild -f jhbuild-win32/configs/gtk-win32.jhbuild build
The Hacks (aka things I don't really understand)
There are a few parts of this setup that still confuse me.
-
/bin/perl is a thin wrapper around perl that quotes args and escapes path slashes. Its necessity is based around the interaction of the msys path escaping and what happens when autotools invokes glib-mkenums. It was based upon what I read here. It also writes a debug log to ~/perllog
-
~/pkg-config.sh is a wrapper around pkg-config.sh that quotes args. It is as described here but with a full prefix to pkg-config (so that you may use one that is installed in your jhbuild prefix). It also writes a debug log to ~/pkglog
Jhbuild Core Modifications
I am currently hosting a bzr branch with my modifications to Jhbuild. I'm still cleaning some things up, but I hope that this is suitable for committing to SVN in the future. My modifications to Jhbuild are basically;
-
Cleaning up all path handling code. That means using os.path.abspath and os.path.join internally. The msys does shell does some shiny things to convert unix paths to windows paths but it works best if the path it is passed is coherent and not a mix of both systems path separators.
-
Some path and environment variable fudging that checks all paths that get passed out to a shell, and converts them to unix format. Python on windows calls the windows CreateProcess function (that expects a windows executable). Therefor we starts a new msys/bin/sh.exe with the first arg the script we originally wished to start, followed by all the inferior args.
-
Added a binary module type. This is basically like a tarball but instead of executing configure, make, make install you can define arbitary shell commands to execute post download and unpack. This is used to provide glib-bin and gtk-bin metapackages (which get binaries from Tor's gtk win32 site). You can depend on the metapackages if you dont feel like compiling the gtk+/glib stack yourself and just want to build your app. You can also insert these binary modules in the dependency chain and use them to execute arbitrary scripts to fix (read: hack) some aspects of packages.
-
New Gtk+ and Glib modules files for binary installation, and source installation.
-
Use pythons build in functions to download files (instead of wget) and to untar/unzip archives (instead of tar and unzip command line programs). Note that this is not strictly necessary as there are builds of tar/unzip and wget for msy - but its easier from a developer perspective.
Remaining Work, Caveats (aka the full story)
-
libjpeg builds ok but doesn't seem able to generate a shared library. Hence libtiff fails and gtk builds without jpeg support.
-
glib-mkenums seems to insert arbiraty data (a directory listing) at the end of .h files. This results in many compiler warning about data after #endif.
-
Read the gtk-win32.jhbuildrc file.
-
Gtk doesn't set the correct paths to the pixbuf loaders and input modules. You will need to run (from the prefix to which Jhbuild installed everything)
-
You can change gtk to use the default theme by doing
- echo 'gtk-theme-name = "MS-Windows"' > $JHBUILDPREFIX/etc/gtk-2.0/gtkrc
-
I have not tested this with SVN modules on windows, only tarball releases. I wasn't prepared to discover the compatibility of the msys provided autotools stack (yet).
-
--disable-docs on which/all modules?
-
Hookup to, and submit build reports to JhAutobuild
-
Python or msys or something buffers all the shell output. This means that there can be long periods (minutes) of nothing getting printed to the shell. If it looks like something has failed to build (no CPU/disk activity) the push 1,3 or 4 to retry, cancel, or drop into a shell respectively (i.e. the normal Jhbuild menu presented upon failure).
Part 2: I Just Want to Build my Application (not Gtk)
It should be noted that it is not necessary to build Gtk (from source) if you just want to build a single application. Using the new binary module type, I have prepared a gtk-binary metamodule that downloads all the needed files from Tor's site, and installs them for you. That sounds confusing, so lets take a randomly chosen example program [2]. Lets install medit, from source, using Jhbuild.
Starting with the example configs/gtk-win32.jhbuild file change the module and modulesets lines to
moduleset = [
os.path.join(thisdir,'gtk-win32-bin.modules'),
os.path.join(thisdir,'meddit.modules')
]
modules = ['meddit']
Add a module_autogen line for meddit
#Module specific configure arguments passed in addition to autogenargs
module_autogenargs['meddit'] = ' --without-xml --disable-help --with-python=no'
And the final step....
jhbuild -f jhbuild-win32/configs/gtk-win32.jhbuild build
Conclusion
I got into this because I wanted to be able to build pygoocanvas on windows, so I could hack on Conduit there. I still have not done that, instead I have;
-
gotten royally distracted from the original goal,
-
hacked on Jhbuild,
-
learned a lot about msys/mingw and autotools on windows,
-
hopefully been able to contribute something useful back to the community that makes it easy for joe normal and jane developer to test their app on windows
-
Oh and its not yet finished (but I have tested it a few times on fresh installs, and it should just work) yet
-
Pretty normal summary for a FOSS hackers week, correct?
I would really like help from some people in testing this further, so go ahead, test it today. Ive made it 12 easy steps for you to follow.......
References (thanks)
[0] A full Gtk+ build from source takes about 6 hours on Windows XP (running in VMware getting both 2.4Ghz Pentium 4 cores)
[1] Ok, not the complete stack. To get around some circular dependency issues I start with binary versions of pkg-config iconv and gettext. With pkg-config present I then build Glib from source. I also use a binary version of poppler because it depends on Gdk while Gtk depends on poppler.
[2] Well not entirely random. It was already advertised as being able to builad on windows. Also it failed during make install, complaining about missing share/mime/packages/freedesktop.org.xml and bin/update-mime-database. I created both as empty files and it was successfully installed.
Dec 13, 2007
The Good News
Following on from my last post I have continued work on getting Jhbuild to run on windows (within msys). The basic problem which I identified in my last post was the struggle between Python (for Windows) expectations about paths, and the hoops that msys and cygwin jump through to allow unix style paths to be used. Basically os.path.join and friends do the correct thing providing the subsequent path is used within python alone. Things get all difficult when that path is then passed out to a configure script, or as an argument to a shell command. With that in mind however, a picture speaks a thousand words....
-
It works...
-
On windows...
-
Through magic....
The Status
Jhbuild is successfully able to build a test moduleset, and even some real modules such as zlib and iconv on windows using the mingw compiler. Things are looking promising and I will continue to play with this tomorrow. Its a bit of a pain to debug because of the inconsistancies between how subprocess.Popen behaves on windows and on linux. So far my changes can be summarised as follows
-
Add Makefile.win32 and patch install-check to just call /usr/bin/install
-
Replace wget/curl with pythons built in urllib.urlretrieve
-
Replace tar/unzip with pythons built in tarfile and zipfile modules
-
Misc fixes to get Jhbuild to start including disabling the tray icon etc
-
Refactor buildscript.execute (the subprocess.Popen) wrapper to clean it up and to special case local script commands from system commands (because of differences in how win32 treats executing executables in the current dir). Best summarised as the following
def execute_script(self, command, args_str='', cwd=None, extra_env=None):
# THIS IS THE MAIN WIN32 HACK
if sys.platform.startswith('win') and command[0].startswith('./'):
command = ['sh', command[0].replace('./','')] + command[1:]
Jhbuild still works on Linux just as it used to (more or less, I need to port over some more things to the new buildscript.execute() functions). Wget/tar/curl/unzip etc are still used on Linux. In general the changes so far have been self contained and should not break regular operation
The Install Procedure
How can you try this pre-alpha code out?. Follow these steps on windows and check out from my bzr branch (or use the patch against SVN)
-
Install MinGW combined installer (MinGW-5.1.3) to C:\mingw
-
Install msysCORE-1.0.11-2007.01.11 to C:\msys
-
Run C:\msys\postinstall\pi.bat, answer yes when it asks whether mingw is already installed and point it to C:/mingw. Launching C:\msys\msys.bat now provides us with a minimal unix-like shell.
-
Install Python 2.5.1 to default location (C:\Python25)
-
Install bzr for windows to the (C:\Bazaar)
-
Start msys.bat (in C:\msys)
-
Install Jhbuild
-
bzr co http://www.gnome.org/~jstowers/bzr/jhbuild-win32 jhbuild-win32
-
cd jhbuild-win32
-
make -f Makefile.win32 install
-
Add python to path. Other path manipulation is done within JHBuild. Add the following line to ~/.profile
export PATH=$PATH:/c/Python25:/c/Python25/bin
You should now have a minimal msys install with JHBUILD. Test by performing
-
cd ~/bin
-
python jhbuild --help
Finally, you can now install the test modules with
python jhbuild -f jhbuild-win32/test.jhbuildrc build test
Standard disclaimers apply, alpha quality code, help appreciated, yada yada yada. Now back to Conduit hacking....
Dec 11, 2007
Background:
I'm going home to my parents house for Christmas, and because I don't have a laptop, I thought it would be the perfect opportunity to use their computer to get Conduit working on windows. However I could not find windows builds of (py)goocanvas anywhere. Never minding a challenge I thought it would be a good experiment to see if I could build pygoocanvas for windows via cross-compilation or natively - both using jhbuild.
I like to think that F/OSS works because people keep re-inventing the wheel in new and obscure ways everytime they do something, so what follows is my rambling tale of cross-compiling-gtk-for-windows wheel reinvention.
Cross compiling Gtk+ for Windows using Jhbuild
Starting with the work of Alberto Ruiz and Piotr Gaczkowski I thought I would setup a jhbuild environment to cross compile the Gtk+ stack. After a bit of fiddling I managed to get everything to build fine using mingw32 and jhbuild. The following files work successfully on Ubuntu gutsy and can build the latest stable releases of Gtk+/cairo/glib/libglade.
At this point I moved onto trying to build pygobject. This is where things got difficult. I tried a few different ways to get the cross compilation environment to see python. With some inspiration from this page on compiling glom for windows, I copied across the entire Python25 directory from a windows box and added it to the appropriate LDFLAGS and CFLAGS in jhbuild. This allowed pygobject to pass configure. However it now failed to build;
In file included from /usr/include/python2.5/Python.h:57,
from pygobject.h:5,
from pygobject-private.h:9,
from gobjectmodule.c:27:
/usr/include/python2.5/pyport.h:246:24: sys/select.h: No such file or directory
I went back to the windows machine and checked to see if sys/select.h was present in mingw. No luck. OK at this point I was stuck. Compiling wasn't looking likely, and I still wasnt even confident that if it did compile it would manange to actually link and run (due to different compiler versions and platforms, etc). I googled for a bit and saw that its now possible to build python extensions using mingw32 and that people had managed to cross compile python using mingw. Hmm, I pondered..... 'lets cross compile Python25 using mingw and then build pygobject using that.' Furthur googling revealed this python bug+patch for cross compiling using mingw. This is going to be easy.......
I added a new section to my modules file that would apply the patch to the Python 2.5.1 before building it. YOU FAIL AGAIN JOHN.
checking for %zd printf() format support... configure: error: cannot run test program while cross compiling
Hmm. I tried several strategies to get around this (including supplying a config.cache file with ac_cv_printf_zd_format=yes) but with no luck. You can see my plea for help on the bug report.......
At this point an entire day of my life had been irrevocably taken from me. I retired to bed. I would not let windows beat me....
Native compilation of Gtk+ for Windows using Jhbuild
OK. At this point I thought I might as well start again, but compile natively on windows using mingw and msys. In hindsight it probbably would have been easier to follow most of the instructions for glom verbaim, but I decided to continue my quest at wheel re-invention.
I read that Alberto was thinking about ways to make Jhbuild work on windows - a good goal. Starting with his patch on that bug report I modified it slightly so that tar, gunzip and friends wouldn't need to be installed. Furthur to that I set about hacking up some patches to allow jhbuild to at least run on windows. A few hours later and I was sure that this was going to take a few more hours :-). I now had a working (ish) minimal easy to set up windows build environment according to these instructions..... or so I thought.
Jhbuild managed to successfully parse my minimal jhbuildrc and module files, download and untar iconv but then it failed. Basically there is considerable work and thought needed to see how this should operate in msys/cygwin. The problem is already explained in part here, but I will summarise.
Python's os.path and friends do a good job at converting paths to the native system format. Unfortunately the msys and cygwin shells also convert paths. For instance msys automagically converts /foo/bar to C:\msys\foo\bar. Basically it means that python converts paths to windows format, but msys likes them in unix format (because its a unix shell environment). This results in files not getting found, and gross composite paths like --prefix=C:\foo\bar/baz/bob getting passed to configure.
So, I had a bit of a look at how this could be solved in Jhbuild. I guess there are a few strategies
-
Create a whole bunch of utility functions join_path(), prepend_path(), etc. And call these instead of the system ones and only escape to windows format if you are running in windows, and not running in windows just kidding you are really running in cygwin haha.
-
Remove all os.path and friends calls. Be really naive and hard code all path operations to use unix semantics. Then explictly note that Jhbuild on windows must be run from within a unix shell like msys/bash or cygwin/bash.
-
Magic. Come up with some sort of paranormal event that would win GNOME $1 million dollars which could then pay a developer to fix all this for me.
Conclusion
Well things didn't work out as well as I hoped. I will probbably go and compile everything without jhbuild and be happy. I did learn that;
-
Jhbuild on windows is a worthwhile cause. It would really help in automatically (jhautobuild) testing things like glib and GIO (at least giving devs a compile yes/no signal)
-
It could make generating the latest tarballs really easy, perhaps even sharing the same modulesets as those maintained by the release team
-
For extra bling it could even automatically create installers by tracking installed files using a strategy similar to checkinstall.
Help, thoughts or advice appreciated. For all the luck I have had the last few days I would not be surprised if I have missed something fundamentally obvious to this whole process. Heck if some fairy was to drop a pygoocanvas installer for windows in my mail box then I would be super happy. Oh and did I mention that im really looking forward to gio!
Dec 6, 2007
The short news is that Conduit now lives in GNOME SVN. The move took a little longer than I would have hoped, but thats largely my fault. I would like to express my thanks to Olav Vitters for importing the SVN repository, his excellent response time to my emails, and the awesome mango system he created.
Whats Good
Conduit trunk is now pretty much feature frozen. It feels really good to finally cross off the TODO list and fix some of the bugs that I created (due to some poor choices when I started the project). While I will blog again soon about the shiny new features for the moment I will talk about one useful piece of code that I am proud of, and that may be useful to others.
Initially I used an in-memory python dictionary based database to store the object mappings which represent a sync relationship between two pieces of data (the mapping DB). I pickled this to disk when conduit closed. Unfortunately this was not only horribly gross (memory usage was huge), it was embarrassingly short sighted, as the one place I care most about types is this code path. We use dictionaries and object hashes extensively in the sync logic and its really hard to debug sync problems when you (for example) don't notice that a dataprovider returns the wrong type until you unpickle it a few days later.
To remedy this I needed (wanted) to use a database that had few or no external dependencies, that was a bit tougher on type safety, and that was usable from multiple threads. After a while I decided that because sqlite was now part of python2.5 this would be a good choice, and that I could get around the threading issues some other way.
Not wanting to re-invent the wheel completely I started with a few existing pieces of code
I glued these together and created a really nifty (yet another) DB abstraction layer that can be used in either single or multiple threaded applications, and then displayed in a Gtk.TreeView with NO additional effort. Whats more, the DB also has a LRU cache that dramatically reduces the number of calls into the DB when displayed in a Gtk.TreeView. For more information check out the following
All of this has been really heavily tested and everything seems to work as expected*........ except for the LRU cache, which I broke somewhere - hmm. I should really fix that.
Nov 10, 2007
Thanks to the awesome work of Thomas Van Machelen, Conduit has the beginnings of a port to the hildon ui. Whats more impressive, is that unlike me (who owns a n800), Thomas has been doing all this in an emulator. So anyone who reads this, consider this an endorsement of Thomas request for a developer discount code!
Conduit on the n8xx is something I have been looking forward to for a while. The first and most tangible benefits will be giving n8xx users the ability to use Conduits photo sync facilities to upload pictures to all the webservices we support. Beyond this however, there is much more to offer;
-
PIM sync using the evolution-python bindings on both the host PC and the n8xx
-
File sync to the n8xx without having to plug it into USB and transfer things via the memory card. By using conduits built in network sync (over python XMLRPC), sync is as easy as letting the PC and n8xx be on the same network.
Oct 31, 2007
Conduit recently reached 1000 commits, and with it came together some functionality that has been brewing for a while.
Another one of those conduit screenshots that always look the same
-
Looser dependence on gtk. This means that conduit can be run from a console (-c command line switch) without gtk/goocanvas installed. This reduces the memory usage by about 10Mb!
-
Massive refactoring effort to aid more code sharing between dataproviders and the complete decoupling of the GUI from the core application. This allows other guis to written very independently of the rest of the application [1]. Stay tuned for a native hildon gui for nokian800 and/or ubuntu mobile.
-
Rhythmbox support. A simple dataprovider has been added that is able to sync rhythmbox static playlists with things. Initially this is used to test sync music with n800 (with the ability to transcode audio formats), but in my ideal world, sync with rhythmbox would involve
- Improving rhythmboxes DBus interface to support adding/removing songs to the library and querying playlists (including smart playlists).
-
Remember because we are gnomevfs aware some cool things are possible. For instance, one can sync + transcode videos/music to/from their phone which is mounted using gnomevfs-obex.
-
Work is currently proceeding furiously on
-
Integrating with opensync
-
Removing the twisted dependence, and using pythons built in xml-rpc server for network synchronization
-
Replacing our current in memory database with one based on sqlite
-
Support for iPod photos
-
Test coverage andbug fixing.
Anyway, svn HEAD is pretty rocky at the moment, but please experiment. Once the database changeover has taken place things should stabilise for a 0.3.5 release. This will be a release candidate and then we will focus on bug fixing only.
Ideally we will release a stable 0.4.0 release at or before gnome.conf.au. As an aside, I would be interested in talking to any of the RH online desktop folks at linux.conf.au as I am unable to make it to any of the northern hemisphere conferences on account of me being poor and in New Zealand.
[1] Excluding a dependence on gnomevfs (yuck). Hopefully gio/gvfs will make all my dreams come true.
Oct 10, 2007
Stumbled upon this. Discuss..
Oct 6, 2007
I just returned to my computer to find that nautilus crashed somewhere in the midst of copying my 50GB (1000 odd holiday videos) from my external hard disk to my NAS via FTP. Not wanting to work out where exactly the crash happened and what files were incomplete or still needing to be copied I thought 'hey, I could use Conduit to do a one way sync'. So I did, and it worked.
Its always cool when software you have written manages to scratch an itch that you didn't design it for!
