Dec 11, 2007
Background:
I'm going home to my parents house for Christmas, and because I don't have a laptop, I thought it would be the perfect opportunity to use their computer to get Conduit working on windows. However I could not find windows builds of (py)goocanvas anywhere. Never minding a challenge I thought it would be a good experiment to see if I could build pygoocanvas for windows via cross-compilation or natively - both using jhbuild.
I like to think that F/OSS works because people keep re-inventing the wheel in new and obscure ways everytime they do something, so what follows is my rambling tale of cross-compiling-gtk-for-windows wheel reinvention.
Cross compiling Gtk+ for Windows using Jhbuild
Starting with the work of Alberto Ruiz and Piotr Gaczkowski I thought I would setup a jhbuild environment to cross compile the Gtk+ stack. After a bit of fiddling I managed to get everything to build fine using mingw32 and jhbuild. The following files work successfully on Ubuntu gutsy and can build the latest stable releases of Gtk+/cairo/glib/libglade.
At this point I moved onto trying to build pygobject. This is where things got difficult. I tried a few different ways to get the cross compilation environment to see python. With some inspiration from this page on compiling glom for windows, I copied across the entire Python25 directory from a windows box and added it to the appropriate LDFLAGS and CFLAGS in jhbuild. This allowed pygobject to pass configure. However it now failed to build;
In file included from /usr/include/python2.5/Python.h:57,
from pygobject.h:5,
from pygobject-private.h:9,
from gobjectmodule.c:27:
/usr/include/python2.5/pyport.h:246:24: sys/select.h: No such file or directory
I went back to the windows machine and checked to see if sys/select.h was present in mingw. No luck. OK at this point I was stuck. Compiling wasn't looking likely, and I still wasnt even confident that if it did compile it would manange to actually link and run (due to different compiler versions and platforms, etc). I googled for a bit and saw that its now possible to build python extensions using mingw32 and that people had managed to cross compile python using mingw. Hmm, I pondered..... 'lets cross compile Python25 using mingw and then build pygobject using that.' Furthur googling revealed this python bug+patch for cross compiling using mingw. This is going to be easy.......
I added a new section to my modules file that would apply the patch to the Python 2.5.1 before building it. YOU FAIL AGAIN JOHN.
checking for %zd printf() format support... configure: error: cannot run test program while cross compiling
Hmm. I tried several strategies to get around this (including supplying a config.cache file with ac_cv_printf_zd_format=yes) but with no luck. You can see my plea for help on the bug report.......
At this point an entire day of my life had been irrevocably taken from me. I retired to bed. I would not let windows beat me....
Native compilation of Gtk+ for Windows using Jhbuild
OK. At this point I thought I might as well start again, but compile natively on windows using mingw and msys. In hindsight it probbably would have been easier to follow most of the instructions for glom verbaim, but I decided to continue my quest at wheel re-invention.
I read that Alberto was thinking about ways to make Jhbuild work on windows - a good goal. Starting with his patch on that bug report I modified it slightly so that tar, gunzip and friends wouldn't need to be installed. Furthur to that I set about hacking up some patches to allow jhbuild to at least run on windows. A few hours later and I was sure that this was going to take a few more hours :-). I now had a working (ish) minimal easy to set up windows build environment according to these instructions..... or so I thought.
Jhbuild managed to successfully parse my minimal jhbuildrc and module files, download and untar iconv but then it failed. Basically there is considerable work and thought needed to see how this should operate in msys/cygwin. The problem is already explained in part here, but I will summarise.
Python's os.path and friends do a good job at converting paths to the native system format. Unfortunately the msys and cygwin shells also convert paths. For instance msys automagically converts /foo/bar to C:\msys\foo\bar. Basically it means that python converts paths to windows format, but msys likes them in unix format (because its a unix shell environment). This results in files not getting found, and gross composite paths like --prefix=C:\foo\bar/baz/bob getting passed to configure.
So, I had a bit of a look at how this could be solved in Jhbuild. I guess there are a few strategies
-
Create a whole bunch of utility functions join_path(), prepend_path(), etc. And call these instead of the system ones and only escape to windows format if you are running in windows, and not running in windows just kidding you are really running in cygwin haha.
-
Remove all os.path and friends calls. Be really naive and hard code all path operations to use unix semantics. Then explictly note that Jhbuild on windows must be run from within a unix shell like msys/bash or cygwin/bash.
-
Magic. Come up with some sort of paranormal event that would win GNOME $1 million dollars which could then pay a developer to fix all this for me.
Conclusion
Well things didn't work out as well as I hoped. I will probbably go and compile everything without jhbuild and be happy. I did learn that;
-
Jhbuild on windows is a worthwhile cause. It would really help in automatically (jhautobuild) testing things like glib and GIO (at least giving devs a compile yes/no signal)
-
It could make generating the latest tarballs really easy, perhaps even sharing the same modulesets as those maintained by the release team
-
For extra bling it could even automatically create installers by tracking installed files using a strategy similar to checkinstall.
Help, thoughts or advice appreciated. For all the luck I have had the last few days I would not be surprised if I have missed something fundamentally obvious to this whole process. Heck if some fairy was to drop a pygoocanvas installer for windows in my mail box then I would be super happy. Oh and did I mention that im really looking forward to gio!
Dec 6, 2007
The short news is that Conduit now lives in GNOME SVN. The move took a little longer than I would have hoped, but thats largely my fault. I would like to express my thanks to Olav Vitters for importing the SVN repository, his excellent response time to my emails, and the awesome mango system he created.
Whats Good
Conduit trunk is now pretty much feature frozen. It feels really good to finally cross off the TODO list and fix some of the bugs that I created (due to some poor choices when I started the project). While I will blog again soon about the shiny new features for the moment I will talk about one useful piece of code that I am proud of, and that may be useful to others.
Initially I used an in-memory python dictionary based database to store the object mappings which represent a sync relationship between two pieces of data (the mapping DB). I pickled this to disk when conduit closed. Unfortunately this was not only horribly gross (memory usage was huge), it was embarrassingly short sighted, as the one place I care most about types is this code path. We use dictionaries and object hashes extensively in the sync logic and its really hard to debug sync problems when you (for example) don't notice that a dataprovider returns the wrong type until you unpickle it a few days later.
To remedy this I needed (wanted) to use a database that had few or no external dependencies, that was a bit tougher on type safety, and that was usable from multiple threads. After a while I decided that because sqlite was now part of python2.5 this would be a good choice, and that I could get around the threading issues some other way.
Not wanting to re-invent the wheel completely I started with a few existing pieces of code
I glued these together and created a really nifty (yet another) DB abstraction layer that can be used in either single or multiple threaded applications, and then displayed in a Gtk.TreeView with NO additional effort. Whats more, the DB also has a LRU cache that dramatically reduces the number of calls into the DB when displayed in a Gtk.TreeView. For more information check out the following
All of this has been really heavily tested and everything seems to work as expected*........ except for the LRU cache, which I broke somewhere - hmm. I should really fix that.
Nov 10, 2007
Thanks to the awesome work of Thomas Van Machelen, Conduit has the beginnings of a port to the hildon ui. Whats more impressive, is that unlike me (who owns a n800), Thomas has been doing all this in an emulator. So anyone who reads this, consider this an endorsement of Thomas request for a developer discount code!
Conduit on the n8xx is something I have been looking forward to for a while. The first and most tangible benefits will be giving n8xx users the ability to use Conduits photo sync facilities to upload pictures to all the webservices we support. Beyond this however, there is much more to offer;
-
PIM sync using the evolution-python bindings on both the host PC and the n8xx
-
File sync to the n8xx without having to plug it into USB and transfer things via the memory card. By using conduits built in network sync (over python XMLRPC), sync is as easy as letting the PC and n8xx be on the same network.
Oct 31, 2007
Conduit recently reached 1000 commits, and with it came together some functionality that has been brewing for a while.
Another one of those conduit screenshots that always look the same
-
Looser dependence on gtk. This means that conduit can be run from a console (-c command line switch) without gtk/goocanvas installed. This reduces the memory usage by about 10Mb!
-
Massive refactoring effort to aid more code sharing between dataproviders and the complete decoupling of the GUI from the core application. This allows other guis to written very independently of the rest of the application [1]. Stay tuned for a native hildon gui for nokian800 and/or ubuntu mobile.
-
Rhythmbox support. A simple dataprovider has been added that is able to sync rhythmbox static playlists with things. Initially this is used to test sync music with n800 (with the ability to transcode audio formats), but in my ideal world, sync with rhythmbox would involve
- Improving rhythmboxes DBus interface to support adding/removing songs to the library and querying playlists (including smart playlists).
-
Remember because we are gnomevfs aware some cool things are possible. For instance, one can sync + transcode videos/music to/from their phone which is mounted using gnomevfs-obex.
-
Work is currently proceeding furiously on
-
Integrating with opensync
-
Removing the twisted dependence, and using pythons built in xml-rpc server for network synchronization
-
Replacing our current in memory database with one based on sqlite
-
Support for iPod photos
-
Test coverage andbug fixing.
Anyway, svn HEAD is pretty rocky at the moment, but please experiment. Once the database changeover has taken place things should stabilise for a 0.3.5 release. This will be a release candidate and then we will focus on bug fixing only.
Ideally we will release a stable 0.4.0 release at or before gnome.conf.au. As an aside, I would be interested in talking to any of the RH online desktop folks at linux.conf.au as I am unable to make it to any of the northern hemisphere conferences on account of me being poor and in New Zealand.
[1] Excluding a dependence on gnomevfs (yuck). Hopefully gio/gvfs will make all my dreams come true.
Oct 6, 2007
I just returned to my computer to find that nautilus crashed somewhere in the midst of copying my 50GB (1000 odd holiday videos) from my external hard disk to my NAS via FTP. Not wanting to work out where exactly the crash happened and what files were incomplete or still needing to be copied I thought 'hey, I could use Conduit to do a one way sync'. So I did, and it worked.
Its always cool when software you have written manages to scratch an itch that you didn't design it for!
Oct 3, 2007
Wahoo!, after mad busy weeks of development, I just released Conduit 0.3.4. Check out the screencast (download here). This release is fresh with new features, and heaps of polish and integration work. These new features include;
-
Google calendar support (Paul Novotny).
-
Youtube dataprovider added (Renato Araujo).
-
Photo sites are now two way, you data is no longer locked with one provider. Also see #480754 for improvements to Fspots DBus interface allowing two way photo sync with FSpot.
-
Nautilus extension. (disabled by default).
-
EOG plugin added. (disabled by default).
-
Conduit now supports resizing and converting photos before uploading them to photo sites. This infrastructure means improved support for other types of conversions, such as transcoding music and video, will be added in the future.
-
Progress indicator shows percent complete during synchronization.
-
Add support for Facebook photo upload.
-
Add included web browser for authenticating against web sites.
Website Authentication
The last point deserves some explanation. Currently, to authenticate against some sites the user must log in either once (Flickr), or once per session (box.net). This is difficult when using the system default web browser, if the browser is currently running on a different virtual desktop to conduit, the user often doesnt notice the web browser go to the login website. This is compounded by the different APIs of some sites. Flickr is good, I can just call am_i_authenticated(token) in a loop, and wait for success, or time out. Facebook is not good. Any call involving the facebook token before it has been validated (i.e. user logs in) invalidates the pending login. Because there is no guarenteed way to block in all cases (i.e. until the user logs in and closes the window), things like gnome.open_url() or python.webbrowser() are unsuitable. This leaves me with two options. I can wait a random amount of time, and hope its long enough for the user to log in, or I can write my own web browser that will block until the user closes it. Suggestions for option 3 are appreciated....
I know this is not ideal. I hope something like web-login-browser can solve my needs. Also, I know my gtkmozembed based browser has some (crasher..) issues (maybe related to this). I hope to fix them and/or move to a pywebkitgtk based browser, depending on what GNOME favors.
Here is a screencast showing the latest version and its features. You may download the original here. Apologies for the google video quality.
[googlevideo]http://video.google.com/videoplay?docid=-262844619875208739&hl=en[/googlevideo]
Boston Summit
Unfortunately, owing to my proximity to the opposite side of the world as Boston, I will not be attending the summit. I understand that the GNOME online desktop is on the agenda, and will no doubt be discussed at some length. I hope those involved in the discussion try out conduit before the summit, as it would be a shame if the future of the GNOME online desktop was set without due consideration. This is not a criticism, sour grapes, or anything like that. Its just a reflection of what appeared to happen after GUADEC - which I also did not attend (New Zealand = isolated and distant).
Have fun with Conduit. Please report bugs, join the mailing list, and abuse me at will. We are currently moving to GNOME servers, so if my server goes down from all the traffic I promise it will be the last time.
Sep 3, 2007
No screenshots this time, but still good news. Conduit tries to be a good citizen, integrating itself with GNOME technologies. Recently the following hacking has been going on (not all items are complete)
-
A Eye Of Gnome plugin allowing photo upload/sync to Flickr, Picasa, SmugMug, Facebook (soon)
-
Gnome user documentation
-
Themable icons
-
Patches to improve FSpots DBus interface to allow two way syncs with photo sites
-
Continued work on OpenSync integration. We have limited support for arbitary opensync plugins!
-
Canola, n800, Rhythmbox and general music synchronization
There have also been some new contributions
-
Two way support for Flickr, Picasa and Smugmug. Now you can move your photos between sites (AKA move my data)
-
A youtube dataprovider (sync your fave videos each morning)
-
Google calendar support (sync evolution with google calendar, iPod, etc)
Anyway if you are looking to get involved with Conduit now is a great time. I have just updated the docs on how to write a dataprovider. Here are some ideas for dataproviders people may wish to write (hint hint)
There will be some new features landing in trunk soon that will allow arbitrary parameters during the conversion process, i.e. transcoding audio/video to another format, resizing photos, compressing files, etc. Watch this space, download conduit, and go tell people it rocks!
Aug 20, 2007
I have a love-hate relationship with autotools. I love the thought of a parallel universe where it doesn't exist, and hate it with the fiery passion of 1000 suns the rest of the time. I just released Conduit 0.3.3 and have had some reports of some build errors. The recurring build error is basically something like
Running intltoolize…
Running aclocal-1.4…
aclocal: configure.ac: 18: macro `AM_PATH_PYTHON’ not found in library
The problem is, by some combination of luck and ignorance I have never been able to reproduce this or fix it (on Ubuntu). The additional confusion is that Conduit is python, so I understand even less of the interaction of autotools. I therefor have an offer;
I will happily pay the person who fixes the horrible autotools setup in Conduit a crisp $NZ50. This cool cash can be redeemed from me in person at linux.conf.au, in cash or beer form, or I will buy the lucky person a piece of FOSS memorabilia (example) to that value.
That means fixing the above bug, deleting autofoo files that don't do anything, checking that the translation stuff is setup optimally and generally making it less shameful and easier to follow. The person will also receive additional love points if they implement a skeleton autotools setup for documentation and for themed application icons. Remember, I hate autotools this much...
Aug 16, 2007
After a short hacking hiatus, soul searching, and general uncertainty regarding Conduits place in the Gnome Online Desktop plan, we the Conduit developers have decided to keep doing what we do best; adding awesome features, and making Conduit into the best desktop synchronization service for the free desktop. I invite you all to spot the new features in the image below;
Going through the new features one by one;
-
The Network datasink - thats how you tell conduit to share data to other people on the network
-
The chad.local dataprovider - ... and thats how you'll choose to consume other peoples shared data
-
The evo2-sync dataprovider - before long you'll be able to use opensync plugins in conduit
-
The f-spot dataprovider - before long that will be two-way and networkable, work is ongoing with upstream
-
Always-Up-To-Date is on the horizon (i.e. auto update and sync changed data)
-
iPod photos support - our patches are accepted upstream, so soon we will support those too
-
The facebook dataprovider - just photos so far, but more facebook features will be exposed soon
Other than those Massive New Features, there has been continued bug fixing and some improvements to the DBus interface to make it more Havoc Compliant. Check out the sample below for how to use Conduits DBus interface to synchronize things...
bus = dbus.SessionBus()
#Create an Interface wrapper for the remote object
remote_object = bus.get_object(APPLICATION_DBUS_IFACE,"/")
app = dbus.Interface(remote_object, APPLICATION_DBUS_IFACE)
#get a dataprovider
path = app.GetDataProvider("TestSource")
source = dbus.Interface(
bus.get_object(DATAPROVIDER_DBUS_IFACE,path),
DATAPROVIDER_DBUS_IFACE
)
#get another
path = app.GetDataProvider("TestSink")
sink = dbus.Interface(
bus.get_object(DATAPROVIDER_DBUS_IFACE,path),
DATAPROVIDER_DBUS_IFACE
)
#get a conduit
path = app.BuildConduit(source, sink)
cond = dbus.Interface(
bus.get_object(CONDUIT_DBUS_IFACE,path),
CONDUIT_DBUS_IFACE
)
print "Synchronize the conduit"
cond.Sync()
I have written an example which shows how to use Conduit to upload photos to any of our supported photo sites with even fewer lines of code. The DBus API documentation lives at the top of this file. Dont forget that by using a desktop wide sync framework you get all the Conduit bling for free, including
Jul 23, 2007
Well I have finally arrived in Canada, which signifies the end of my travels around Europe. Back in NZ in a week to start my PhD. Aside from the photos which I have been taking and irregularly uploading I have;
-
Bought a Nokia n800 (watch for a conduit port soon)
-
Played with an iPhone (its the little things, attention to detail of all transitions between operations)
-
Watched Transformers (not bad)
-
Read (1984, Fahrenheit 911, The God Delusion)
-
Hacked on Conduit
-
Missed GUADEC (I hope the videos and slides will go up soon)
-
Updated to Gutsy (went smoothly)
Conduit Schmonduit
I released 0.3.2 last week or so. Aside from bug fixes we also added box.net support (sync evolution/files/tomboy notes/etc to/from/via box.net), and also added the ability to delete photos from the supported photo sites (Flickr, picasaweb, smugmug). This means that Conduit can now better keep your photos (in a folder, or in f-spot) in sync. This includes removing them when they are removed from the synchronized folder, or tagged/untagged in F-spot.
Most of the work is now happening in the better-mvc branch which will land soon. This is a general tidy up of the code base and a better split of the synchronization logic from the GUI. This will not only allow for improved good looks; but other improvements too.
Mock-up Credit: Karl Lattimer
