Johns Blog

Jhbuild Anything on Windows - In 12 Steps

Part 1: The Complete Gtk+ Stack from Source

Its been a long week spent watching things compile [0], but I'm happy to report that Jhbuild is now able to build the complete [1] stable Gtk+ stack on windows using the msys/mingw tool chain. That literally means that you can now build you Gtk app, or Gtk itself, for windows, in 12 steps. You still getting all the normal benefits that Jhbuild allows. So, the steps are;

1. Install MinGW combined installer (MinGW-5.1.3) to C:\mingw

2. Untar msysCORE-1.0.11-2007.01.11 to C:\msys

3. Run C:\msys\postinstall\pi.bat, answer yes when it asks whether mingw is already installed and point it to C:\ming

4. Install Python 2.5.1 to default location (C:\Python25)

5. Install ActivePerl to C:\Perl. You can probably also install a MingW-provided perl package, but the latest version of this was some 5.6 at the time of this writing, while ActivePerl is 5.8.

6. Install bzr for windows to C:\Bazaar

OK thats all the preparation steps. Launch C:\msys\msys.bat now to start a minimal unix-like shell. Now onto the meaty part;

1. bzr branch http://www.gnome.org/~jstowers/bzr/jhbuild-win32

2. cp jhbuild-win32/win32-hacks/* . && mv profile .profile && mv perl.sh /bin/perl

3. edit pkg-config.sh to reflect the prefix you are going to install (or have installed) pkg-config to

4. edit jhbuild-win32/configs/gtk-win32.jhbuild so that full unix paths reflect the correct location of pkg-config.sh. Make sure /bin/perl and pkg-config.sh are executable

5. Close the shell and restart it so that it picks up your new .profile. Check that you can run all needed executables perl --version && python --version && jhbuild --help && ./pkg-config.sh --version

Oh, I seem to be forgetting one more thing.... The following command will build gtk+,atk,cairo,pango,zlib,libpng, and glib from source. No really, that is it [0].

1. jhbuild -f jhbuild-win32/configs/gtk-win32.jhbuild build

The Hacks (aka things I don't really understand)

There are a few parts of this setup that still confuse me.

• /bin/perl is a thin wrapper around perl that quotes args and escapes path slashes. Its necessity is based around the interaction of the msys path escaping and what happens when autotools invokes glib-mkenums. It was based upon what I read here. It also writes a debug log to ~/perllog

• ~/pkg-config.sh is a wrapper around pkg-config.sh that quotes args. It is as described here but with a full prefix to pkg-config (so that you may use one that is installed in your jhbuild prefix). It also writes a debug log to ~/pkglog

Jhbuild Core Modifications

I am currently hosting a bzr branch with my modifications to Jhbuild. I'm still cleaning some things up, but I hope that this is suitable for committing to SVN in the future. My modifications to Jhbuild are basically;

• Cleaning up all path handling code. That means using os.path.abspath and os.path.join internally. The msys does shell does some shiny things to convert unix paths to windows paths but it works best if the path it is passed is coherent and not a mix of both systems path separators.

• Some path and environment variable fudging that checks all paths that get passed out to a shell, and converts them to unix format. Python on windows calls the windows CreateProcess function (that expects a windows executable). Therefor we starts a new msys/bin/sh.exe with the first arg the script we originally wished to start, followed by all the inferior args.

• Added a binary module type. This is basically like a tarball but instead of executing configure, make, make install you can define arbitary shell commands to execute post download and unpack. This is used to provide glib-bin and gtk-bin metapackages (which get binaries from Tor's gtk win32 site). You can depend on the metapackages if you dont feel like compiling the gtk+/glib stack yourself and just want to build your app. You can also insert these binary modules in the dependency chain and use them to execute arbitrary scripts to fix (read: hack) some aspects of packages.

• New Gtk+ and Glib modules files for binary installation, and source installation.

• Use pythons build in functions to download files (instead of wget) and to untar/unzip archives (instead of tar and unzip command line programs). Note that this is not strictly necessary as there are builds of tar/unzip and wget for msy - but its easier from a developer perspective.

Remaining Work, Caveats (aka the full story)

• libjpeg builds ok but doesn't seem able to generate a shared library. Hence libtiff fails and gtk builds without jpeg support.

• glib-mkenums seems to insert arbiraty data (a directory listing) at the end of .h files. This results in many compiler warning about data after #endif.

• Read the gtk-win32.jhbuildrc file.

• Gtk doesn't set the correct paths to the pixbuf loaders and input modules. You will need to run (from the prefix to which Jhbuild installed everything)

  • $JHBUILDPREFIX/bin/gtk-query-immodules-2.0.exe > $JHBUILDPREFIX/etc/gtk-2.0/gtk.immodules

  • $JHBUILDPREFIX/bin/gdk-pixbuf-query-loaders.exe > $JHBUILDPREFIX/etc/gtk-2.0/gdk-pixbuf.loaders

• You can change gtk to use the default theme by doing

  • echo 'gtk-theme-name = "MS-Windows"' > $JHBUILDPREFIX/etc/gtk-2.0/gtkrc

• I have not tested this with SVN modules on windows, only tarball releases. I wasn't prepared to discover the compatibility of the msys provided autotools stack (yet).

• --disable-docs on which/all modules?

• Hookup to, and submit build reports to JhAutobuild

• Python or msys or something buffers all the shell output. This means that there can be long periods (minutes) of nothing getting printed to the shell. If it looks like something has failed to build (no CPU/disk activity) the push 1,3 or 4 to retry, cancel, or drop into a shell respectively (i.e. the normal Jhbuild menu presented upon failure).

Part 2: I Just Want to Build my Application (not Gtk)

It should be noted that it is not necessary to build Gtk (from source) if you just want to build a single application. Using the new binary module type, I have prepared a gtk-binary metamodule that downloads all the needed files from Tor's site, and installs them for you. That sounds confusing, so lets take a randomly chosen example program [2]. Lets install medit, from source, using Jhbuild.

Starting with the example configs/gtk-win32.jhbuild file change the module and modulesets lines to

moduleset = [
    os.path.join(thisdir,'gtk-win32-bin.modules'),
    os.path.join(thisdir,'meddit.modules')
]
modules = ['meddit']

Add a module_autogen line for meddit

#Module specific configure arguments passed in addition to autogenargs
module_autogenargs['meddit'] =  ' --without-xml --disable-help --with-python=no'

And the final step....

jhbuild -f jhbuild-win32/configs/gtk-win32.jhbuild build

Conclusion

I got into this because I wanted to be able to build pygoocanvas on windows, so I could hack on Conduit there. I still have not done that, instead I have;

• gotten royally distracted from the original goal,

• hacked on Jhbuild,

• learned a lot about msys/mingw and autotools on windows,

• hopefully been able to contribute something useful back to the community that makes it easy for joe normal and jane developer to test their app on windows

• Oh and its not yet finished (but I have tested it a few times on fresh installs, and it should just work) yet

• Pretty normal summary for a FOSS hackers week, correct?

I would really like help from some people in testing this further, so go ahead, test it today. Ive made it 12 easy steps for you to follow.......

References (thanks)

• Building Gtk+/Gimp on Win32 (May 2005, Tor Lillqvist)

• Building Evolution on Windows (Feb 2007)

• An attempt at Win32 build instructions (Jan 2005, Owen Taylor)

• Compiling Glom on Windows (Dec 2007, Armin Burgmeier)

• pkg-config.sh (Jun 2007, Cody Russel)

• Circular deps of pkg-config (Tor Lillqvist)

[0] A full Gtk+ build from source takes about 6 hours on Windows XP (running in VMware getting both 2.4Ghz Pentium 4 cores)

[1] Ok, not the complete stack. To get around some circular dependency issues I start with binary versions of pkg-config iconv and gettext. With pkg-config present I then build Glib from source. I also use a binary version of poppler because it depends on Gdk while Gtk depends on poppler.

[2] Well not entirely random. It was already advertised as being able to builad on windows. Also it failed during make install, complaining about missing share/mime/packages/freedesktop.org.xml and bin/update-mime-database. I created both as empty files and it was successfully installed.

Kororaa XGL Live CD

Well I have just tried the Kororaa live cd on my P4 2.4Ghz with Nvidia FX5200 and it works flawlessly. Xgl and Compiz are amazing!

I only hope that the recent news that Compiz will soon run on Xgl AND aiglx doesn't spell the end (or slow painful death) or metacity as the gnome window manager. If this business has shown me anything its that David (xgl and compiz author) is a man machine and can accomplish anything!

ROS and Gtk for Laboratory Control

At the lab in which I work (Andrew Straw, strawlab) we study the visual flight behaviour of Drosophila using virtual reality. The implementation of this will be explained in future posts and papers however for this post I am going to describe how I used Gtk¹ and ROS to build an interface to control and monitor running experiments (called the 'Operator Console').

A future post will address and release all the ROS+GObject² glue that lets these interfaces scale dynamically as nodes (dis)appear. This just shows the relevant Gtk parts and has some comments on what I would like from Gtk to make these sort of interfaces easier.

The screenshow shows the first tab of the 'Operator Console'³.

Implementation Notes

• I use the secondary icon support of Gtk.Entry to show the contents contain sensible data. Maybe validation support in Gtk would be useful here bug.
• The 'Description' entry is a Gtk.TextView, not a Gtk.Entry. It was necessary to apply custom CSS to make it look reasonably similar. Sadly, it does not support the full/same set of CSS properties as Gtk.Entry, so it was impossible to show the same border radius and focus colors bug. Perhaps a multi-line Gtk.Entry would be better.
• The bottom half of the window shows the utilisation of all computers. I tried a few versions of this, and simple sensibly formatted monospaced text looked much better than anything else I tried. Any suggestions?

This screenshot shows an example screen where we mix the control and monitoring or many instances of the same ROS node.

Implementation Notes

• The Gtk.Switch simultainously displays the status of the projector, and also allows control of the node. The is a common use-case in the software, and due to the asynchronous nature of the ROS messages, I need to distinguish these from user-generated signals. I have wrappers such as the following for many widgets⁴. Advice on how to distinguish this use-case would be preferred.

class UpdateableGtkSwitch(Gtk.Switch):
    def __init__(self, *args, **kwargs):
        Gtk.Switch.__init__(self, *args, **kwargs)
        self._changing = False
        self.connect_after("notify::active", self._changed)

    def _changed(self, *args):
        if self._changing:
            self.stop_emission("notify::active")

    def set_active(self, is_active):
        self._changing = True
        Gtk.Switch.set_active(self, is_active)
        self._changing = False

    def connect(self, *args, **kwargs):
        self.connect_after(*args, **kwargs)

• The "Standby (Computer1)" is for display only and mirrors the status of a ROS topic. I would like some way visually to inicate that this widget is not actually an editable Gtk.Entry. Currently the Gtk.Entry is set editable = False, it looks to out of place with sensitive = False. Perhaps I should add some custom CSS to color it slightly different. Suggestions are appreciated.

Closing Remarks

I'm really happy with the status of the PyGObject bindings. We have a few quite large applications built using them (and ROS) and I have no complaints about performance⁵ or otherwise. The conventional wisdom was that PyGTK (and GTK) were not suitable for threaded workloads but the threading model of ROS guarentees that the 'operator-console' shown above manages upwards of 50 background threads asynchronously updating the GUI state.

Life Updates

So I thought I should spice up my blog with a post about non nerdy stuff. Currently I am travelling around Europe and will continue doing that until about June/July at which time I fly to Canada.

I was initially planning on doing a University exchange at the University of British Columbia but have since decided to take up a PhD at the University of Canterbury. I will (hopefully) be working on UAV stuff, likely machine vision related, in the Geospatial Research Centre at the university, beginning on the 1st of August. Travelling by car across Europe by car is so far been exceptionally awesome. Have me some cool characters and have certainly become a better driver. It has allowed us the freedom to go where we want, and when split three ways, has been just as economic as train travel.

Lightweight Development Prefixes

Lately I have been writing a lot of native code against multiple OpenCV versions. Like many Linux developers I tend to keep different development prefixes isolated using LD_LIBRARY_PATH (and friends).

I recently took the time to clean up a little script I use for this purpose, and posted it online. Inspired by virtualenv, it now has a prompt!

To use the script:

1. create the development directory
2. copy the script there
3. source /path/to/env.sh

please leave comments or suggestions on the gist.

Linux.conf.au

Well I had a fantastically great time at linux.conf.au. I was fortunate enough to present two talks at the conference;

• Synchronization and GNOME (at the GNOME Miniconf)

  • Video

  • Slides

• Albatross - The Open UAV (presented at the main conference in conjunction with Hugo)

  • Video

  • Slides

But I think the coolest thing was meeting heaps of interesting (and impressive) people including Jono, Eugene, Manish, Andy, Davyd, Karla, Nigel, Rafael, etc, etc. I will put some more photos and some of my new BHAGS for Conduit and Albatross up soon.

Linux.conf.au Talk

Just a quick note to say that I'm giving a talk on Conduit at 9:20am Tuesday 29th of January at GNOME.conf.au. Come along if you are interested. I will be going through some demos showing the many ways you can use Conduit including;

• As a multiple site photo uploader

• To keep your Tomboy notes in sync

• As a file/folder sync tool

• Keep your GConf preferences in sync over multiple computers

• Keeping two computers in sync over the local network

• Sycing with removable devices like Nokia n800 and iPods

• As a PIM sync tool supporting a number of online services

I will also be demonstrating how to write your own dataprovider, and hopefully I will have time to discuss more difficult issues such as

• Furthur integration with GNOME (Roadmap to 2.24)

• SynCE/Opensync/SyncML

• Progress on the Maemo/Hildon port

Map Updates

I recently made a number of improvements to osm-gps-map, the easy to use mapping widget. The motivation for these came at the request of the foxtrotGPS developers (foxtrotGPS is a community developed fork of TangoGPS). These changes enhanced the API for adding images and tracks to the map, and in addition allowed me to clean up the basic API making it easier to use for the common case. But, there is more, especially relevant to Gtk+/GNOME 3.0.

• Introspection Support (JavaScript example)

• GTK+ 3.0 Compatibility

• API Documentation (finally...)

• Improved examples (C, Python, JavaScript)

Map Widget Release

I made a new April resolution to start blogging reguarly again. The first step of that long journey begins now.

OpenStreetMap GPS Mapping Widget - 0.3

I just released v0.3 of osm-gps-map, the easy to use Gtk+ mapping widget. Highlights for this release include;

• A new major contributor, Alberto Mardegan, who worked on many of the new features of this release. Thanks a lot Alberto!

• Draw map tracks with Cairo by default.

• Interpolate between zoom levels while waiting for a tile to download.

• Stop using GET_PRIVATE, and cache priv* for performance.

• Keep an extra border of images offscreen for smoother scrolling at the edges of the map.

• Keep the last N tiles in memory to improve render performance (previously they were loaded from disk)

• Add some new api; osm_gps_set_center, osm_gps_map_scroll.

Conduit Hacking

Hacking on Conduit continues, with much happyness for two reasons;

• GNOME has moved to git! Wahoo! I owe the sysadmin team many beers, except for Jc2k who I know could not handle them anyway.

• Conduit got a SOC project, congratulations Alexandre!

• Work is ongoing to merge in the new configuration and settings serialization code.

• More on this later, I promise...

Matlab and Simulink on Dapper

Note to self: If you need to unstall simulink on ubuntu dapper then make sure you install libxft1