Johns Blog

Google Online Desktop

Stumbled upon this. Discuss..

Gtk+ Map Widget

It has been a long time between blogs. I thought I should talk about the piece of software that has been responsible for the most emails in my inbox over the last few days - osm-gps-map, the Gtk+ based map widget. What started as a widget for use in one small application of mine has grown considerably.

• I recently released 0.4.0, a bugfix release.

• I created a mailing list, if you are a user or interested osm-gps-map, them please join.

But I thought I should take some time to highlight some of the most interesting users of osm-gps-map, particualry those users on the Maemo platform.

Maep, OSM2Go and GPXView (by Till Harbaum)

• Maep seems to be the flagship tile mapping application for the Nokia n900. Most notable is the wonderful onscreen display that Till has created on top of osm-gps-map.

• OSM2Go is an editor for openstreetmap.org map data

• GPXView is a viewer for Geocaching.com GPX files.

BrainStorm (by Adam Boggs)

• BrainStorm is a storm chasing application that plots your track, and overlays current radar and severe warning imagery on the map.

eCoach

• eCoach is an application for recording and managing sport activities with Nokia N900, it records heart rate data from various monitors, and plots your path on the map as you exercise.

Conclusion

• With the help of these users, the future of osm-gps-map looks very positive.

• I am currently working on merging Till's improvements to master to discourage people from copying osm-gps-map source into their application.

• If you are a user of osm-gps-map and I have forgotten you then I am sorry. Please contact me and join the osm-gps-map mailing list.

GTK+ Viewer for Microsoft Kinect

I'm hacking on the new Kinect at the moment and the OpenGL viewer didn't work for me so I threw together this terrible quality gtk+ one. I'll clean up the code and try to get this into OpenKinect ASAP.

GUADEC Report: Late As Usual

GUADEC was effing awesome. I have successfully repayed my sleep debt, and can reflect on all that I observed and learned for the week. In bullet point form;

_Insurance for the last night (WAKE ME UP, English and Turkish) _

• My Conduit talk went really well. I thought I was able to reach a good balance between 'this is cool for users', 'this is cool for developers', and 'this is a flagrantly useless technical demo because I can'. Following the talk, and over the rest of the week I had a number of chats with people regarding

  • Improvement hints for the UI. Thanks a lot Bastien and co.

  • What is considered a useful, and achievable, level of mobile device support.

  • Some technical clarifications on matters how to integrate into the desktop (webkit, gio, ui-via-dbus, etc)

  • Slides available here

• I felt the reception to Conduit was positive, and I look forward to the module inclusion discussions for 2.24.

• I was not productive on my summer of code project for that week, but I have come back with an overflowing amount of motivation to not only complete my work on binding syncml, but to also continue to add improvements that will help Conduit operate on Maemo and iPhone, and operate really well with Windows mobile devices, and Nokia devices (via gnome-phone-manager).

  • Dear Team (aka lazyweb): I have quite a large conduit branch I am working on. Whats the plan with bzr-playground.gnome.org. Can I easily push my existing branch there?

• It was great to meet some of the people I have been corresponding with over email, and have never actually seen. You know who you are!

• Electronica Festival Istanbul was a great adventure. Thanks to all that made it happen

• Thanks to the GUADEC volunteers.

• The 'tab meme' on planet.gnome.org was pretty funny. I was most fooled by Davyds. (am I the only one who kinda likes this metaphor?). I was also fooled in Federico's key note, when he proposed the NET_WM_I_CAN_HAS_TABS window hint for easy window -> tab transitioning (just kidding).

• Captain common sense awards are hereby given to:

  • Team Gtk+: For the reasoned future of Gtk+ discussion/keynote. While I was disappointed to learn aboutthe lack of canvas plans, it is not really surprising. I use Goocanvas, but I am hardly a power user, and without a chosen Canvas, I have zero motivation to switch. I would describe the Gtk3.0+ plan as necessary+pragmatic. That is a damn site better than what it has appeard to be over the last few years (stagnant+idealistic).

  • Realease Time: Nice reveal on the Gnome 2.30 -> Gnome 3.0 animation

• The BZR vs. git thing seemed to reach a whole new level of weirdness at GUADEC. First there was team BZR, whom I shall liken to a Turkish taxi driver. They seem to promise everything, they have the hard sell on, and you can never really be sure how much the cost will be. The git team seems to be like the seasoned traveller. They are showing the finger to the taxi drivers, and generally alternate between cursing at the incompetence of the taxi service, and ignoring them with glee. Instead of discussing the price with the taxi drivers, they instead have a prepared presentation on how flying cars are actually a million times better than taxis because they look so hot.

  • I actually dont really care what I use. Im not a power user. If I could somehow pay some sysadmin money to setup a unified GNOME themed viewvc that tracked both git and bzr I would. Thats all I want - to be able to track and see what folks are up to. I wrote more about this subject here.

  • I also support the notion of 'Pick the system that makes it easiest for the distributor to work with'. That likely means git. As a thought experiment, what might be the result of choosing git, or more correctly, rejecting bzr, in the wake of the Ubuntu is moving to KDE fear meme.

• I Didnt realize the enormous (positive) influence that Nokia has on GNOME. Positive influence is a polite way of saying 'how much money they spend on Gtk+ related subcontractors, and employees'. Apparently something cool is coming for Maemo $NEXT_VERSION

• If I have forgotten anyone, apologies. See you all next GUADEC!

• Does anyone look like their Hackergotchi? (and can someone make me one - if you have a photo of me, and its moderately acceptable...)

• Funniest moments would have to be

  • John Carr with a rats tail and silver necklace!

  • Me and Marks completely ineffective adventure from one side of Istanbul to the other, with no phone, no watch, no computers, no idea where anyone is, and death by hangover.

  • The humiliation/irony of singing really bad English Kareoke in a country whose official language is Turkish.

• I have some interesting Conduit work going to land soon

  • Officially supported Windows Mobile sync support

  • Nokia phone support via gnome-phone-manager

  • Massive refactoring to improve platform abstractions

  • Which includes a move to GIO

  • Which adds a pygtkwebkit based browser

  • Which removes the hard dependency on gnome-keyring

  • Which adds and native python file/vfs abstraction

  • These improve performance and make it a billion times easier to run Conduit on Windows and Mac.

Holiday Debrief

Well I have finally arrived in Canada, which signifies the end of my travels around Europe. Back in NZ in a week to start my PhD. Aside from the photos which I have been taking and irregularly uploading I have;

• Bought a Nokia n800 (watch for a conduit port soon)

• Played with an iPhone (its the little things, attention to detail of all transitions between operations)

• Watched Transformers (not bad)

• Read (1984, Fahrenheit 911, The God Delusion)

• Hacked on Conduit

• Missed GUADEC (I hope the videos and slides will go up soon)

• Updated to Gutsy (went smoothly)

Conduit Schmonduit

I released 0.3.2 last week or so. Aside from bug fixes we also added box.net support (sync evolution/files/tomboy notes/etc to/from/via box.net), and also added the ability to delete photos from the supported photo sites (Flickr, picasaweb, smugmug). This means that Conduit can now better keep your photos (in a folder, or in f-spot) in sync. This includes removing them when they are removed from the synchronized folder, or tagged/untagged in F-spot.

Most of the work is now happening in the better-mvc branch which will land soon. This is a general tidy up of the code base and a better split of the synchronization logic from the GUI. This will not only allow for improved good looks; but other improvements too.

Mock-up Credit: Karl Lattimer

I'm at GUADEC

An organised person would have blogged about going to GUADEC before the event started. I am not that person. I am there now. Come see me if you want to talk about / hack on

• GNOME tweak tool

• Scientific computing with PyGObject (improving interaction with numpy, etc)

• Windows builds of PyGObject + GTK3

• Real time charting (https://github.com/nzjrs/uber-graph)

Interfacing Python + C + OpenCV via ctypes

I was recently asked to help a colleague access his image processing C-library from python; quite a common task. As those of you who are familiar with Python might realise, there are a whole bag of ways that this can be accomplished;

• SWIG

• Cython

• ctypes

• PybindGen

In this case the colleague only needed to access a single function from the library returning image data, and then hand this result onto OpenCV. One happy side effect of the new (> v2.1) python-opencv bindings is that they do no validation on CvImage.SetData, which means you can pass an arbitrary string/pointer. Because of this I advised him I thought using something like SWIG was overkill, and he could just write a wrapper to his library using ctypes, or a thin python extension directly.

Image data contains embedded NULLs, and I could not find a concise example of dealing with non null-terminated, non-string char * arrays via ctypes so I wrote one.

# char *test_get_data_nulls(int *len);

func = lib.test_get_data_nulls
func.restype = POINTER(c_char)
func.argtypes = [POINTER(c_int)]

l = c_int()
data = func(byref(l))

print data,l,data.contents

and, another approach

# void test_get_data_nulls_out(char **data, int *len);

func_out = lib.test_get_data_nulls_out
func_out.argtypes = [POINTER(POINTER(c_char)), POINTER(c_int)]
func.restype = None

l2 = c_int()
data2 = POINTER(c_char)()
func_out(byref(data2), byref(l2))

print data2,l2,data2.contents

The full code can be found here and contains examples showing how to deal with data of this type using ctypes, and by writing a simple python extension linking with the library in question.

iPhone Synchronization on Linux

Now that Matt has gone ahead and announced this, I think I should send some more traffic his way. If you are interested in using Conduit to synchronize your iPhone contacts, calendar and notes then go and check out his work.

I hear people like this "iPhone" thingee

Also, I am still looking for volunteers to help me maintain the Google contacts/calendar Conduit dataproviders. They need some love, and I am just one developer.

Jhbuild Adventures on Windows

Background: I'm going home to my parents house for Christmas, and because I don't have a laptop, I thought it would be the perfect opportunity to use their computer to get Conduit working on windows. However I could not find windows builds of (py)goocanvas anywhere. Never minding a challenge I thought it would be a good experiment to see if I could build pygoocanvas for windows via cross-compilation or natively - both using jhbuild.

I like to think that F/OSS works because people keep re-inventing the wheel in new and obscure ways everytime they do something, so what follows is my rambling tale of cross-compiling-gtk-for-windows wheel reinvention.

Cross compiling Gtk+ for Windows using Jhbuild

Starting with the work of Alberto Ruiz and Piotr Gaczkowski I thought I would setup a jhbuild environment to cross compile the Gtk+ stack. After a bit of fiddling I managed to get everything to build fine using mingw32 and jhbuild. The following files work successfully on Ubuntu gutsy and can build the latest stable releases of Gtk+/cairo/glib/libglade.

• gtk+-win32-john.jhbuildrc

• gtk+-win32-john.moduleset

At this point I moved onto trying to build pygobject. This is where things got difficult. I tried a few different ways to get the cross compilation environment to see python. With some inspiration from this page on compiling glom for windows, I copied across the entire Python25 directory from a windows box and added it to the appropriate LDFLAGS and CFLAGS in jhbuild. This allowed pygobject to pass configure. However it now failed to build;

In file included from /usr/include/python2.5/Python.h:57,
    from pygobject.h:5,
    from pygobject-private.h:9,
    from gobjectmodule.c:27:
    /usr/include/python2.5/pyport.h:246:24: sys/select.h: No such file or directory

I went back to the windows machine and checked to see if sys/select.h was present in mingw. No luck. OK at this point I was stuck. Compiling wasn't looking likely, and I still wasnt even confident that if it did compile it would manange to actually link and run (due to different compiler versions and platforms, etc). I googled for a bit and saw that its now possible to build python extensions using mingw32 and that people had managed to cross compile python using mingw. Hmm, I pondered..... 'lets cross compile Python25 using mingw and then build pygobject using that.' Furthur googling revealed this python bug+patch for cross compiling using mingw. This is going to be easy.......

I added a new section to my modules file that would apply the patch to the Python 2.5.1 before building it. YOU FAIL AGAIN JOHN.

checking for %zd printf() format support... configure: error: cannot run test program while cross compiling

Hmm. I tried several strategies to get around this (including supplying a config.cache file with ac_cv_printf_zd_format=yes) but with no luck. You can see my plea for help on the bug report.......

At this point an entire day of my life had been irrevocably taken from me. I retired to bed. I would not let windows beat me....

Native compilation of Gtk+ for Windows using Jhbuild

OK. At this point I thought I might as well start again, but compile natively on windows using mingw and msys. In hindsight it probbably would have been easier to follow most of the instructions for glom verbaim, but I decided to continue my quest at wheel re-invention.

I read that Alberto was thinking about ways to make Jhbuild work on windows - a good goal. Starting with his patch on that bug report I modified it slightly so that tar, gunzip and friends wouldn't need to be installed. Furthur to that I set about hacking up some patches to allow jhbuild to at least run on windows. A few hours later and I was sure that this was going to take a few more hours :-). I now had a working (ish) minimal easy to set up windows build environment according to these instructions..... or so I thought.

Jhbuild managed to successfully parse my minimal jhbuildrc and module files, download and untar iconv but then it failed. Basically there is considerable work and thought needed to see how this should operate in msys/cygwin. The problem is already explained in part here, but I will summarise.

Python's os.path and friends do a good job at converting paths to the native system format. Unfortunately the msys and cygwin shells also convert paths. For instance msys automagically converts /foo/bar to C:\msys\foo\bar. Basically it means that python converts paths to windows format, but msys likes them in unix format (because its a unix shell environment). This results in files not getting found, and gross composite paths like --prefix=C:\foo\bar/baz/bob getting passed to configure.

So, I had a bit of a look at how this could be solved in Jhbuild. I guess there are a few strategies

1. Create a whole bunch of utility functions join_path(), prepend_path(), etc. And call these instead of the system ones and only escape to windows format if you are running in windows, and not running in windows just kidding you are really running in cygwin haha.

2. Remove all os.path and friends calls. Be really naive and hard code all path operations to use unix semantics. Then explictly note that Jhbuild on windows must be run from within a unix shell like msys/bash or cygwin/bash.

3. Magic. Come up with some sort of paranormal event that would win GNOME $1 million dollars which could then pay a developer to fix all this for me.

Conclusion

Well things didn't work out as well as I hoped. I will probbably go and compile everything without jhbuild and be happy. I did learn that;

• Jhbuild on windows is a worthwhile cause. It would really help in automatically (jhautobuild) testing things like glib and GIO (at least giving devs a compile yes/no signal)

• It could make generating the latest tarballs really easy, perhaps even sharing the same modulesets as those maintained by the release team

• For extra bling it could even automatically create installers by tracking installed files using a strategy similar to checkinstall.

Help, thoughts or advice appreciated. For all the luck I have had the last few days I would not be surprised if I have missed something fundamentally obvious to this whole process. Heck if some fairy was to drop a pygoocanvas installer for windows in my mail box then I would be super happy. Oh and did I mention that im really looking forward to gio!

Jhbuild and Windows - We Meet Again

The Good News

Following on from my last post I have continued work on getting Jhbuild to run on windows (within msys). The basic problem which I identified in my last post was the struggle between Python (for Windows) expectations about paths, and the hoops that msys and cygwin jump through to allow unix style paths to be used. Basically os.path.join and friends do the correct thing providing the subsequent path is used within python alone. Things get all difficult when that path is then passed out to a configure script, or as an argument to a shell command. With that in mind however, a picture speaks a thousand words....

1. It works...

2. On windows...

3. Through magic....

The Status

Jhbuild is successfully able to build a test moduleset, and even some real modules such as zlib and iconv on windows using the mingw compiler. Things are looking promising and I will continue to play with this tomorrow. Its a bit of a pain to debug because of the inconsistancies between how subprocess.Popen behaves on windows and on linux. So far my changes can be summarised as follows

1. Add Makefile.win32 and patch install-check to just call /usr/bin/install

2. Replace wget/curl with pythons built in urllib.urlretrieve

3. Replace tar/unzip with pythons built in tarfile and zipfile modules

4. Misc fixes to get Jhbuild to start including disabling the tray icon etc

5. Refactor buildscript.execute (the subprocess.Popen) wrapper to clean it up and to special case local script commands from system commands (because of differences in how win32 treats executing executables in the current dir). Best summarised as the following

   def execute_script(self, command, args_str='', cwd=None, extra_env=None): # THIS IS THE MAIN WIN32 HACK if sys.platform.startswith('win') and command[0].startswith('./'): command = ['sh', command[0].replace('./','')] + command[1:]

Jhbuild still works on Linux just as it used to (more or less, I need to port over some more things to the new buildscript.execute() functions). Wget/tar/curl/unzip etc are still used on Linux. In general the changes so far have been self contained and should not break regular operation

The Install Procedure

How can you try this pre-alpha code out?. Follow these steps on windows and check out from my bzr branch (or use the patch against SVN)

1. Install MinGW combined installer (MinGW-5.1.3) to C:\mingw

2. Install msysCORE-1.0.11-2007.01.11 to C:\msys

3. Run C:\msys\postinstall\pi.bat, answer yes when it asks whether mingw is already installed and point it to C:/mingw. Launching C:\msys\msys.bat now provides us with a minimal unix-like shell.

4. Install Python 2.5.1 to default location (C:\Python25)

5. Install bzr for windows to the (C:\Bazaar)

6. Start msys.bat (in C:\msys)

7. Install Jhbuild

   1. bzr co http://www.gnome.org/~jstowers/bzr/jhbuild-win32 jhbuild-win32

   2. cd jhbuild-win32

   3. make -f Makefile.win32 install

8. Add python to path. Other path manipulation is done within JHBuild. Add the following line to ~/.profile

   export PATH=$PATH:/c/Python25:/c/Python25/bin

You should now have a minimal msys install with JHBUILD. Test by performing

• cd ~/bin

• python jhbuild --help

Finally, you can now install the test modules with

python jhbuild -f jhbuild-win32/test.jhbuildrc build test

Standard disclaimers apply, alpha quality code, help appreciated, yada yada yada. Now back to Conduit hacking....