Jul 15, 2011
I was recently asked to help a colleague access his image processing C-library from python; quite a common task. As those of you who are familiar with Python might realise, there are a whole bag of ways that this can be accomplished;
In this case the colleague only needed to access a single function from the library returning image data, and then hand this result onto OpenCV. One happy side effect of the new (> v2.1) python-opencv bindings is that they do no validation on CvImage.SetData, which means you can pass an arbitrary string/pointer. Because of this I advised him I thought using something like SWIG was overkill, and he could just write a wrapper to his library using ctypes, or a thin python extension directly.
Image data contains embedded NULLs, and I could not find a concise example of dealing with non null-terminated, non-string char * arrays via ctypes so I wrote one.
# char *test_get_data_nulls(int *len);
func = lib.test_get_data_nulls
func.restype = POINTER(c_char)
func.argtypes = [POINTER(c_int)]
l = c_int()
data = func(byref(l))
print data,l,data.contents
and, another approach
# void test_get_data_nulls_out(char **data, int *len);
func_out = lib.test_get_data_nulls_out
func_out.argtypes = [POINTER(POINTER(c_char)), POINTER(c_int)]
func.restype = None
l2 = c_int()
data2 = POINTER(c_char)()
func_out(byref(data2), byref(l2))
print data2,l2,data2.contents
The full code can be found here and contains examples showing how to deal with data of this type using ctypes, and by writing a simple python extension linking with the library in question.
Dec 13, 2007
The Good News
Following on from my last post I have continued work on getting Jhbuild to run on windows (within msys). The basic problem which I identified in my last post was the struggle between Python (for Windows) expectations about paths, and the hoops that msys and cygwin jump through to allow unix style paths to be used. Basically os.path.join and friends do the correct thing providing the subsequent path is used within python alone. Things get all difficult when that path is then passed out to a configure script, or as an argument to a shell command. With that in mind however, a picture speaks a thousand words....
-
It works...
-
On windows...
-
Through magic....
The Status
Jhbuild is successfully able to build a test moduleset, and even some real modules such as zlib and iconv on windows using the mingw compiler. Things are looking promising and I will continue to play with this tomorrow. Its a bit of a pain to debug because of the inconsistancies between how subprocess.Popen behaves on windows and on linux. So far my changes can be summarised as follows
-
Add Makefile.win32 and patch install-check to just call /usr/bin/install
-
Replace wget/curl with pythons built in urllib.urlretrieve
-
Replace tar/unzip with pythons built in tarfile and zipfile modules
-
Misc fixes to get Jhbuild to start including disabling the tray icon etc
-
Refactor buildscript.execute (the subprocess.Popen) wrapper to clean it up and to special case local script commands from system commands (because of differences in how win32 treats executing executables in the current dir). Best summarised as the following
def execute_script(self, command, args_str='', cwd=None, extra_env=None):
# THIS IS THE MAIN WIN32 HACK
if sys.platform.startswith('win') and command[0].startswith('./'):
command = ['sh', command[0].replace('./','')] + command[1:]
Jhbuild still works on Linux just as it used to (more or less, I need to port over some more things to the new buildscript.execute() functions). Wget/tar/curl/unzip etc are still used on Linux. In general the changes so far have been self contained and should not break regular operation
The Install Procedure
How can you try this pre-alpha code out?. Follow these steps on windows and check out from my bzr branch (or use the patch against SVN)
-
Install MinGW combined installer (MinGW-5.1.3) to C:\mingw
-
Install msysCORE-1.0.11-2007.01.11 to C:\msys
-
Run C:\msys\postinstall\pi.bat, answer yes when it asks whether mingw is already installed and point it to C:/mingw. Launching C:\msys\msys.bat now provides us with a minimal unix-like shell.
-
Install Python 2.5.1 to default location (C:\Python25)
-
Install bzr for windows to the (C:\Bazaar)
-
Start msys.bat (in C:\msys)
-
Install Jhbuild
-
bzr co http://www.gnome.org/~jstowers/bzr/jhbuild-win32 jhbuild-win32
-
cd jhbuild-win32
-
make -f Makefile.win32 install
-
Add python to path. Other path manipulation is done within JHBuild. Add the following line to ~/.profile
export PATH=$PATH:/c/Python25:/c/Python25/bin
You should now have a minimal msys install with JHBUILD. Test by performing
-
cd ~/bin
-
python jhbuild --help
Finally, you can now install the test modules with
python jhbuild -f jhbuild-win32/test.jhbuildrc build test
Standard disclaimers apply, alpha quality code, help appreciated, yada yada yada. Now back to Conduit hacking....
Dec 6, 2007
The short news is that Conduit now lives in GNOME SVN. The move took a little longer than I would have hoped, but thats largely my fault. I would like to express my thanks to Olav Vitters for importing the SVN repository, his excellent response time to my emails, and the awesome mango system he created.
Whats Good
Conduit trunk is now pretty much feature frozen. It feels really good to finally cross off the TODO list and fix some of the bugs that I created (due to some poor choices when I started the project). While I will blog again soon about the shiny new features for the moment I will talk about one useful piece of code that I am proud of, and that may be useful to others.
Initially I used an in-memory python dictionary based database to store the object mappings which represent a sync relationship between two pieces of data (the mapping DB). I pickled this to disk when conduit closed. Unfortunately this was not only horribly gross (memory usage was huge), it was embarrassingly short sighted, as the one place I care most about types is this code path. We use dictionaries and object hashes extensively in the sync logic and its really hard to debug sync problems when you (for example) don't notice that a dataprovider returns the wrong type until you unpickle it a few days later.
To remedy this I needed (wanted) to use a database that had few or no external dependencies, that was a bit tougher on type safety, and that was usable from multiple threads. After a while I decided that because sqlite was now part of python2.5 this would be a good choice, and that I could get around the threading issues some other way.
Not wanting to re-invent the wheel completely I started with a few existing pieces of code
I glued these together and created a really nifty (yet another) DB abstraction layer that can be used in either single or multiple threaded applications, and then displayed in a Gtk.TreeView with NO additional effort. Whats more, the DB also has a LRU cache that dramatically reduces the number of calls into the DB when displayed in a Gtk.TreeView. For more information check out the following
All of this has been really heavily tested and everything seems to work as expected*........ except for the LRU cache, which I broke somewhere - hmm. I should really fix that.
May 30, 2007
In order to add evolution support (contacts, task items and memos) to Conduit I needed to be able to access the relevant parts of evolution-data-server from python. evolution-python is an incomplete wrapper around libebook and libecal. It has now got to the stage of being useful, so here is a somewhat nervous announcement of evolution-python v0.0.0
This should be considered a work in progress and will be changing a lot over the next few days as I implement methods needed for the two way synchronization of evolution data using Conduit. However the basic skeleton is there, and I invite people to help with the bindings. An easy way to access evolution-data-server from python has been missing for a while now so I hope evolution-python will be useful for others.
There are some known issues and items to work on listed on the project website. Here is a small screenshot of the included test application.
Mar 12, 2007
At the request of a few people I have made a demo to share, showing how I do threading in a pygtk application (Conduit). I find the following approach seems to work reliably and requires little code. Other approaches can be found here and here.
This approach takes advantage of the fact that signal emission in glib has been threadsafe since glib 2.8 (IIRC). All communication with the GUI is done via gobject signals. There is definitely a compromise in the level of GUI fiddling that you can do, particularly when compared with the threads_enter/leave approach. However, I have found this signal based approach sufficent in my case, and that it encourages me to decouple the slow blocking tasks from the GUI.
A lot of the tasks in conduit take a long time (network limited), and there is no real need for extensive GUI interaction with them once they have been started. I am really only interested in their progress, and when they complete. With this in mind I have implemented the following approach;
-
FooThreadManager
This class is the entry point for starting threads (the make_thread() method) . Its basically just a threadpool that starts threads with the appropriate arguments, while restricting the number of concurrent running threads below a user defined limit. It also connects the threads to the supplied user callbacks.
-
_IdleObject
Like a normal gobject.GObject but emits all signals in the main thread
-
__FooThread_
A simple class which derives from both threading.Thread and _IdleObject. All work is done in run() and a signals are emitted when the thread completes and to show progress
-
Demo
A simple demo (see screenshot) which can start a whole bunch of threads and receive notification when they complete.
Anyway, the Example Code is a bit contrived and will certainly need some customization by the user but nonetheless may still be useful to others.
Update: Thanks to comments I fixed up a thread-safety issue. I had misunderstood that signal handlers get run immediately in emit(). Now the code emit()s on an idle_handler so all signals and callbacks are run in the main thread. As i mentioned, I use this approach in situations where the threads may run and block for a long time, so the burden of processing all the signals in the main thread is not a big deal as they do not occur frequently.
Update 2: Added progress reporting
Nov 20, 2006
Is it just me or are all the cool apps written in python these days?
-
Conduit (hehe thats me!)
-
Listen (music management and playback)
-
Exaile! (amarok styly music app for GTK)
-
GPixPod (manage pictures for your ipod)
-
Specto (why procrastinate when you can have an application do it for you)
-
elisa and pigment (media center by the dudes at Fluendo)
-
Gimmie (desktop revisited)
-
Deskbar (the most useful app on my desktop)
For extra bonus points I can also freely borrow code from these projects because they are GPL licensed. Yay for FOSS!
