Saturday, March 21, 2009

so long, Pyrex!

Almost four years ago PySoy started as a fork of a homebrew game engine called Soya3D. Despite being rewritten four times we were unable to find a suitable replacement for Pyrex, the meta-language Soya3D was written in.

Pyrex is alright for many purposes, but not for what we were using it for. Cython (a fork of Pyrex) has improved many things but not in the areas we need. Our "development" has ground into a series of tracking down bugs generated automatically in .c code because we didn't tell Pyrex not to do something, or because Pyrex's author didn't consider a certain use case when he wrote it.

After almost four years I've found a viable escape plan all thanks glib's GObjects and PyGObject.

The challenge that we've always faced is threading in PySoy without using the GIL (Global Interpreter Lock). 90%+ of the code in PySoy is only run in threads which never hold the GIL to ensure each background thread will only ever block on it's own functionality; rendering thread blocks on the GPU, physics thread blocks on rendering a scene, IO thread blocks in poll(), audio thread blocks on the sound card, etc. We've used glib's AsyncQueue to use Python callbacks from these no-GIL threads without that callback code interrupting what they're doing.

We used to believe that Pyrex made this all easy since the nogil C code that operated on a Python object could be written in a class-like manner with C attributes accessed like Python class attributes, ie:

cdef class Foo :
cdef int alpha
def __init__(self, value) :
self.alpha = value
def __call__(self) :
return alpha

Like Soya3D, Pyrex's simplicity and elegance lures you in and so long as you don't try to, say, build a multithreaded game engine with it, it works great. Please don't misunderstand me, Pyrex is great for most applications that use it, we just out-grew it years ago and have wasted far too much time trying to make it work.

GObjects provides an even more elegant solution, write the C code in C, and write the interface code (import soy) in Python using PyGObject. The C code never includes Python.h and thus can never have any issues with the GIL.

So here begins our 5th revision of PySoy; function by function, type by type, extension by extension, refactored into C as GObjects. It'll take a bit to shift the code over in pieces without breaking everything as we do, but when this is done we can replace all the Pyrex source files with a Python package.

No custom languages for developers to learn, no more "with nogil:" everywhere, or having to refactor every C header we want to use to a .pxd file, or having to search through the C sources Pyrex generates for bugs.

so long, Pyrex! You made PySoy possible, but also caused us all countless hours of extra work and frustration. In retrospect, I wish we never met you.


Sin said...

Why not ctypes?

Arc Riley said...

Much of PySoy is using Python to create/destroy PyObject* and then passing this around, accessing the C members of the Python object structs, plus mutex to ensure that if Python destroys an object it's removed cleanly from all threads.

GObject adds OO on top of C in a manner almost identical to PyObject, and it's "free" for us since we're already using glib heavily.

Also PyGObject is much cleaner and simpler than ctypes, not to mention safer, if you're aleady using glib/GObject it'd be silly to use ctypes.

Sin said...

That makes sense. Perhaps when Vala and PyBank get more mature, this kind of thing will be easier. C with GObject is horrible to read.

ajaksu said...

Maybe you should mention things that would have helped you as possible Cython RFEs? Are some of them suitable for PSF's GSoC slots?

Arc Riley said...

In truth, it's not really a lacking in Cython, just a difference of goals.

PySoy is 90%+ C that runs in background threads that never hold the GIL. Cython may be able to provide the Python bindings for this, and some glue code, but it's an uphill battle to use it to effectively write a non-Python C library with.