Saturday, March 21, 2009

so long, Pyrex!

Almost four years ago PySoy started as a fork of a homebrew game engine called Soya3D. Despite being rewritten four times we were unable to find a suitable replacement for Pyrex, the meta-language Soya3D was written in.

Pyrex is alright for many purposes, but not for what we were using it for. Cython (a fork of Pyrex) has improved many things but not in the areas we need. Our "development" has ground into a series of tracking down bugs generated automatically in .c code because we didn't tell Pyrex not to do something, or because Pyrex's author didn't consider a certain use case when he wrote it.

After almost four years I've found a viable escape plan all thanks glib's GObjects and PyGObject.

The challenge that we've always faced is threading in PySoy without using the GIL (Global Interpreter Lock). 90%+ of the code in PySoy is only run in threads which never hold the GIL to ensure each background thread will only ever block on it's own functionality; rendering thread blocks on the GPU, physics thread blocks on rendering a scene, IO thread blocks in poll(), audio thread blocks on the sound card, etc. We've used glib's AsyncQueue to use Python callbacks from these no-GIL threads without that callback code interrupting what they're doing.

We used to believe that Pyrex made this all easy since the nogil C code that operated on a Python object could be written in a class-like manner with C attributes accessed like Python class attributes, ie:

cdef class Foo :
cdef int alpha
def __init__(self, value) :
self.alpha = value
def __call__(self) :
return alpha

Like Soya3D, Pyrex's simplicity and elegance lures you in and so long as you don't try to, say, build a multithreaded game engine with it, it works great. Please don't misunderstand me, Pyrex is great for most applications that use it, we just out-grew it years ago and have wasted far too much time trying to make it work.

GObjects provides an even more elegant solution, write the C code in C, and write the interface code (import soy) in Python using PyGObject. The C code never includes Python.h and thus can never have any issues with the GIL.

So here begins our 5th revision of PySoy; function by function, type by type, extension by extension, refactored into C as GObjects. It'll take a bit to shift the code over in pieces without breaking everything as we do, but when this is done we can replace all the Pyrex source files with a Python package.

No custom languages for developers to learn, no more "with nogil:" everywhere, or having to refactor every C header we want to use to a .pxd file, or having to search through the C sources Pyrex generates for bugs.

so long, Pyrex! You made PySoy possible, but also caused us all countless hours of extra work and frustration. In retrospect, I wish we never met you.

Wednesday, March 18, 2009

sqlite and postgres support in Python 3 - today

I've been working with the SQLAlchemy package in Python 3, their pre-0.6 branch anyway.

Currently it offers sqlite (included with Py3) and postgres (via py8000) with their Object Relational Mapper and lower-level SQL Expression Language.

My needs are covered given that sqlite and postgres are the only two databases I find myself normally using. Of course there are those who will balk, "but it doesn't support {'MySQL', 'Oracle', 'MSSQL'} yet!". To the people who need those databases, it's easier to spend a few days helping to port the appropriate dbapi package than wasting energy with complaints.

For web frameworks, where the multi-db abstraction SQLAlchemy provides is vital, it's available to develop on today (svn checkout, python3 sa2to3.py). They'll provide a separate -py3 tarball with the 0.6 release planned for after PyCon.

Sunday, March 08, 2009

git vs mercurial

In the past we've stuck with subversion for PySoy over git because about half our team are Windows devs and git has very poor Windows support. Some of us have been using git-svn to use git locally while not making life difficult for the other half.

Last night I migrated the Concordance-XMPP project to mercurial as a trial.

First, I have to say that Mercurial's subversion migration tools need work - simple things like author lookup tables (svn username -> Full Name <email>), option to turn off auto-tagging, and a template for modifying commit messages so we can turn off marking the history up with "[rXXX]".

These things were easy enough given that Mercurial is Python-based. Two hours later I had a proper migration.

In contrast with my earlier attempts at migrating to git, mercurial was actually a joy to setup on the server. WSGI, using our existing .htusers file from Trac, a small config script, done. This contrast to git which would require some funky SSH sandboxing and/or PAM setup - yes it can be done, but such hacks shouldn't be needed.

To all our surprise "hg clone" was extremely fast even compared to subversion and especially in comparison to git. One development commented that it seemed "instant". Even for the small commit history (2 months, about 130 commits) git could take several minutes to clone and process Concordance's history.

I'm disappointed to see GPLv2-only on Mercurial as we won't be able to use it directly in any GPLv3 or AGPLv3 projects. Upward license compatability good, anti-FSF paranoia bad.

Thankfully hg's protocol is easily derived and can be implemented by other Python software without having to actually import any part of hg. Hopefully the Mercurial's license situation will be solved at some point to minimize redundant work.