Python meme 2009

Here’s a short, 5 questions, 2009 Python meme. Copy-paste the questions, and blog your answers !

1. What’s the coolest Python application, framework or library you have discovered in 2009 ?

SQLAlchemy, not only does it fully (properly) abstract the SQL database for you, but provides a full Object Relational Mapping. What's more, it's Python 3 ready.

2. What new programming technique did you learn in 2009 ?

Using GObject Introspection to mix several languages (ie, Python and Javascript) in the same application.

3. What’s the name of the open source project you contributed the most in 2009 ? What did you do ?

Concordance XMPP, an XMPP service framework for Python I started January 1st 2009.

4. What was the Python blog or website you read the most in 2009 ?

Python Planet

5. What are the three top things you want to learn in 2010 ?

1. SHARC DSPs
2. Welding
3. Aikido

peer1/serverbeach severe failure

We've seen a greater than average share of service outages at Serverbeach; incompetent electricians blowing out the battery backup system causing an evening of downtime, network failures taking the server down for hours at a time, other random outages without explanation.

Of course these outages affect large numbers of customers so you get 500 errors when trying to access the customer portal or put on hold for up to an hour waiting to talk to someone only able to enter a ticket into the system on your behalf.

This latest outage tops them all, however. Yesterday afternoon they attempted to merge the Peer1 and Serverbeach customer portals which knocked out DNS for a "large number" of customers. They sent out an email to everyone and I fully expected it to be a short outage.

Last night at around 8pm all of our domains were still down. I attempted to log into the customer portal to find my old password not working, and after a lengthy process to recover it I found myself locked out of it unless I agreed to a lengthy new service agreement which I certainly don't have time to read through in the middle of an outage.

After sitting on hold with customer service a ticket was filed. Which I replied to, and replied to, and have only received blanket emails asking to reply again if we're still out. More than 24 hours later we are still down.

Serverbeach claims they refund customers for outages, but I have yet to receive a dime credited for any past issues and I fully expect them to make getting credit for this difficult as well. At this point I don't think it's fair to myself or the communities which rely on this server to continue hosting with them.

I've found a really decent deal with Hurricane Electric with the added perks of IPV6, more bandwidth, and newer hardware for less than I'm paying now.

Gentoo wins again with Python 3

Despite being heavily involved in Ubuntu advocacy and promotion, being a member of the Ubuntu project, and using it on my personal systems, I've had to keep my servers and development machines running Gentoo.

When developing software, we need to work on what will be "mainstream" in 6 to 12 months, not what's already mainstream. When a new version of our physics library is released, it may depreciate certain methods or include new features we'll really want in our release, but that version won't be packaged until 2-8 months after the new library was released, given the typical release cycles and "feature freeze" stopping new packages from being included too close to release time.

Python 3 is a great example of this; Gnome 3 will apparently use Python 3 for many applets, and Gnome 3's release schedule sets it for inclusion in Ubuntu 10.10 (next Fall), but at present Python 3.1 is packaged but none of the 3rd party packages are available for it - even those which explicitly support Python 3 already.

While doing my periodic Gentoo upgrade, I just noticed that Portage, the package system for Gentoo, can now be built to run on Python 3 directly. When you have both Python 2 and 3 installed on your system, packages wich support both will build and install for both, and their packaged versions are often available just days after release. From what I've seen, Gentoo is one of the best distributions for Python developers right now.

I wouldn't recommend that my family run Gentoo, many of whom happily run Ubuntu, but it's lack of support for developers is concerning to me. I do not believe it should be difficult to both support new Linux users and experienced developers with the same distribution.

Dell employee discounts a joke

I just learned that the Dell employee discount, which is offered to employees of numerous Dell partners, does not apply to their N-series laptops, even though many people only have this discount because of their work with Ubuntu.

The irony continues to drip as the models I compared between the employee purchase price and the Ubuntu versions of the same, such as the Inspiron 15 and Inspiron 15n, roughly discounted the added cost of Windows 7 only. It appears that the employee discount equates to free Windows 7.

The current Dell Ubuntu offerings lack in many other areas, such as not offering gigabit ethernet or 802.11n as options. While I would love to support Dell in working with the Ubuntu community more, I really can't justify buying a sub-par laptop at the prices they're offering them at.

My friend Mackenzie showed off her System76 laptop at the 9.10 release party a few weeks ago. All of their laptops include gigabit ethernet, 802.11agn wireless, and many other options Dell doesn't offer. They're a bit more pricey, but according to Mackenzie their service is significantly better.

moving to Bullet

Bullet has some really great features over ODE; soft body physics, particle physics, and hardware acceleration among the top.

After talking with a Crystal Space developer about this, I put it on my list of things to look into when PySoy is a bit more mature. However, a Debian packager's decision has accelerated these plans.

ODE is not designed to be installed as a single shared library, it has many build-time options which radically change it's behaviour - such as whether the library uses single or double precision, or whether to calculate gyroscopic force. As a general rule, distributions stick to the defaults when packaging as much as possible, but Debian (and thus Ubuntu and other Debian-based distros) ship a single system-wide library with non-default compile flags (Double precision) that would result in sub-par performance, network sync issues, and in some cases bugs in games.

Debian could have shipped four or eight different versions of the ODE library, such that games could be linked against the one it was designed for and work as expected, or ODE could have been designed to build both a single and double precision library with the other options as runtime flags, but instead we game developers are caught in the middle.

We fought this battle before with Soya, for some time actually, and in the end Soya had to ship their own version of ODE embedded in the source and statically linked to get around the problem. Even then it was an issue to get the package including a static ODE included. It's a battle not worth fighting.

While Bullet also supports building as double precision, it's an option so rarely used that no distro builds with it (Gentoo doesn't even have a USE flag to switch it), and given that Bullet is currently not packaged for Debian it'd be easier to add it as a new proper package than fight Debian packages to improve their build of ODE.

I am not looking forward to working with Bullet's incomplete C-API, likely having to fill in some gaps as we work on it, nor writing the Vala bindings for Bullet, but it'd be a lot faster and fun - not to mention resulting in new features.

beta-3 roadmap updated

I updated the Roadmap for 1.0-beta3 today.

In short, we're dropping ambitions for audio support in this release. Once our migration to Vala is complete and the related changes implemented we'll release without further delay. After that we'll begin work on the newer features like audio, networking, and browser embedding for beta4 while the community tests and gives feedback on beta3.

There are a lot of new features already compared to the beta2 release. We're aiming for a beta3 release by January 1st 2010, 2 years from the beta2 release.

Kudos for Mercurial going GPLv2+

The largest problem with integrating Mercurial with our new web framework is being resolved; they are upgrading Mercurial's license to GPLv2 or later.

Thanks to this, our favoured dVCS will be available for integrating into GPLv3 and AGPLv3 licensed projects without any weird hacks or work-arounds.

adding full Python support to Vala

We've found a way to finish full Python support for Vala; plugin support will be added to valac soon and a PyObject plugin added to our distutils extension for handling Vala and Genie extension source files.

I'll be focusing on this over the next two weeks, with a target on Python 3.1. We're going with 3.1 as baseline for PyCapsule, which will help ensure type safety when passing GType objects through Python.

In the meantime work on libsoy can continue by the rest of the team.

vala support beyond python-3.0.vapi

While the vapi is still a primary need, we also need PyObject and PyTypeObject support in Vala itself to generate the correct code.

This will result in both cleaner .vala source and more readable generated .c code, so this time investment will pay off in PySoy development in the long run.

the sad state of domain registrars

My first business, started when I was 17, was an ISP - run out of my bedroom with a Portmaster and Redhat servers. I offered some unused bandwidth for a Tucows mirror and was later offered to join their pilot OpenSRS program.

Of course, there's many OpenSRS resellers today. Anyone paying a nominal fee can get setup to sell $10/year domain names purchased as bulk domain credits from OpenSRS. It's a great service that has really opened up hosting and registration for small businesses.

When pysoy.org was registered, I went with an OpenSRS reseller called IPower who was advertising domain registration at only a small margin above cost. The first trouble we ran into was, while the credit card was charged and control panel access granted, the registrations didn't actually get done on OpenSRS. It took 3+ hours on the phone with a tech support rep who finally had to re-create the accounts to get the registrations through their flaky system.

Their control panel was made of a series of icons, appearing to have been made using Windows Paint, with almost all of them not working because I wasn't getting hosting from them. After quite awhile of clicking into poorly named icons I found that it would not allow me to update the nameservers, but would allow me to setup custom zone files on their nameservers - with a limited number of subdomains.

Now I will say, this has been working mostly trouble-free since 2006. Last year my former partner, who was paying for these domains on his credit card, elected to remove his card from the account and we disabled auto-renewal. This year their notices for renewal ended up in my spam folder, and they ignored or reset the auto-renew settings, charging his card once again. He calls them, where they set him up as the authorized contact (note, his name only appears as the credit card owner which was removed a year prior, he is none of the domain contacts) in the agreement that he chooses not to challenge their illegal charges to his card and they agree to remove his card from the accounts. In short, it wasn't worth $30 to fight it.

Now, everything is amicable between us, but their willingness to add someone to my account without contacting me is a bit frightening. In checking my spam folder this morning I noticed some new emails from them stating that they would now also be charging $8.99 per domain for domain privacy, which was previously free and which payment was never agreed to, unless I opted-out by tomorrow.

Of course, calling them wouldn't be so easy, after three attempts to contact billing only to get put on hold for ~10 minutes each only to get dead air and a "click". After speaking to several reps, who all demanded the security code set by my former partner when he called (which I don't have) and over an hour struggling to get their control panel to give me new passwords, I finally get a security code set for one of the reps to speak to me. His tells me "a specialist" will get back to me about removing domain privacy.

I am very tired of this, and the truth is this only slightly worse than my experience with most domain registrars. I'm going to do what I should have done in 2006, near the end of the registration periods of all my domains I'm just going to transfer them to my own OpenSRS reseller account and pay the flat $10/year.

I'll extend this to any other free software project managers out there, if you're tired of dealing with crap like what I've gone through with IPower, I'll give you at-cost pricing on your domains. If your site is Python-powered (WSGI) I'll throw in hosting for a beer next time you see me.

My email address is my full name @ubuntu.com.

pysoy.org website upgrades

As some have noted, we recently added an ad to PySoy's website. This is actually only one of the many changes to prepare for the beta3 release.

I'll be putting in a good deal of work over the next few weeks upgrading the website for HTML/5 and XMPP, better separation of functionality to subdomains and standardizing the templates between these subdomains and hg.pysoy.org and planet.pysoy.org/.

Help is, of course, always sought and welcome! If you're good with CSS or want some experience working with HTML/5 drop me a line.

Vala migration continues

We're pretty far into the big Vala migration now, enough to talk about where we're going with it.

The most notable new component is libsoy, a glib-based library written in Vala that integrates all our dependencies to provide the non-Python functionality of PySoy.

This abstraction ensures that all PySoy's background threads remain GIL-free and will be useful for many implementations of "cloud gaming", which I'll go into more detail on later.

The largest task remaining is python-3.0.vapi, which will allow us to write the Python extension modules for libsoy. These will not be simple Python bindings which could be provided with PyBank as PySoy includes many complex Python datatypes which the gobject model cannot provide.

summer of code students announced

It's been a stressful few weeks, but we've selected 30 students to work with the Python Software Foundation for Google's Summer of Code.

I don't know what I was thinking in volunteering to be the PSF's admin this year, but I'm happy to have had the opportunity. I believe we have more students working on core Python projects this year than all previous years and certainly a good start with getting students involved in our community.

There's a lot more work to be done; getting students setup on the SoC Python planet, getting all the backup mentors and coordinators organized, making sure students are taking advantage of the community bonding period, etc.

PySoy ended up with one student, Jo Yo (aka Lucas Westine) working on character animation. He worked with the Ogre3d team last year on a similar project. He's being mentored by Eric Stein (aka Toba) who wrote PySoy's soy.fields module as a Summer of Code student in 2007.

Building a better Pyrex - part 2

Vala, the C#'ish language from the Gnome project for building GObject-based applications, may not seem like the most intuitive choice for building Python extension modules.

For PySoy, she may be our long-sought salvation from Pyrex.

First to note, PySoy is based heavily on GLib. While much of GLib is redundant to types and functions available with Python's c-api, the GLib functions don't require Python's GIL to be used. This is what allows PySoy to take full advantage of multiple cores and run Python code in parallel with rendering and physics processing.

Second of note, as I wrote in my last post, Pyrex/Cython are not appropriate for PySoy as an overwhelming majority of our code is inside with nogil:. Many Pyrex/Cython language features, what makes them easy to work with, are thus unavailable to us. The language becomes a burden and "GIL minefield" that makes working with it difficult.

Like Pyrex, Vala "compiles" clean object-oriented code to C, unlike Pyrex we have full and direct access to all datatypes and classes without the GIL. Like Pyrex, Vala needs pseudo-headers to instruct it how to use C libraries, unlike Pyrex the .vapi files give us an OO API for working with those libraries.

This was the real deal maker though; Vala is flexible enough to use PyObject and family directly. Not only do we get GObject access but with python.vapi, full OO access to Python's c-api with automatic incref/decref and easy Python/C type conversion.

so long, Pyrex!

Almost four years ago PySoy started as a fork of a homebrew game engine called Soya3D. Despite being rewritten four times we were unable to find a suitable replacement for Pyrex, the meta-language Soya3D was written in.

Pyrex is alright for many purposes, but not for what we were using it for. Cython (a fork of Pyrex) has improved many things but not in the areas we need. Our "development" has ground into a series of tracking down bugs generated automatically in .c code because we didn't tell Pyrex not to do something, or because Pyrex's author didn't consider a certain use case when he wrote it.

After almost four years I've found a viable escape plan all thanks glib's GObjects and PyGObject.

The challenge that we've always faced is threading in PySoy without using the GIL (Global Interpreter Lock). 90%+ of the code in PySoy is only run in threads which never hold the GIL to ensure each background thread will only ever block on it's own functionality; rendering thread blocks on the GPU, physics thread blocks on rendering a scene, IO thread blocks in poll(), audio thread blocks on the sound card, etc. We've used glib's AsyncQueue to use Python callbacks from these no-GIL threads without that callback code interrupting what they're doing.

We used to believe that Pyrex made this all easy since the nogil C code that operated on a Python object could be written in a class-like manner with C attributes accessed like Python class attributes, ie:

cdef class Foo :
  cdef int alpha
  def __init__(self, value) :
    self.alpha = value
  def __call__(self) :
    return alpha

Like Soya3D, Pyrex's simplicity and elegance lures you in and so long as you don't try to, say, build a multithreaded game engine with it, it works great. Please don't misunderstand me, Pyrex is great for most applications that use it, we just out-grew it years ago and have wasted far too much time trying to make it work.

GObjects provides an even more elegant solution, write the C code in C, and write the interface code (import soy) in Python using PyGObject. The C code never includes Python.h and thus can never have any issues with the GIL.

So here begins our 5th revision of PySoy; function by function, type by type, extension by extension, refactored into C as GObjects. It'll take a bit to shift the code over in pieces without breaking everything as we do, but when this is done we can replace all the Pyrex source files with a Python package.

No custom languages for developers to learn, no more "with nogil:" everywhere, or having to refactor every C header we want to use to a .pxd file, or having to search through the C sources Pyrex generates for bugs.

so long, Pyrex! You made PySoy possible, but also caused us all countless hours of extra work and frustration. In retrospect, I wish we never met you.

sqlite and postgres support in Python 3 - today

I've been working with the SQLAlchemy package in Python 3, their pre-0.6 branch anyway.

Currently it offers sqlite (included with Py3) and postgres (via py8000) with their Object Relational Mapper and lower-level SQL Expression Language.

My needs are covered given that sqlite and postgres are the only two databases I find myself normally using. Of course there are those who will balk, "but it doesn't support {'MySQL', 'Oracle', 'MSSQL'} yet!". To the people who need those databases, it's easier to spend a few days helping to port the appropriate dbapi package than wasting energy with complaints.

For web frameworks, where the multi-db abstraction SQLAlchemy provides is vital, it's available to develop on today (svn checkout, python3 sa2to3.py). They'll provide a separate -py3 tarball with the 0.6 release planned for after PyCon.

git vs mercurial

In the past we've stuck with subversion for PySoy over git because about half our team are Windows devs and git has very poor Windows support. Some of us have been using git-svn to use git locally while not making life difficult for the other half.

Last night I migrated the Concordance-XMPP project to mercurial as a trial.

First, I have to say that Mercurial's subversion migration tools need work - simple things like author lookup tables (svn username -> Full Name <email>), option to turn off auto-tagging, and a template for modifying commit messages so we can turn off marking the history up with "[rXXX]".

These things were easy enough given that Mercurial is Python-based. Two hours later I had a proper migration.

In contrast with my earlier attempts at migrating to git, mercurial was actually a joy to setup on the server. WSGI, using our existing .htusers file from Trac, a small config script, done. This contrast to git which would require some funky SSH sandboxing and/or PAM setup - yes it can be done, but such hacks shouldn't be needed.

To all our surprise "hg clone" was extremely fast even compared to subversion and especially in comparison to git. One development commented that it seemed "instant". Even for the small commit history (2 months, about 130 commits) git could take several minutes to clone and process Concordance's history.

I'm disappointed to see GPLv2-only on Mercurial as we won't be able to use it directly in any GPLv3 or AGPLv3 projects. Upward license compatability good, anti-FSF paranoia bad.

Thankfully hg's protocol is easily derived and can be implemented by other Python software without having to actually import any part of hg. Hopefully the Mercurial's license situation will be solved at some point to minimize redundant work.

Avahi to be considered harmful

Let me paint a picture as it's come into focus for me recently. Apple develops a protocol called mDNS for zeroconf via multicast DNS. They call it Bonjour and are serious enough about making it a standard they release a free software library called mDNSResponder to support it which runs on GNU/Linux, OSX, and Windows.

A community project, Avahi, implements their own mDNS suite for GNU/Linux. Avahi adds some nice features, such as dbus support. To help apps use either option they support compatibility with mDNSResponder's API.

It's great to have this sort of cooperative-competition, but they took this a bit too far by printing a nag warning when their mDNSResponder compatibility API is used by an application and otherwise pressuring mDNS-using apps to drop support for mDNSResponder.

I sincerely hope Python projects aiming to implement mDNS build against the mDNSResponder API, for cross platform support if nothing else, and encourage their users to avoid using Avahi when questions about this nag page come up.

Concordance register.py

I have the main Python callback in Concordance working now to handle <iq/>, <message/>, and <presence/> elements. Eventually it'll only handle these elements when they're directed to the server, otherwise the server will route the message to it's destination without Python needing to process it first.

examples/register.py demonstrates user registration handling using ElementTree (or at least the start of the process for now). The result is shown here.

Concordance lives

Concordance-XMPP now listens and responds to new XMPP connections with basic XML parsing via expat.

David has been working with me to maintain Windows compatability. I'm working on SASL support and the Python callback API.

With luck the first (very early) release will be ready this weekend which can handle logging in and intra-server routing between users.

Python 3.0 - first extension module

After hours of hunting down docs and debugging my build environment (turns out I had a 3.0-alpha5 lurking around mucking up setup.py)..

Python 3.0 (r30:67503, Jan  2 2009, 00:20:15) 
[GCC 4.3.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import concordance
>>> concordance.__file__
'/usr/lib/python3.0/site-packages/concordance.so'
>>>

New Year, New Project

The first day of 2009, PySoy's copyright notices updated and a few tickets closed, I hit a point where I really need a decent XMPP server.

I've started a new project called Concordance, a XMPP/Jingle server written in C (message routing and RTP mixing) and Python 3.0 (authentication, login, MUC handling, etc).

The goal is a framework giving Python full control over the server, for nearly unlimited configurability, while having the speed to serve a large user base with a minimal of CPU use.