We've found a way to finish full Python support for Vala; plugin support will be added to valac soon and a PyObject plugin added to our distutils extension for handling Vala and Genie extension source files.
I'll be focusing on this over the next two weeks, with a target on Python 3.1. We're going with 3.1 as baseline for PyCapsule, which will help ensure type safety when passing GType objects through Python.
In the meantime work on libsoy can continue by the rest of the team.
Monday, June 29, 2009
Saturday, June 27, 2009
vala support beyond python-3.0.vapi
Wednesday, June 24, 2009
the sad state of domain registrars
My first business, started when I was 17, was an ISP - run out of my bedroom with a Portmaster and Redhat servers. I offered some unused bandwidth for a Tucows mirror and was later offered to join their pilot OpenSRS program.
Of course, there's many OpenSRS resellers today. Anyone paying a nominal fee can get setup to sell $10/year domain names purchased as bulk domain credits from OpenSRS. It's a great service that has really opened up hosting and registration for small businesses.
When pysoy.org was registered, I went with an OpenSRS reseller called IPower who was advertising domain registration at only a small margin above cost. The first trouble we ran into was, while the credit card was charged and control panel access granted, the registrations didn't actually get done on OpenSRS. It took 3+ hours on the phone with a tech support rep who finally had to re-create the accounts to get the registrations through their flaky system.
Their control panel was made of a series of icons, appearing to have been made using Windows Paint, with almost all of them not working because I wasn't getting hosting from them. After quite awhile of clicking into poorly named icons I found that it would not allow me to update the nameservers, but would allow me to setup custom zone files on their nameservers - with a limited number of subdomains.
Now I will say, this has been working mostly trouble-free since 2006. Last year my former partner, who was paying for these domains on his credit card, elected to remove his card from the account and we disabled auto-renewal. This year their notices for renewal ended up in my spam folder, and they ignored or reset the auto-renew settings, charging his card once again. He calls them, where they set him up as the authorized contact (note, his name only appears as the credit card owner which was removed a year prior, he is none of the domain contacts) in the agreement that he chooses not to challenge their illegal charges to his card and they agree to remove his card from the accounts. In short, it wasn't worth $30 to fight it.
Now, everything is amicable between us, but their willingness to add someone to my account without contacting me is a bit frightening. In checking my spam folder this morning I noticed some new emails from them stating that they would now also be charging $8.99 per domain for domain privacy, which was previously free and which payment was never agreed to, unless I opted-out by tomorrow.
Of course, calling them wouldn't be so easy, after three attempts to contact billing only to get put on hold for ~10 minutes each only to get dead air and a "click". After speaking to several reps, who all demanded the security code set by my former partner when he called (which I don't have) and over an hour struggling to get their control panel to give me new passwords, I finally get a security code set for one of the reps to speak to me. His tells me "a specialist" will get back to me about removing domain privacy.
I am very tired of this, and the truth is this only slightly worse than my experience with most domain registrars. I'm going to do what I should have done in 2006, near the end of the registration periods of all my domains I'm just going to transfer them to my own OpenSRS reseller account and pay the flat $10/year.
I'll extend this to any other free software project managers out there, if you're tired of dealing with crap like what I've gone through with IPower, I'll give you at-cost pricing on your domains. If your site is Python-powered (WSGI) I'll throw in hosting for a beer next time you see me.
My email address is my full name @ubuntu.com.
Of course, there's many OpenSRS resellers today. Anyone paying a nominal fee can get setup to sell $10/year domain names purchased as bulk domain credits from OpenSRS. It's a great service that has really opened up hosting and registration for small businesses.
When pysoy.org was registered, I went with an OpenSRS reseller called IPower who was advertising domain registration at only a small margin above cost. The first trouble we ran into was, while the credit card was charged and control panel access granted, the registrations didn't actually get done on OpenSRS. It took 3+ hours on the phone with a tech support rep who finally had to re-create the accounts to get the registrations through their flaky system.
Their control panel was made of a series of icons, appearing to have been made using Windows Paint, with almost all of them not working because I wasn't getting hosting from them. After quite awhile of clicking into poorly named icons I found that it would not allow me to update the nameservers, but would allow me to setup custom zone files on their nameservers - with a limited number of subdomains.
Now I will say, this has been working mostly trouble-free since 2006. Last year my former partner, who was paying for these domains on his credit card, elected to remove his card from the account and we disabled auto-renewal. This year their notices for renewal ended up in my spam folder, and they ignored or reset the auto-renew settings, charging his card once again. He calls them, where they set him up as the authorized contact (note, his name only appears as the credit card owner which was removed a year prior, he is none of the domain contacts) in the agreement that he chooses not to challenge their illegal charges to his card and they agree to remove his card from the accounts. In short, it wasn't worth $30 to fight it.
Now, everything is amicable between us, but their willingness to add someone to my account without contacting me is a bit frightening. In checking my spam folder this morning I noticed some new emails from them stating that they would now also be charging $8.99 per domain for domain privacy, which was previously free and which payment was never agreed to, unless I opted-out by tomorrow.
Of course, calling them wouldn't be so easy, after three attempts to contact billing only to get put on hold for ~10 minutes each only to get dead air and a "click". After speaking to several reps, who all demanded the security code set by my former partner when he called (which I don't have) and over an hour struggling to get their control panel to give me new passwords, I finally get a security code set for one of the reps to speak to me. His tells me "a specialist" will get back to me about removing domain privacy.
I am very tired of this, and the truth is this only slightly worse than my experience with most domain registrars. I'm going to do what I should have done in 2006, near the end of the registration periods of all my domains I'm just going to transfer them to my own OpenSRS reseller account and pay the flat $10/year.
I'll extend this to any other free software project managers out there, if you're tired of dealing with crap like what I've gone through with IPower, I'll give you at-cost pricing on your domains. If your site is Python-powered (WSGI) I'll throw in hosting for a beer next time you see me.
My email address is my full name @ubuntu.com.
Monday, June 15, 2009
pysoy.org website upgrades
As some have noted, we recently added an ad to PySoy's website. This is actually only one of the many changes to prepare for the beta3 release.
I'll be putting in a good deal of work over the next few weeks upgrading the website for HTML/5 and XMPP, better separation of functionality to subdomains and standardizing the templates between these subdomains and hg.pysoy.org and planet.pysoy.org/.
Help is, of course, always sought and welcome! If you're good with CSS or want some experience working with HTML/5 drop me a line.
I'll be putting in a good deal of work over the next few weeks upgrading the website for HTML/5 and XMPP, better separation of functionality to subdomains and standardizing the templates between these subdomains and hg.pysoy.org and planet.pysoy.org/.
Help is, of course, always sought and welcome! If you're good with CSS or want some experience working with HTML/5 drop me a line.
Saturday, June 13, 2009
Vala migration continues
We're pretty far into the big Vala migration now, enough to talk about where we're going with it.
The most notable new component is libsoy, a glib-based library written in Vala that integrates all our dependencies to provide the non-Python functionality of PySoy.
This abstraction ensures that all PySoy's background threads remain GIL-free and will be useful for many implementations of "cloud gaming", which I'll go into more detail on later.
The largest task remaining is python-3.0.vapi, which will allow us to write the Python extension modules for libsoy. These will not be simple Python bindings which could be provided with PyBank as PySoy includes many complex Python datatypes which the gobject model cannot provide.
The most notable new component is libsoy, a glib-based library written in Vala that integrates all our dependencies to provide the non-Python functionality of PySoy.
This abstraction ensures that all PySoy's background threads remain GIL-free and will be useful for many implementations of "cloud gaming", which I'll go into more detail on later.
The largest task remaining is python-3.0.vapi, which will allow us to write the Python extension modules for libsoy. These will not be simple Python bindings which could be provided with PyBank as PySoy includes many complex Python datatypes which the gobject model cannot provide.
Tuesday, April 21, 2009
summer of code students announced
It's been a stressful few weeks, but we've selected 30 students to work with the Python Software Foundation for Google's Summer of Code.
I don't know what I was thinking in volunteering to be the PSF's admin this year, but I'm happy to have had the opportunity. I believe we have more students working on core Python projects this year than all previous years and certainly a good start with getting students involved in our community.
There's a lot more work to be done; getting students setup on the SoC Python planet, getting all the backup mentors and coordinators organized, making sure students are taking advantage of the community bonding period, etc.
PySoy ended up with one student, Jo Yo (aka Lucas Westine) working on character animation. He worked with the Ogre3d team last year on a similar project. He's being mentored by Eric Stein (aka Toba) who wrote PySoy's soy.fields module as a Summer of Code student in 2007.
I don't know what I was thinking in volunteering to be the PSF's admin this year, but I'm happy to have had the opportunity. I believe we have more students working on core Python projects this year than all previous years and certainly a good start with getting students involved in our community.
There's a lot more work to be done; getting students setup on the SoC Python planet, getting all the backup mentors and coordinators organized, making sure students are taking advantage of the community bonding period, etc.
PySoy ended up with one student, Jo Yo (aka Lucas Westine) working on character animation. He worked with the Ogre3d team last year on a similar project. He's being mentored by Eric Stein (aka Toba) who wrote PySoy's soy.fields module as a Summer of Code student in 2007.
Wednesday, April 01, 2009
Building a better Pyrex - part 2
Vala, the C#'ish language from the Gnome project for building GObject-based applications, may not seem like the most intuitive choice for building Python extension modules.For PySoy, she may be our long-sought salvation from Pyrex.
First to note, PySoy is based heavily on GLib. While much of GLib is redundant to types and functions available with Python's c-api, the GLib functions don't require Python's GIL to be used. This is what allows PySoy to take full advantage of multiple cores and run Python code in parallel with rendering and physics processing.
Second of note, as I wrote in my last post, Pyrex/Cython are not appropriate for PySoy as an overwhelming majority of our code is inside
with nogil:. Many Pyrex/Cython language features, what makes them easy to work with, are thus unavailable to us. The language becomes a burden and "GIL minefield" that makes working with it difficult.Like Pyrex, Vala "compiles" clean object-oriented code to C, unlike Pyrex we have full and direct access to all datatypes and classes without the GIL. Like Pyrex, Vala needs pseudo-headers to instruct it how to use C libraries, unlike Pyrex the .vapi files give us an OO API for working with those libraries.
This was the real deal maker though; Vala is flexible enough to use PyObject and family directly. Not only do we get GObject access but with python.vapi, full OO access to Python's c-api with automatic incref/decref and easy Python/C type conversion.
Saturday, March 21, 2009
so long, Pyrex!
Almost four years ago PySoy started as a fork of a homebrew game engine called Soya3D. Despite being rewritten four times we were unable to find a suitable replacement for Pyrex, the meta-language Soya3D was written in.
Pyrex is alright for many purposes, but not for what we were using it for. Cython (a fork of Pyrex) has improved many things but not in the areas we need. Our "development" has ground into a series of tracking down bugs generated automatically in .c code because we didn't tell Pyrex not to do something, or because Pyrex's author didn't consider a certain use case when he wrote it.
After almost four years I've found a viable escape plan all thanks glib's GObjects and PyGObject.
The challenge that we've always faced is threading in PySoy without using the GIL (Global Interpreter Lock). 90%+ of the code in PySoy is only run in threads which never hold the GIL to ensure each background thread will only ever block on it's own functionality; rendering thread blocks on the GPU, physics thread blocks on rendering a scene, IO thread blocks in poll(), audio thread blocks on the sound card, etc. We've used glib's AsyncQueue to use Python callbacks from these no-GIL threads without that callback code interrupting what they're doing.
We used to believe that Pyrex made this all easy since the nogil C code that operated on a Python object could be written in a class-like manner with C attributes accessed like Python class attributes, ie:
Like Soya3D, Pyrex's simplicity and elegance lures you in and so long as you don't try to, say, build a multithreaded game engine with it, it works great. Please don't misunderstand me, Pyrex is great for most applications that use it, we just out-grew it years ago and have wasted far too much time trying to make it work.
GObjects provides an even more elegant solution, write the C code in C, and write the interface code (import soy) in Python using PyGObject. The C code never includes Python.h and thus can never have any issues with the GIL.
So here begins our 5th revision of PySoy; function by function, type by type, extension by extension, refactored into C as GObjects. It'll take a bit to shift the code over in pieces without breaking everything as we do, but when this is done we can replace all the Pyrex source files with a Python package.
No custom languages for developers to learn, no more "with nogil:" everywhere, or having to refactor every C header we want to use to a .pxd file, or having to search through the C sources Pyrex generates for bugs.
so long, Pyrex! You made PySoy possible, but also caused us all countless hours of extra work and frustration. In retrospect, I wish we never met you.
Pyrex is alright for many purposes, but not for what we were using it for. Cython (a fork of Pyrex) has improved many things but not in the areas we need. Our "development" has ground into a series of tracking down bugs generated automatically in .c code because we didn't tell Pyrex not to do something, or because Pyrex's author didn't consider a certain use case when he wrote it.
After almost four years I've found a viable escape plan all thanks glib's GObjects and PyGObject.
The challenge that we've always faced is threading in PySoy without using the GIL (Global Interpreter Lock). 90%+ of the code in PySoy is only run in threads which never hold the GIL to ensure each background thread will only ever block on it's own functionality; rendering thread blocks on the GPU, physics thread blocks on rendering a scene, IO thread blocks in poll(), audio thread blocks on the sound card, etc. We've used glib's AsyncQueue to use Python callbacks from these no-GIL threads without that callback code interrupting what they're doing.
We used to believe that Pyrex made this all easy since the nogil C code that operated on a Python object could be written in a class-like manner with C attributes accessed like Python class attributes, ie:
cdef class Foo :
cdef int alpha
def __init__(self, value) :
self.alpha = value
def __call__(self) :
return alpha
Like Soya3D, Pyrex's simplicity and elegance lures you in and so long as you don't try to, say, build a multithreaded game engine with it, it works great. Please don't misunderstand me, Pyrex is great for most applications that use it, we just out-grew it years ago and have wasted far too much time trying to make it work.
GObjects provides an even more elegant solution, write the C code in C, and write the interface code (import soy) in Python using PyGObject. The C code never includes Python.h and thus can never have any issues with the GIL.
So here begins our 5th revision of PySoy; function by function, type by type, extension by extension, refactored into C as GObjects. It'll take a bit to shift the code over in pieces without breaking everything as we do, but when this is done we can replace all the Pyrex source files with a Python package.
No custom languages for developers to learn, no more "with nogil:" everywhere, or having to refactor every C header we want to use to a .pxd file, or having to search through the C sources Pyrex generates for bugs.
so long, Pyrex! You made PySoy possible, but also caused us all countless hours of extra work and frustration. In retrospect, I wish we never met you.
Wednesday, March 18, 2009
sqlite and postgres support in Python 3 - today
I've been working with the SQLAlchemy package in Python 3, their pre-0.6 branch anyway.
Currently it offers sqlite (included with Py3) and postgres (via py8000) with their Object Relational Mapper and lower-level SQL Expression Language.
My needs are covered given that sqlite and postgres are the only two databases I find myself normally using. Of course there are those who will balk, "but it doesn't support {'MySQL', 'Oracle', 'MSSQL'} yet!". To the people who need those databases, it's easier to spend a few days helping to port the appropriate dbapi package than wasting energy with complaints.
For web frameworks, where the multi-db abstraction SQLAlchemy provides is vital, it's available to develop on today (svn checkout, python3 sa2to3.py). They'll provide a separate -py3 tarball with the 0.6 release planned for after PyCon.
Currently it offers sqlite (included with Py3) and postgres (via py8000) with their Object Relational Mapper and lower-level SQL Expression Language.
My needs are covered given that sqlite and postgres are the only two databases I find myself normally using. Of course there are those who will balk, "but it doesn't support {'MySQL', 'Oracle', 'MSSQL'} yet!". To the people who need those databases, it's easier to spend a few days helping to port the appropriate dbapi package than wasting energy with complaints.
For web frameworks, where the multi-db abstraction SQLAlchemy provides is vital, it's available to develop on today (svn checkout, python3 sa2to3.py). They'll provide a separate -py3 tarball with the 0.6 release planned for after PyCon.
Sunday, March 08, 2009
git vs mercurial
In the past we've stuck with subversion for PySoy over git because about half our team are Windows devs and git has very poor Windows support. Some of us have been using git-svn to use git locally while not making life difficult for the other half.
Last night I migrated the Concordance-XMPP project to mercurial as a trial.
First, I have to say that Mercurial's subversion migration tools need work - simple things like author lookup tables (svn username -> Full Name <email>), option to turn off auto-tagging, and a template for modifying commit messages so we can turn off marking the history up with "[rXXX]".
These things were easy enough given that Mercurial is Python-based. Two hours later I had a proper migration.
In contrast with my earlier attempts at migrating to git, mercurial was actually a joy to setup on the server. WSGI, using our existing .htusers file from Trac, a small config script, done. This contrast to git which would require some funky SSH sandboxing and/or PAM setup - yes it can be done, but such hacks shouldn't be needed.
To all our surprise "hg clone" was extremely fast even compared to subversion and especially in comparison to git. One development commented that it seemed "instant". Even for the small commit history (2 months, about 130 commits) git could take several minutes to clone and process Concordance's history.
I'm disappointed to see GPLv2-only on Mercurial as we won't be able to use it directly in any GPLv3 or AGPLv3 projects. Upward license compatability good, anti-FSF paranoia bad.
Thankfully hg's protocol is easily derived and can be implemented by other Python software without having to actually import any part of hg. Hopefully the Mercurial's license situation will be solved at some point to minimize redundant work.
Last night I migrated the Concordance-XMPP project to mercurial as a trial.
First, I have to say that Mercurial's subversion migration tools need work - simple things like author lookup tables (svn username -> Full Name <email>), option to turn off auto-tagging, and a template for modifying commit messages so we can turn off marking the history up with "[rXXX]".
These things were easy enough given that Mercurial is Python-based. Two hours later I had a proper migration.
In contrast with my earlier attempts at migrating to git, mercurial was actually a joy to setup on the server. WSGI, using our existing .htusers file from Trac, a small config script, done. This contrast to git which would require some funky SSH sandboxing and/or PAM setup - yes it can be done, but such hacks shouldn't be needed.
To all our surprise "hg clone" was extremely fast even compared to subversion and especially in comparison to git. One development commented that it seemed "instant". Even for the small commit history (2 months, about 130 commits) git could take several minutes to clone and process Concordance's history.
I'm disappointed to see GPLv2-only on Mercurial as we won't be able to use it directly in any GPLv3 or AGPLv3 projects. Upward license compatability good, anti-FSF paranoia bad.
Thankfully hg's protocol is easily derived and can be implemented by other Python software without having to actually import any part of hg. Hopefully the Mercurial's license situation will be solved at some point to minimize redundant work.
Subscribe to:
Posts (Atom)