We've found a way to finish full Python support for Vala; plugin support will be added to valac soon and a PyObject plugin added to our distutils extension for handling Vala and Genie extension source files.
I'll be focusing on this over the next two weeks, with a target on Python 3.1. We're going with 3.1 as baseline for PyCapsule, which will help ensure type safety when passing GType objects through Python.
In the meantime work on libsoy can continue by the rest of the team.
Monday, June 29, 2009
Saturday, June 27, 2009
vala support beyond python-3.0.vapi
Wednesday, June 24, 2009
the sad state of domain registrars
My first business, started when I was 17, was an ISP - run out of my bedroom with a Portmaster and Redhat servers. I offered some unused bandwidth for a Tucows mirror and was later offered to join their pilot OpenSRS program.
Of course, there's many OpenSRS resellers today. Anyone paying a nominal fee can get setup to sell $10/year domain names purchased as bulk domain credits from OpenSRS. It's a great service that has really opened up hosting and registration for small businesses.
When pysoy.org was registered, I went with an OpenSRS reseller called IPower who was advertising domain registration at only a small margin above cost. The first trouble we ran into was, while the credit card was charged and control panel access granted, the registrations didn't actually get done on OpenSRS. It took 3+ hours on the phone with a tech support rep who finally had to re-create the accounts to get the registrations through their flaky system.
Their control panel was made of a series of icons, appearing to have been made using Windows Paint, with almost all of them not working because I wasn't getting hosting from them. After quite awhile of clicking into poorly named icons I found that it would not allow me to update the nameservers, but would allow me to setup custom zone files on their nameservers - with a limited number of subdomains.
Now I will say, this has been working mostly trouble-free since 2006. Last year my former partner, who was paying for these domains on his credit card, elected to remove his card from the account and we disabled auto-renewal. This year their notices for renewal ended up in my spam folder, and they ignored or reset the auto-renew settings, charging his card once again. He calls them, where they set him up as the authorized contact (note, his name only appears as the credit card owner which was removed a year prior, he is none of the domain contacts) in the agreement that he chooses not to challenge their illegal charges to his card and they agree to remove his card from the accounts. In short, it wasn't worth $30 to fight it.
Now, everything is amicable between us, but their willingness to add someone to my account without contacting me is a bit frightening. In checking my spam folder this morning I noticed some new emails from them stating that they would now also be charging $8.99 per domain for domain privacy, which was previously free and which payment was never agreed to, unless I opted-out by tomorrow.
Of course, calling them wouldn't be so easy, after three attempts to contact billing only to get put on hold for ~10 minutes each only to get dead air and a "click". After speaking to several reps, who all demanded the security code set by my former partner when he called (which I don't have) and over an hour struggling to get their control panel to give me new passwords, I finally get a security code set for one of the reps to speak to me. His tells me "a specialist" will get back to me about removing domain privacy.
I am very tired of this, and the truth is this only slightly worse than my experience with most domain registrars. I'm going to do what I should have done in 2006, near the end of the registration periods of all my domains I'm just going to transfer them to my own OpenSRS reseller account and pay the flat $10/year.
I'll extend this to any other free software project managers out there, if you're tired of dealing with crap like what I've gone through with IPower, I'll give you at-cost pricing on your domains. If your site is Python-powered (WSGI) I'll throw in hosting for a beer next time you see me.
My email address is my full name @ubuntu.com.
Of course, there's many OpenSRS resellers today. Anyone paying a nominal fee can get setup to sell $10/year domain names purchased as bulk domain credits from OpenSRS. It's a great service that has really opened up hosting and registration for small businesses.
When pysoy.org was registered, I went with an OpenSRS reseller called IPower who was advertising domain registration at only a small margin above cost. The first trouble we ran into was, while the credit card was charged and control panel access granted, the registrations didn't actually get done on OpenSRS. It took 3+ hours on the phone with a tech support rep who finally had to re-create the accounts to get the registrations through their flaky system.
Their control panel was made of a series of icons, appearing to have been made using Windows Paint, with almost all of them not working because I wasn't getting hosting from them. After quite awhile of clicking into poorly named icons I found that it would not allow me to update the nameservers, but would allow me to setup custom zone files on their nameservers - with a limited number of subdomains.
Now I will say, this has been working mostly trouble-free since 2006. Last year my former partner, who was paying for these domains on his credit card, elected to remove his card from the account and we disabled auto-renewal. This year their notices for renewal ended up in my spam folder, and they ignored or reset the auto-renew settings, charging his card once again. He calls them, where they set him up as the authorized contact (note, his name only appears as the credit card owner which was removed a year prior, he is none of the domain contacts) in the agreement that he chooses not to challenge their illegal charges to his card and they agree to remove his card from the accounts. In short, it wasn't worth $30 to fight it.
Now, everything is amicable between us, but their willingness to add someone to my account without contacting me is a bit frightening. In checking my spam folder this morning I noticed some new emails from them stating that they would now also be charging $8.99 per domain for domain privacy, which was previously free and which payment was never agreed to, unless I opted-out by tomorrow.
Of course, calling them wouldn't be so easy, after three attempts to contact billing only to get put on hold for ~10 minutes each only to get dead air and a "click". After speaking to several reps, who all demanded the security code set by my former partner when he called (which I don't have) and over an hour struggling to get their control panel to give me new passwords, I finally get a security code set for one of the reps to speak to me. His tells me "a specialist" will get back to me about removing domain privacy.
I am very tired of this, and the truth is this only slightly worse than my experience with most domain registrars. I'm going to do what I should have done in 2006, near the end of the registration periods of all my domains I'm just going to transfer them to my own OpenSRS reseller account and pay the flat $10/year.
I'll extend this to any other free software project managers out there, if you're tired of dealing with crap like what I've gone through with IPower, I'll give you at-cost pricing on your domains. If your site is Python-powered (WSGI) I'll throw in hosting for a beer next time you see me.
My email address is my full name @ubuntu.com.
Monday, June 15, 2009
pysoy.org website upgrades
As some have noted, we recently added an ad to PySoy's website. This is actually only one of the many changes to prepare for the beta3 release.
I'll be putting in a good deal of work over the next few weeks upgrading the website for HTML/5 and XMPP, better separation of functionality to subdomains and standardizing the templates between these subdomains and hg.pysoy.org and planet.pysoy.org/.
Help is, of course, always sought and welcome! If you're good with CSS or want some experience working with HTML/5 drop me a line.
I'll be putting in a good deal of work over the next few weeks upgrading the website for HTML/5 and XMPP, better separation of functionality to subdomains and standardizing the templates between these subdomains and hg.pysoy.org and planet.pysoy.org/.
Help is, of course, always sought and welcome! If you're good with CSS or want some experience working with HTML/5 drop me a line.
Saturday, June 13, 2009
Vala migration continues
We're pretty far into the big Vala migration now, enough to talk about where we're going with it.
The most notable new component is libsoy, a glib-based library written in Vala that integrates all our dependencies to provide the non-Python functionality of PySoy.
This abstraction ensures that all PySoy's background threads remain GIL-free and will be useful for many implementations of "cloud gaming", which I'll go into more detail on later.
The largest task remaining is python-3.0.vapi, which will allow us to write the Python extension modules for libsoy. These will not be simple Python bindings which could be provided with PyBank as PySoy includes many complex Python datatypes which the gobject model cannot provide.
The most notable new component is libsoy, a glib-based library written in Vala that integrates all our dependencies to provide the non-Python functionality of PySoy.
This abstraction ensures that all PySoy's background threads remain GIL-free and will be useful for many implementations of "cloud gaming", which I'll go into more detail on later.
The largest task remaining is python-3.0.vapi, which will allow us to write the Python extension modules for libsoy. These will not be simple Python bindings which could be provided with PyBank as PySoy includes many complex Python datatypes which the gobject model cannot provide.
Tuesday, April 21, 2009
summer of code students announced
It's been a stressful few weeks, but we've selected 30 students to work with the Python Software Foundation for Google's Summer of Code.
I don't know what I was thinking in volunteering to be the PSF's admin this year, but I'm happy to have had the opportunity. I believe we have more students working on core Python projects this year than all previous years and certainly a good start with getting students involved in our community.
There's a lot more work to be done; getting students setup on the SoC Python planet, getting all the backup mentors and coordinators organized, making sure students are taking advantage of the community bonding period, etc.
PySoy ended up with one student, Jo Yo (aka Lucas Westine) working on character animation. He worked with the Ogre3d team last year on a similar project. He's being mentored by Eric Stein (aka Toba) who wrote PySoy's soy.fields module as a Summer of Code student in 2007.
I don't know what I was thinking in volunteering to be the PSF's admin this year, but I'm happy to have had the opportunity. I believe we have more students working on core Python projects this year than all previous years and certainly a good start with getting students involved in our community.
There's a lot more work to be done; getting students setup on the SoC Python planet, getting all the backup mentors and coordinators organized, making sure students are taking advantage of the community bonding period, etc.
PySoy ended up with one student, Jo Yo (aka Lucas Westine) working on character animation. He worked with the Ogre3d team last year on a similar project. He's being mentored by Eric Stein (aka Toba) who wrote PySoy's soy.fields module as a Summer of Code student in 2007.
Wednesday, April 01, 2009
Building a better Pyrex - part 2
Vala, the C#'ish language from the Gnome project for building GObject-based applications, may not seem like the most intuitive choice for building Python extension modules.For PySoy, she may be our long-sought salvation from Pyrex.
First to note, PySoy is based heavily on GLib. While much of GLib is redundant to types and functions available with Python's c-api, the GLib functions don't require Python's GIL to be used. This is what allows PySoy to take full advantage of multiple cores and run Python code in parallel with rendering and physics processing.
Second of note, as I wrote in my last post, Pyrex/Cython are not appropriate for PySoy as an overwhelming majority of our code is inside
with nogil:. Many Pyrex/Cython language features, what makes them easy to work with, are thus unavailable to us. The language becomes a burden and "GIL minefield" that makes working with it difficult.Like Pyrex, Vala "compiles" clean object-oriented code to C, unlike Pyrex we have full and direct access to all datatypes and classes without the GIL. Like Pyrex, Vala needs pseudo-headers to instruct it how to use C libraries, unlike Pyrex the .vapi files give us an OO API for working with those libraries.
This was the real deal maker though; Vala is flexible enough to use PyObject and family directly. Not only do we get GObject access but with python.vapi, full OO access to Python's c-api with automatic incref/decref and easy Python/C type conversion.
Saturday, March 21, 2009
so long, Pyrex!
Almost four years ago PySoy started as a fork of a homebrew game engine called Soya3D. Despite being rewritten four times we were unable to find a suitable replacement for Pyrex, the meta-language Soya3D was written in.
Pyrex is alright for many purposes, but not for what we were using it for. Cython (a fork of Pyrex) has improved many things but not in the areas we need. Our "development" has ground into a series of tracking down bugs generated automatically in .c code because we didn't tell Pyrex not to do something, or because Pyrex's author didn't consider a certain use case when he wrote it.
After almost four years I've found a viable escape plan all thanks glib's GObjects and PyGObject.
The challenge that we've always faced is threading in PySoy without using the GIL (Global Interpreter Lock). 90%+ of the code in PySoy is only run in threads which never hold the GIL to ensure each background thread will only ever block on it's own functionality; rendering thread blocks on the GPU, physics thread blocks on rendering a scene, IO thread blocks in poll(), audio thread blocks on the sound card, etc. We've used glib's AsyncQueue to use Python callbacks from these no-GIL threads without that callback code interrupting what they're doing.
We used to believe that Pyrex made this all easy since the nogil C code that operated on a Python object could be written in a class-like manner with C attributes accessed like Python class attributes, ie:
Like Soya3D, Pyrex's simplicity and elegance lures you in and so long as you don't try to, say, build a multithreaded game engine with it, it works great. Please don't misunderstand me, Pyrex is great for most applications that use it, we just out-grew it years ago and have wasted far too much time trying to make it work.
GObjects provides an even more elegant solution, write the C code in C, and write the interface code (import soy) in Python using PyGObject. The C code never includes Python.h and thus can never have any issues with the GIL.
So here begins our 5th revision of PySoy; function by function, type by type, extension by extension, refactored into C as GObjects. It'll take a bit to shift the code over in pieces without breaking everything as we do, but when this is done we can replace all the Pyrex source files with a Python package.
No custom languages for developers to learn, no more "with nogil:" everywhere, or having to refactor every C header we want to use to a .pxd file, or having to search through the C sources Pyrex generates for bugs.
so long, Pyrex! You made PySoy possible, but also caused us all countless hours of extra work and frustration. In retrospect, I wish we never met you.
Pyrex is alright for many purposes, but not for what we were using it for. Cython (a fork of Pyrex) has improved many things but not in the areas we need. Our "development" has ground into a series of tracking down bugs generated automatically in .c code because we didn't tell Pyrex not to do something, or because Pyrex's author didn't consider a certain use case when he wrote it.
After almost four years I've found a viable escape plan all thanks glib's GObjects and PyGObject.
The challenge that we've always faced is threading in PySoy without using the GIL (Global Interpreter Lock). 90%+ of the code in PySoy is only run in threads which never hold the GIL to ensure each background thread will only ever block on it's own functionality; rendering thread blocks on the GPU, physics thread blocks on rendering a scene, IO thread blocks in poll(), audio thread blocks on the sound card, etc. We've used glib's AsyncQueue to use Python callbacks from these no-GIL threads without that callback code interrupting what they're doing.
We used to believe that Pyrex made this all easy since the nogil C code that operated on a Python object could be written in a class-like manner with C attributes accessed like Python class attributes, ie:
cdef class Foo :
cdef int alpha
def __init__(self, value) :
self.alpha = value
def __call__(self) :
return alpha
Like Soya3D, Pyrex's simplicity and elegance lures you in and so long as you don't try to, say, build a multithreaded game engine with it, it works great. Please don't misunderstand me, Pyrex is great for most applications that use it, we just out-grew it years ago and have wasted far too much time trying to make it work.
GObjects provides an even more elegant solution, write the C code in C, and write the interface code (import soy) in Python using PyGObject. The C code never includes Python.h and thus can never have any issues with the GIL.
So here begins our 5th revision of PySoy; function by function, type by type, extension by extension, refactored into C as GObjects. It'll take a bit to shift the code over in pieces without breaking everything as we do, but when this is done we can replace all the Pyrex source files with a Python package.
No custom languages for developers to learn, no more "with nogil:" everywhere, or having to refactor every C header we want to use to a .pxd file, or having to search through the C sources Pyrex generates for bugs.
so long, Pyrex! You made PySoy possible, but also caused us all countless hours of extra work and frustration. In retrospect, I wish we never met you.
Wednesday, March 18, 2009
sqlite and postgres support in Python 3 - today
I've been working with the SQLAlchemy package in Python 3, their pre-0.6 branch anyway.
Currently it offers sqlite (included with Py3) and postgres (via py8000) with their Object Relational Mapper and lower-level SQL Expression Language.
My needs are covered given that sqlite and postgres are the only two databases I find myself normally using. Of course there are those who will balk, "but it doesn't support {'MySQL', 'Oracle', 'MSSQL'} yet!". To the people who need those databases, it's easier to spend a few days helping to port the appropriate dbapi package than wasting energy with complaints.
For web frameworks, where the multi-db abstraction SQLAlchemy provides is vital, it's available to develop on today (svn checkout, python3 sa2to3.py). They'll provide a separate -py3 tarball with the 0.6 release planned for after PyCon.
Currently it offers sqlite (included with Py3) and postgres (via py8000) with their Object Relational Mapper and lower-level SQL Expression Language.
My needs are covered given that sqlite and postgres are the only two databases I find myself normally using. Of course there are those who will balk, "but it doesn't support {'MySQL', 'Oracle', 'MSSQL'} yet!". To the people who need those databases, it's easier to spend a few days helping to port the appropriate dbapi package than wasting energy with complaints.
For web frameworks, where the multi-db abstraction SQLAlchemy provides is vital, it's available to develop on today (svn checkout, python3 sa2to3.py). They'll provide a separate -py3 tarball with the 0.6 release planned for after PyCon.
Sunday, March 08, 2009
git vs mercurial
In the past we've stuck with subversion for PySoy over git because about half our team are Windows devs and git has very poor Windows support. Some of us have been using git-svn to use git locally while not making life difficult for the other half.
Last night I migrated the Concordance-XMPP project to mercurial as a trial.
First, I have to say that Mercurial's subversion migration tools need work - simple things like author lookup tables (svn username -> Full Name <email>), option to turn off auto-tagging, and a template for modifying commit messages so we can turn off marking the history up with "[rXXX]".
These things were easy enough given that Mercurial is Python-based. Two hours later I had a proper migration.
In contrast with my earlier attempts at migrating to git, mercurial was actually a joy to setup on the server. WSGI, using our existing .htusers file from Trac, a small config script, done. This contrast to git which would require some funky SSH sandboxing and/or PAM setup - yes it can be done, but such hacks shouldn't be needed.
To all our surprise "hg clone" was extremely fast even compared to subversion and especially in comparison to git. One development commented that it seemed "instant". Even for the small commit history (2 months, about 130 commits) git could take several minutes to clone and process Concordance's history.
I'm disappointed to see GPLv2-only on Mercurial as we won't be able to use it directly in any GPLv3 or AGPLv3 projects. Upward license compatability good, anti-FSF paranoia bad.
Thankfully hg's protocol is easily derived and can be implemented by other Python software without having to actually import any part of hg. Hopefully the Mercurial's license situation will be solved at some point to minimize redundant work.
Last night I migrated the Concordance-XMPP project to mercurial as a trial.
First, I have to say that Mercurial's subversion migration tools need work - simple things like author lookup tables (svn username -> Full Name <email>), option to turn off auto-tagging, and a template for modifying commit messages so we can turn off marking the history up with "[rXXX]".
These things were easy enough given that Mercurial is Python-based. Two hours later I had a proper migration.
In contrast with my earlier attempts at migrating to git, mercurial was actually a joy to setup on the server. WSGI, using our existing .htusers file from Trac, a small config script, done. This contrast to git which would require some funky SSH sandboxing and/or PAM setup - yes it can be done, but such hacks shouldn't be needed.
To all our surprise "hg clone" was extremely fast even compared to subversion and especially in comparison to git. One development commented that it seemed "instant". Even for the small commit history (2 months, about 130 commits) git could take several minutes to clone and process Concordance's history.
I'm disappointed to see GPLv2-only on Mercurial as we won't be able to use it directly in any GPLv3 or AGPLv3 projects. Upward license compatability good, anti-FSF paranoia bad.
Thankfully hg's protocol is easily derived and can be implemented by other Python software without having to actually import any part of hg. Hopefully the Mercurial's license situation will be solved at some point to minimize redundant work.
Friday, January 23, 2009
Avahi to be considered harmful
Let me paint a picture as it's come into focus for me recently. Apple develops a protocol called mDNS for zeroconf via multicast DNS. They call it Bonjour and are serious enough about making it a standard they release a free software library called mDNSResponder to support it which runs on GNU/Linux, OSX, and Windows.
A community project, Avahi, implements their own mDNS suite for GNU/Linux. Avahi adds some nice features, such as dbus support. To help apps use either option they support compatibility with mDNSResponder's API.
It's great to have this sort of cooperative-competition, but they took this a bit too far by printing a nag warning when their mDNSResponder compatibility API is used by an application and otherwise pressuring mDNS-using apps to drop support for mDNSResponder.
I sincerely hope Python projects aiming to implement mDNS build against the mDNSResponder API, for cross platform support if nothing else, and encourage their users to avoid using Avahi when questions about this nag page come up.
A community project, Avahi, implements their own mDNS suite for GNU/Linux. Avahi adds some nice features, such as dbus support. To help apps use either option they support compatibility with mDNSResponder's API.
It's great to have this sort of cooperative-competition, but they took this a bit too far by printing a nag warning when their mDNSResponder compatibility API is used by an application and otherwise pressuring mDNS-using apps to drop support for mDNSResponder.
I sincerely hope Python projects aiming to implement mDNS build against the mDNSResponder API, for cross platform support if nothing else, and encourage their users to avoid using Avahi when questions about this nag page come up.
Saturday, January 10, 2009
Concordance register.py
I have the main Python callback in Concordance working now to handle <iq/>, <message/>, and <presence/> elements. Eventually it'll only handle these elements when they're directed to the server, otherwise the server will route the message to it's destination without Python needing to process it first.examples/register.py demonstrates user registration handling using ElementTree (or at least the start of the process for now). The result is shown here.
Wednesday, January 07, 2009
Concordance lives
Concordance-XMPP now listens and responds to new XMPP connections with basic XML parsing via expat.
David has been working with me to maintain Windows compatability. I'm working on SASL support and the Python callback API.
With luck the first (very early) release will be ready this weekend which can handle logging in and intra-server routing between users.
David has been working with me to maintain Windows compatability. I'm working on SASL support and the Python callback API.
With luck the first (very early) release will be ready this weekend which can handle logging in and intra-server routing between users.
Friday, January 02, 2009
Python 3.0 - first extension module
After hours of hunting down docs and debugging my build environment (turns out I had a 3.0-alpha5 lurking around mucking up setup.py)..
Python 3.0 (r30:67503, Jan 2 2009, 00:20:15)
[GCC 4.3.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import concordance
>>> concordance.__file__
'/usr/lib/python3.0/site-packages/concordance.so'
>>>
Thursday, January 01, 2009
New Year, New Project
The first day of 2009, PySoy's copyright notices updated and a few tickets closed, I hit a point where I really need a decent XMPP server.
I've started a new project called Concordance, a XMPP/Jingle server written in C (message routing and RTP mixing) and Python 3.0 (authentication, login, MUC handling, etc).
The goal is a framework giving Python full control over the server, for nearly unlimited configurability, while having the speed to serve a large user base with a minimal of CPU use.
I've started a new project called Concordance, a XMPP/Jingle server written in C (message routing and RTP mixing) and Python 3.0 (authentication, login, MUC handling, etc).
The goal is a framework giving Python full control over the server, for nearly unlimited configurability, while having the speed to serve a large user base with a minimal of CPU use.
Tuesday, December 30, 2008
great cheese frustration
I overall like Gnome, but the guys working on Cheese appear to be making some horrible design choices.
Such as, they've removed any possible way to set the video input device, either via gstreamer's properties, or command line, or within the cheese app itself.
Now I'm hunting for another app that does video capture from a video4linux device, or possibly writing one using PySoy.
Such as, they've removed any possible way to set the video input device, either via gstreamer's properties, or command line, or within the cheese app itself.
Now I'm hunting for another app that does video capture from a video4linux device, or possibly writing one using PySoy.
Saturday, September 13, 2008
Beta3, new features on Ohloh
So we're trying to get Beta3 out the door. No date yet, there's still a lot of cleanup work to do.
For smaller updates, Ohloh has a new "Journal" feature that resembles Twitter. What's really cool is once you associate your XMPP id with your Ohloh account you can use XMPP (via your normal IM client) to post to your journal.
Check PySoy's journal for smaller, more frequent updates on what we're doing.
For smaller updates, Ohloh has a new "Journal" feature that resembles Twitter. What's really cool is once you associate your XMPP id with your Ohloh account you can use XMPP (via your normal IM client) to post to your journal.
Check PySoy's journal for smaller, more frequent updates on what we're doing.
Thursday, May 22, 2008
bike accident
I got side-swiped by a car Tuesday. Only injuries are some scraped skin and sprained wrist (I'm either really lucky or all too experienced in surviving bike accidents), but the latter has dramatically slowed by dev pace due to having to type with only one hand.
The sprain is mending well and i should be able to do light stuff (like typing) with it this weekend, provided I keep it immobile.
In other news, bump mapping is working again, but our
The sprain is mending well and i should be able to do light stuff (like typing) with it this weekend, provided I keep it immobile.
In other news, bump mapping is working again, but our
blocks.py lacks proper normals, texcoords, and tangents. If anyone is willing to spend an hour or two working out the calcs for these please get in touch.
Tuesday, May 20, 2008
hacking at 3am, seeing rainbows
There are some real advantages to hacking into the wee hours of the morning, this gem of a screenshot is one of the best of late.
Thing is, I'm sitting here after refactoring several hundred lines of code and fixing two typos that was causing our normalisation cubemaps to be bugged, staring at this window, and unable to understand why.
The rainbow effect is obviously from the normalisation cubemap, a data construct that maps tangent-space bumpmaps to object-space on the video card (without shaders). I just can't determine at this late hour why it's showing up instead of rendering the bumpmap as it used to.
I'm sure it'll make much more sense after 8 hours sleep. In the meantime, enjoy the screenshot, or compile PySoy from subversion and run
TexBlocks.py yourself!
Saturday, May 17, 2008
Mesh code refactoring
I spent most of the day fixing, shifting, folding, documenting, and otherwise refactoring soy.models.Mesh and it's related datatypes and atoms. I'll likely spend most of tomorrow finishing this up.
In short, the render process we used was crude, lacked some really easy features, looped through 6 different classes for each Mesh (Mesh, MaterialList, Material, Texture, FaceList, VertexList) and had data management code in these plus the two related atom classes (Face and Vertex).
In short, not only was the code difficult to grok (even for me and I wrote much of it) but was slower that need be and increasingly difficult to expand. Game engines need both rendering speed (fps) and easy expandability, so this is a high priority ticket.
The new structure moves all of the actual Mesh data into the Mesh objects themselves; MaterialList is gone (to be replaced by MaterialDict later), FaceList and VertexList will now only store a reference to their parent Mesh to draw data from.
I'm going to be centralizing data management functions in Mesh as well, so the atoms can be much simpler and we can actually talk about optimizing the VBO layout. We'll also be doing dynamic passes, based on how many textures/pass a GL implementation supports, and supporting more textures targets (ie, specularity, emissive)
Again, sprint via IRC this weekend complete with remote peer programming via Gobby. Join us in #PySoy on irc.freenode.net
In short, the render process we used was crude, lacked some really easy features, looped through 6 different classes for each Mesh (Mesh, MaterialList, Material, Texture, FaceList, VertexList) and had data management code in these plus the two related atom classes (Face and Vertex).
In short, not only was the code difficult to grok (even for me and I wrote much of it) but was slower that need be and increasingly difficult to expand. Game engines need both rendering speed (fps) and easy expandability, so this is a high priority ticket.
The new structure moves all of the actual Mesh data into the Mesh objects themselves; MaterialList is gone (to be replaced by MaterialDict later), FaceList and VertexList will now only store a reference to their parent Mesh to draw data from.
I'm going to be centralizing data management functions in Mesh as well, so the atoms can be much simpler and we can actually talk about optimizing the VBO layout. We'll also be doing dynamic passes, based on how many textures/pass a GL implementation supports, and supporting more textures targets (ie, specularity, emissive)
Again, sprint via IRC this weekend complete with remote peer programming via Gobby. Join us in #PySoy on irc.freenode.net
why can't XMPP keep their website up?
Several times this month I've found www.xmpp.org/, the organization responsible for the Jabber protocol and it's many extensions, non-responsive. Their site has been down for at least 7 hours tonight, the longest yet (to my knowledge).
For our purposes I'll be mirroring their XEP's on pysoy.org as soon as they're back up. It's seriously frustrating to be working on this when the documents you need are often unavailable.
If anyone knows of an existing mirror we could use, please reply to this posting.
For our purposes I'll be mirroring their XEP's on pysoy.org as soon as they're back up. It's seriously frustrating to be working on this when the documents you need are often unavailable.
If anyone knows of an existing mirror we could use, please reply to this posting.
Friday, May 16, 2008
some real progress toward beta-3
With Pyrex 0.9.8 released yesterday, including a number of key changes we needed to move forward, we're back in action for PySoy belated Beta-3 release.
The new forward declarations are a bit kludgy (example), and I don't think it'll be very maintainable in the long term. I think changing a class's inheritance should require editing one file, not two (the .pxd) or more (every .pxd that uses it).
It works for now, until we can wipe out the entire
The nogil allowance for cdef methods is a massive step toward stomping out the multicore race conditions. One of the developers testing just the fixes to
One of the roadblocks I hit is the nogil fix can't be used with our
While our usage is GIL-safe, the current version of Pyrex still wants to throw in an
In the meantime, I've called for an IRC sprint this weekend for the pending cleanup, refactoring, documentation, and testing that needs doing before beta-3. Join
The new forward declarations are a bit kludgy (example), and I don't think it'll be very maintainable in the long term. I think changing a class's inheritance should require editing one file, not two (the .pxd) or more (every .pxd that uses it).
It works for now, until we can wipe out the entire
include/ directory, and to be fair it's only so cumbersome due to PySoy's size and how the classes are interconnected.The nogil allowance for cdef methods is a massive step toward stomping out the multicore race conditions. One of the developers testing just the fixes to
soy.bodies noticed the examples often running longer than before, though there's a lot more cleanup to be done before this is package-wide.One of the roadblocks I hit is the nogil fix can't be used with our
Children class, which stores pointers to Python objects of the same type and uses those pointers (through a typecast) to access their cdef methods in a for loop.While our usage is GIL-safe, the current version of Pyrex still wants to throw in an
INCREF just to be extra-cautious, which makes it not. I've let Greg know about this, we'll see if it can't be fixed soon.In the meantime, I've called for an IRC sprint this weekend for the pending cleanup, refactoring, documentation, and testing that needs doing before beta-3. Join
#PySoy on irc.freenode.net if you want to pitch-in!
Tuesday, May 13, 2008
Building a better Pyrex - part 1
With our recent decision to drop the Pyrex codebase and start from scratch, we find ourselves repeating a process from 2 years ago;
When PySoy first forked off of Soya, we had a small list of things that needed to be fixed based on our negative experiences working with that project. The API needed to be cleaned up, pydocs added, Shape needed to be mutable and implemented with vertex arrays or VBOs rather than display lists, etc.
As we worked on it, that list grew longer, including rewriting large parts of the codebase, until we found ourselves at a point where starting over would be more expedient. Of course now there's little doubt that was a good move, learning from both Soya's mistakes and our own failed early attempts we now have a pretty rocking architecture.
The first and most foundational change we're making from Pyrex, as previously stated, is in the lexical scanner and parser. Both Pyrex and Cython are based on Plex, which was Greg's answer to processing a Python-like language.
In contrast, we'll be using Python's own
This way we have less code to debug and a solid foundation to start with. With lexing and parsing almost "for free" we'll be able to focus on the important bits and get the platform able to support the kind of features we need quickly.
We've also reached a rough consensus on the following syntax changes from Pyrex:
There's more to come, which I'll post in additional installments as work progresses on this project.
When PySoy first forked off of Soya, we had a small list of things that needed to be fixed based on our negative experiences working with that project. The API needed to be cleaned up, pydocs added, Shape needed to be mutable and implemented with vertex arrays or VBOs rather than display lists, etc.
As we worked on it, that list grew longer, including rewriting large parts of the codebase, until we found ourselves at a point where starting over would be more expedient. Of course now there's little doubt that was a good move, learning from both Soya's mistakes and our own failed early attempts we now have a pretty rocking architecture.
The first and most foundational change we're making from Pyrex, as previously stated, is in the lexical scanner and parser. Both Pyrex and Cython are based on Plex, which was Greg's answer to processing a Python-like language.
In contrast, we'll be using Python's own
tokenize module for our lexical scanner and an ASDL-based parser akin to the parser used in Python 3.0. We're extending Python.asdl with cdef, ctypedef, cimport, cinclude, etc.This way we have less code to debug and a solid foundation to start with. With lexing and parsing almost "for free" we'll be able to focus on the important bits and get the platform able to support the kind of features we need quickly.
We've also reached a rough consensus on the following syntax changes from Pyrex:
- unicode, generators, decorators, etc will be supported
- all custom
forsyntaxes dropped in favor of Python's standard with nogil:replaced with fullwithsupport- special methods will always have
selfas first argument (ie, both __sub__ and __rsub__ used) cincludeadded for direct C header parsing
There's more to come, which I'll post in additional installments as work progresses on this project.
setup-pyrex.py works with 0.9.7.2
Greg fixed the latest bug that caused PySoy to raise a dict assignment error on import with 0.9.7.
Pyrex 0.9.7.2 was released a few hours ago with this fix, and I've updated
Beta-3 is still blocked by the multicore race condition caused by redundant incref/decref calls on Python arguments to class methods in this latest version of Pyrex.
Pyrex 0.9.7.2 was released a few hours ago with this fix, and I've updated
setup-pyrex.py to work with either 0.9.6.4 or >=0.9.7.2.Beta-3 is still blocked by the multicore race condition caused by redundant incref/decref calls on Python arguments to class methods in this latest version of Pyrex.
Monday, May 12, 2008
setup-pyrex.py now requires EXACTLY Pyrex 0.9.6.4
16:02 < maacl> import soy
16:02 < maacl> File "LoopThread.pym", line 51, in
soy._internals.LoopThread.__cinit__
16:02 < maacl> TypeError: 'dict' object does not support item assignment
16:03 < Arc> from threading import _active, _active_limbo_lock
16:03 < Arc> self._active = _active
16:03 < Arc> [...] and line 51:
16:03 < Arc> self._active[self._threadID] = self
I'm not going to even look into this further. Pyrex 0.9.7, for a reason I'm not going to even contemplate, raises an error about dict item assignment on the last line. None of us use it due to it's barfing of depreciated lines for all our
for statements, but this brave tested did and reported the problem today.I just committed an update to
setup-pyrex.py that fails when any Pyrex version besides 0.9.6.4 is used.
Sunday, May 11, 2008
subversion vs git
I stayed up late last night working to install a shared git repository for the Pyrex replacement project. First of all, copious thanks to
I'm very much an enthusiast of Trac, it's an excellent engine for project websites and has all the essential tools, builtin and via plugins, that any project should need. Ok, so there's a few things a project could need that isn't already written, but new plugins are easy enough (it's all Python, after all).
As such, I like to leave user administration to Trac, and it maintains a .htusers and .access file local to each project for that purpose. Likewise, for SVN, we use WebDAV on Apache, whereas the ssh+svn method would require adding those users to the local system or PAM hacking.
Thus, the most direct path would seem to install GIT via WebDAV and run it from there. As
The other available solutions could work, but first I'd want to write some new Trac plugins for accepting developer ssh keys and using these on the backend to govern access or figure out the PAM configuration for .htusers to have restricted SSH access for GIT pushes.
There are certainly advantages to GIT, but server setup difficulty and added complications for Windows developers leads me to sticking with subversion servers with
johnw and Ilari on #git for their help on this.I'm very much an enthusiast of Trac, it's an excellent engine for project websites and has all the essential tools, builtin and via plugins, that any project should need. Ok, so there's a few things a project could need that isn't already written, but new plugins are easy enough (it's all Python, after all).
As such, I like to leave user administration to Trac, and it maintains a .htusers and .access file local to each project for that purpose. Likewise, for SVN, we use WebDAV on Apache, whereas the ssh+svn method would require adding those users to the local system or PAM hacking.
Thus, the most direct path would seem to install GIT via WebDAV and run it from there. As
Ilari pointed out, GIT's DAV has many problems over SVN, from lockups to a failed push sometimes corrupting the shared repository. I tried anyway, with hours of help from joshw, and at 5am I had to throw in the towel. I'm sure it's possible, but not with my current lack of GIT knowledge.The other available solutions could work, but first I'd want to write some new Trac plugins for accepting developer ssh keys and using these on the backend to govern access or figure out the PAM configuration for .htusers to have restricted SSH access for GIT pushes.
There are certainly advantages to GIT, but server setup difficulty and added complications for Windows developers leads me to sticking with subversion servers with
git-svn client-side for now.
Saturday, May 10, 2008
a radical redirection
We've been at a deadlock for a few weeks now, our previous coding pace has turned into a lot of contemplation and side discussions about the greater issues with the PySoy codebase.
I think we have a roadmap forward, or enough of one to post these in a blog entry:
First, PySoy 1.0 will not be released until after Python 3.0 this Fall, and it would be pointless to continue developing with 2.x in target. Python 3.0 also has a large number of useful changes for our project. Our target platform should therefore be 3.0 (and it's alpha/beta's for now).
Second, with the release of Pyrex 0.9.7 Greg has, once again, introduced a major language change (the "for" statement) without even soliciting input from the community. This has sealed the deal for us, even if our immediate issues are fixed we cannot continue to subject ourselves to a language that changes in incompatible ways between 2 micro versions (0.9.6 -> 0.9.8, when the old syntax will not be supported).
We earlier started a Cython variant to support PySoy's codebase, but given recent developments in their community, the timelines of our projects, and their choice to stick with Python 2.x for now, we're moving forward on a Pyrex replacement written from scratch and targeting Python 3.0.
We're going to use a similar language style to Pyrex, Greg certainly had some good ideas, and will keep language porting in mind, but there will be changes. We will also not be using any of his code including his Plex module, Pyrex and Cython's lexical analyzer which is the core of those packages.
PySoy Beta3, earlier aimed for release in "early Spring", has been indefinably postponed as we work on that. The subversion repository is of course open, and development can and will continue, but we cannot release until the new build system is complete.
I think we have a roadmap forward, or enough of one to post these in a blog entry:
First, PySoy 1.0 will not be released until after Python 3.0 this Fall, and it would be pointless to continue developing with 2.x in target. Python 3.0 also has a large number of useful changes for our project. Our target platform should therefore be 3.0 (and it's alpha/beta's for now).
Second, with the release of Pyrex 0.9.7 Greg has, once again, introduced a major language change (the "for" statement) without even soliciting input from the community. This has sealed the deal for us, even if our immediate issues are fixed we cannot continue to subject ourselves to a language that changes in incompatible ways between 2 micro versions (0.9.6 -> 0.9.8, when the old syntax will not be supported).
We earlier started a Cython variant to support PySoy's codebase, but given recent developments in their community, the timelines of our projects, and their choice to stick with Python 2.x for now, we're moving forward on a Pyrex replacement written from scratch and targeting Python 3.0.
We're going to use a similar language style to Pyrex, Greg certainly had some good ideas, and will keep language porting in mind, but there will be changes. We will also not be using any of his code including his Plex module, Pyrex and Cython's lexical analyzer which is the core of those packages.
PySoy Beta3, earlier aimed for release in "early Spring", has been indefinably postponed as we work on that. The subversion repository is of course open, and development can and will continue, but we cannot release until the new build system is complete.
Friday, May 02, 2008
initial impressions of Trac 0.11
Trac 0.11rc1 was released this week, marking a feature freeze for 0.11.
One of the major improvements over 0.10.4 is the Genshi template system allowing complete control over a site's HTML. For example, you can copy the default
Of course, this is also means a Trac site can be radically redesigned in structure without editing the source code. Hopefully this leads to a greater diversity in project site design.
Another major feature is ticket workflows are now customizable, allowing projects using Trac to decide how their tickets will be processed. For example, a "testing" state could be added to signify a ticket as "though to be complete, but waiting for verification".
I plan on upgrading pysoy.org this weekend along with a few new Trac plugins and a minor site redesign.
One of the major improvements over 0.10.4 is the Genshi template system allowing complete control over a site's HTML. For example, you can copy the default
layout.html to your site's templates/ directory to remove the obnoxious "Trac" string at the end of every page's title and the other Edgewall advertising splayed all over a default Trac site.Of course, this is also means a Trac site can be radically redesigned in structure without editing the source code. Hopefully this leads to a greater diversity in project site design.
Another major feature is ticket workflows are now customizable, allowing projects using Trac to decide how their tickets will be processed. For example, a "testing" state could be added to signify a ticket as "though to be complete, but waiting for verification".
I plan on upgrading pysoy.org this weekend along with a few new Trac plugins and a minor site redesign.
Wednesday, April 02, 2008
some race conditions found
Well after many weeks we found the source of the race conditions on multicore CPUs; they were not in PySoy at all but in the Pyrex compiler.
Throughout the generated .c we found INCREF/DECREF where it didn't belong, namely in every cdef method for __pyx_v_self. In other words, even though the calling function must already hold a reference to instance in order to access it's C methods, thus preventing the instance from being garbage collected during method execution, the Pyrex.Compiler inserts redundant INCREF/DECREF calls.
These would normally just waste CPU cycles, but since our 3d engine has non-GIL background threads these refcount adjusting calls are accessed in a thread-unsafe manner and will inevitably cause either a segfault or a memory leak.
The solution is to not consider "self" a normal Python argument, which would also allow these C methods to be called with nogil: set, provided checking is done to ensure no Python methods, properties, or variables of self are accessed.
We'll be fixing this problem with our new source compiler, to be announced soon.
Throughout the generated .c we found INCREF/DECREF where it didn't belong, namely in every cdef method for __pyx_v_self. In other words, even though the calling function must already hold a reference to instance in order to access it's C methods, thus preventing the instance from being garbage collected during method execution, the Pyrex.Compiler inserts redundant INCREF/DECREF calls.
These would normally just waste CPU cycles, but since our 3d engine has non-GIL background threads these refcount adjusting calls are accessed in a thread-unsafe manner and will inevitably cause either a segfault or a memory leak.
The solution is to not consider "self" a normal Python argument, which would also allow these C methods to be called with nogil: set, provided checking is done to ensure no Python methods, properties, or variables of self are accessed.
We'll be fixing this problem with our new source compiler, to be announced soon.
Friday, March 21, 2008
1.0_beta3 release and a Sprint this weekend
There is a rumor going around that because our 1.0_beta3 milestone is marked as due today (3/21) we're planning a release today. This is false, we'll be releasing sometime in early spring as advertised - the due dates marked for our milestones are our preferred date only.
We're still tracking down one or more nasty race conditions which only show up on multicore systems, we still have a bit more work to do before the next release, and a good deal more testing is needed after these bugs are found to demonstrate stability. If you're experience with debugging we could use some help - some of our tests seem to indicate mutex locks being ignored.
This weekend is a good time, too. Starting today (Friday) we're running a pre-Beta3 online sprint. In our IRC channel (#PySoy on irc.freenode.net) you can find a good number of developers working almost around the clock on various parts of the 3D engine and a few working in Gobby for remote pair programming.
If you're not into debugging, or don't have a multicore system, there are many other tasks - large and small - that need doing.
We're still tracking down one or more nasty race conditions which only show up on multicore systems, we still have a bit more work to do before the next release, and a good deal more testing is needed after these bugs are found to demonstrate stability. If you're experience with debugging we could use some help - some of our tests seem to indicate mutex locks being ignored.
This weekend is a good time, too. Starting today (Friday) we're running a pre-Beta3 online sprint. In our IRC channel (#PySoy on irc.freenode.net) you can find a good number of developers working almost around the clock on various parts of the 3D engine and a few working in Gobby for remote pair programming.
If you're not into debugging, or don't have a multicore system, there are many other tasks - large and small - that need doing.
Wednesday, February 27, 2008
State of the Soy @ r1000
Last night our humble 3D game engine reached 1000 commits to it's Subversion repository.
PySoy has been contributed to by 20 developers, 7 of which are currently active. The project's IRC channel is in nearly constant use by developers working on different tasks.
The external API has drawn considerable praise by users, despite it's state as beta, and the constructive feedback we've received has led many of the post-Beta2 API changes.
The internal API, while having some rough edges, has been commented on by several new developers as remarkably clean. We've passed the point which the COMPLETED list exceeds the TODO list, despite the latter constantly growing, and focus has largely shifted toward debugging, documentation, and polish.
Most notably, we've reached the point of needing a legal entity for the project. Using the Software Freedom Conservancy as a rough template, we've started the process to form a non-profit corporation with a focus on copyleft game projects. An announcement about this will be posted as more details are firmed up.
PySoy has been contributed to by 20 developers, 7 of which are currently active. The project's IRC channel is in nearly constant use by developers working on different tasks.
The external API has drawn considerable praise by users, despite it's state as beta, and the constructive feedback we've received has led many of the post-Beta2 API changes.
The internal API, while having some rough edges, has been commented on by several new developers as remarkably clean. We've passed the point which the COMPLETED list exceeds the TODO list, despite the latter constantly growing, and focus has largely shifted toward debugging, documentation, and polish.
Most notably, we've reached the point of needing a legal entity for the project. Using the Software Freedom Conservancy as a rough template, we've started the process to form a non-profit corporation with a focus on copyleft game projects. An announcement about this will be posted as more details are firmed up.
Tuesday, February 26, 2008
PySoy goes from 3 threads to 5
We're doing the final work now to replace CoreLoop with TransportLoop, WindowLoop, and SceneLoop. All three are instances of soy._internals.LoopThread which (should) play well with Python's threading system and should never hold the GIL.
The increased performance this has brought for multi-core users is amazing, some have reported more than twice the framerate!
To be fair, an equal number of subversion Trunk users are experiencing frequent segfaults since we've not finished adding mutexes and rearranging the internal API post-move. Obviously these issues are being resolved prior to the Beta-3 release next month.
The increased performance this has brought for multi-core users is amazing, some have reported more than twice the framerate!
To be fair, an equal number of subversion Trunk users are experiencing frequent segfaults since we've not finished adding mutexes and rearranging the internal API post-move. Obviously these issues are being resolved prior to the Beta-3 release next month.
Subscribe to:
Posts (Atom)
