Saturday, December 16, 2006

PySoy, a Historical Perspective

With the reorganization discussion underway and so many people joining over the last few weeks I'd like to present a historical perspective of PySoy and where I see things going.

If you're reading Planet PySoy you can access the full text here. It's far too lengthy for syndication.

So it all starts with a quirky vegetarian college kid in France using the moniker "Jiba".

Jiba likes to design video games. He isn't good at math or programming and wants a way to build 3d games without having to be good at either.

At some point, perhaps in the late 90's, he knows Visual Basic so he writes a 3d engine (in VB) called Soya. Then he learns Java and writes a new engine called Opale.Soya (Soya 0.2?) for Java with his brother, then in 2003 he and his brother rewrote it again for Python calling it Soya 0.3.

This trend continues, Soya is redesigned and rewritten (in various mixes of C, Python, and PyRex) several times with it's API radically changing in each version. Jiba gets a lot of help from his older brother and friends at school with various parts and Jiba releases a few games that use it.

The story would likely end there if Soya didn't attract interest from developers from other parts of Europe and the US. In 2004-2005 one such developer from the UK, going by the moniker "Dunk" or d1223m, joined the Soya group by adding a new widget system called Pudding. During the same time I (Arc Riley) got involved with Soya as I was using it for my GNU project Aetherspace and to teach 3d programming at a community center in upstate NY.

I organized Team Soya during the first PyWeek competition which created a lot of new energy in the Soya project (including Soya's 10.1 release). Shortly after the competition Dunk and Jiba had a falling out after a number of disagreements where Dunk suggested that Soya was designed so haphazardly and so flawed that it needed to be rewritten from scratch.

He took several of the newer developers with him, including all those Soya picked up during the PyWeek competition, and started a new 3d engine project (which fizzled out after a few weeks). His earlier work toward this effort was called Soya 2 or "Evil Soya".

I and a few of the developers who stayed with Soya worked out an agreement with Jiba to transform Soya into a community project to allow Soya to turn into something awesome. This agreement was in exchange for not joining Dunk's project, something Jiba fought to prevent.

Fast forward about 8 months.

We saw no new Soya releases, little contributed code was accepted, and we barely saw or heard from Jiba. It was seeming like the only way for things to move forward was for the rest of us to take a leadership role and make them happen. I registered as a Trac site (wiki, subversion, tickets, etc) and many of us started using it.

I organized Soya's participation in the Google Summer of Code and, once again, Soya gets a flood of interest from developers (including Jiba). This flood brought a lot of new ideas, many of which Jiba disagreed with, and the "community leadership" agreement we made a year prior turned to dust as Jiba struggled to maintain control against what he perceived as a hijacking of his project.

Emotions were really strong over this. On IRC one night, after an argument, Jiba suggested (perhaps not seriously) we fork Soya. By the time he returned to IRC the next day the PySoy domain ( was registered and the fork was officially announced.

After a heated few days the 2 Summer of Code students were divided as per their mentors; Jiba's local friend Pierre-Yves (marmoute) stayed with Soya and Buddha's (blaisepascal) student Palle (praabjerg) with PySoy.

This unfortunately required employees of Google to sort out as Jiba was demanding both students and organized many of his friends to "rally" for him. Blog entries were written, #PySoy started getting flooded by friends of Jiba, and they petitioned support "in defense" of Soya from the larger Python community.

Issues spanned the range of our use of the word "Soy" being disambiguous with "Soya", who mentored the Summer of Code students, our retention of the domain for forwarding ticket URLs to, our copying of wiki content and documentation (Jiba claims it was copyright violation since he never freely licensed it), and accusations of malicious server infiltration and domain hijacking.

This became serious enough for several Freenode IRC admins (mainly Rob Levin) to get involved ending in most of France (entire netblocks) banned from #PySoy and moderated on our mailing lists. These policies have continued to date.

With the fork issue finally calming down we went to work. Tasks were divided, discussions were held, issues were sorted out. Several former Soya developers, hearing about the fork from all the noise Jiba's friends were kicking up, joined us.

Zach became the maintainer and I, after getting distracted with other projects, stayed on to manage the website and any business matters that came up.

In my perspective, Zach was a bit intimidated by the lack of structure. We run by democracy, great, but who votes? How? When? Where? Does this mean some guy just joining is counted equally as someone who's poured their heart and soul into the project for a year? Or does it mean only people we hand-pick get to vote?

The result of this, as I see it, was a move toward insularization. Democracy, yes, but only within a closed group with membership limited to trusted developers. IRC stopped happening, the website stopped being used, the mailing list went silent, subversion updates slowed, even blogs were only updated when I asked something be posted.

This wasn't what any of us wanted to see but it kept chaos at bay in lack of any other structure, and productive work was being done, so this is the way it's been for months now.

The team Zach organized reached the same assessment (unanimously?) that Dunk reached the previous year; it would be easier to write a new engine from scratch than fix Soya even after most of the API was cleaned up. Doing so would also clear up a sticky legal situation with Jiba holding a copyright on much of the code and being unwilling/unable to defend the GPL in court.

We like the general idea of Soya - Jiba was brilliant in some of his ideas on how to abstract OpenGL behavior to a high level language such as Python. It's implementation that he fell short on.

Building an engine from scratch was a much larger scope than we started with. On one hand every release goal was getting thrown away, it has taken months (and will take many more) to get a working release out. On the other hand, PySoy will be faster and easier to work with as a result.

Although Zach was in support of this he was obviously becoming overwhelmed. On top of being the chief architect and maintainer of PySoy he had people emailing him, wanting to get involved, and helping them get started. He apparently also had several new people making commitments, contributing some code, then disappearing before they finished.

In the last two weeks alone I've seen 8 versions of the phrase "it would have been faster/easier for me do it myself" in emails from Zach as entire source files contributed by absent developers got nixed or rewritten. I'd have to say he's contributed roughly 3/4 of the current codebase himself, much of that code a new developer promised to do and dropped the ball on.

Three days go I got an email from Zach that sounded like he was quitting as maintainer, two others CC'ed on that email thought the same thing. In response to my request for clarification he proposed what we're now calling the "Rank" system.

While I have reservations with some elements of the rank proposal (and some of my concerns have been addressed), overall, I think this is a really healthy direction to move in. It's certainly in the spirit that PySoy was created.

I think a bit of perspective is needed, however, as we discuss it.

Even once implemented it's not set in stone. If we implement it and it doesn't work, or stops working, we'll modify it or try something entirely different. The question is not "will this work for us forever". We don't need to craft a structure that will scale from a dozen developers to several hundred.

The question is, simply, "will this work for us right now".

The PySoy community has adapted, and will continue to adapt, to our current needs just as it has over the past 3+ years. All we need for now is a structure to get through the first release cycle. We'll adapt it as needed.

No comments: