Wednesday, December 20, 2006

another re-org update

I've collected comments/concerns from 9 people now and, after a discussion with Zach on these, I've published a revised PySoy Rank proposal

The new proposal eliminates Initiate/Intern votes to remove the "leads/directors can get a bunch of non-involved friends as initiates to swing a vote" loophole. It also changes the ratio between developers to make Leads more powerful and the Maintainer less powerful as follows:

  • contributor -> lead vote ratio from 150% to 200%

  • lead -> maintainer vote ratio from 166% to 150%

  • board -> maintainer vote ratio from 125% to 100%

Director term limits are now proposed for 1 year, previously directors could only voluntarily step down or removed through 2/3rd majority, thus giving contributing/lead developers more power to replace directors who do not represent them.

The powers granted to the Maintainer has also been reduced by delegating them to other Directors (which now have named seats and defined responsibility). Ie, the Maintainer cannot unilaterally spend money, handle code packaging, schedule monthly meetings, etc. In many ways the role of Maintainer has been changed to just being an inter-Lead mediator rather than being responsible for the entire codebase.

I think we're getting close to voting on this. Continue to send me comments, comment on the wiki, speak up on IRC, etc.

Monday, December 18, 2006

more on the re-org

I'm facilitating the PySoy re-org discussion as Zach has noted his non-neutrality..

In addition to comments from Zach and myself I've heard from 3 developers, two in Zach's current dev team and one who has not been able to contribute recently due to work not being on svn.

Planet PySoy readers click here for more.


In no order, their questions/comments and responses:

how does this address the issue of one person (maintainer) running everything? (answer from Zach, summarized)

The maintainer has only limited exclusive control and can be removed if he/she acts irresponsibly. As a member of the board, the maintainer is like a president. The maintainer's vote < (2 * board members | 2 * lead developers | 3 * contributing developers). Zach's idea is to make the maintainer's voice louder than others but not over-poweringly so.


who decides who will fill the ranks (or how will current developers be ranked)? (answered myself)

This will not change what role anyone is in, only formalize it. We'll send out an all-call to everyone who's signifigantly contributed to PySoy over the past 6 months to vote on the re-org proposal. If it passes those who voted (yay, nay, or obstain) will be included, those who did not will not be. Those currently interested in contributing, but have not really gotten involved yet, will be paired with a mentor once the re-org is complete.


why so many ranks (or why not make everyone equal)? (answer from Zach, verbatim)

I'm afraid that would be counter-productive to a stable, long-lasting pySoy. Ranks are a numerical representation of each developer's contribution level. It's a balence between allowing every developer to be involved in decisions, not having every minor decision have to go up for vote, and having the project remain stable with a single maintainer. It does not mean I view myself as better than others. :)


why can't an outside group just join to absorb us? (answered myself)

The mentorship period lets us screen people who are not really interested PySoy beyond it's name recognition


can't board members cheat votes by mentoring friends just before an election? (answered myself)

In theory, yes, but this is still better than the current situation where all developers have equal vote but Zach exclusivly controls who can be a developer


how can a new structure be implemented if/when this one stops working? (from myself, verbatim answer from Zach)

A `Vote for Change' can be made to replace the Rank structure.



Anyone have more questions or concerns? Throw them at me (Arc on Freenode or arc at pysoy) or publically to the pysoy-dev (@ pysoy) list. Don't be shy, you don't have to be a current developer to comment!

Saturday, December 16, 2006

PySoy, a Historical Perspective

With the reorganization discussion underway and so many people joining over the last few weeks I'd like to present a historical perspective of PySoy and where I see things going.

If you're reading Planet PySoy you can access the full text here. It's far too lengthy for syndication.




So it all starts with a quirky vegetarian college kid in France using the moniker "Jiba".

Jiba likes to design video games. He isn't good at math or programming and wants a way to build 3d games without having to be good at either.

At some point, perhaps in the late 90's, he knows Visual Basic so he writes a 3d engine (in VB) called Soya. Then he learns Java and writes a new engine called Opale.Soya (Soya 0.2?) for Java with his brother, then in 2003 he and his brother rewrote it again for Python calling it Soya 0.3.

This trend continues, Soya is redesigned and rewritten (in various mixes of C, Python, and PyRex) several times with it's API radically changing in each version. Jiba gets a lot of help from his older brother and friends at school with various parts and Jiba releases a few games that use it.

The story would likely end there if Soya didn't attract interest from developers from other parts of Europe and the US. In 2004-2005 one such developer from the UK, going by the moniker "Dunk" or d1223m, joined the Soya group by adding a new widget system called Pudding. During the same time I (Arc Riley) got involved with Soya as I was using it for my GNU project Aetherspace and to teach 3d programming at a community center in upstate NY.

I organized Team Soya during the first PyWeek competition which created a lot of new energy in the Soya project (including Soya's 10.1 release). Shortly after the competition Dunk and Jiba had a falling out after a number of disagreements where Dunk suggested that Soya was designed so haphazardly and so flawed that it needed to be rewritten from scratch.

He took several of the newer developers with him, including all those Soya picked up during the PyWeek competition, and started a new 3d engine project (which fizzled out after a few weeks). His earlier work toward this effort was called Soya 2 or "Evil Soya".

I and a few of the developers who stayed with Soya worked out an agreement with Jiba to transform Soya into a community project to allow Soya to turn into something awesome. This agreement was in exchange for not joining Dunk's project, something Jiba fought to prevent.

Fast forward about 8 months.

We saw no new Soya releases, little contributed code was accepted, and we barely saw or heard from Jiba. It was seeming like the only way for things to move forward was for the rest of us to take a leadership role and make them happen. I registered Soya3d.org as a Trac site (wiki, subversion, tickets, etc) and many of us started using it.

I organized Soya's participation in the Google Summer of Code and, once again, Soya gets a flood of interest from developers (including Jiba). This flood brought a lot of new ideas, many of which Jiba disagreed with, and the "community leadership" agreement we made a year prior turned to dust as Jiba struggled to maintain control against what he perceived as a hijacking of his project.

Emotions were really strong over this. On IRC one night, after an argument, Jiba suggested (perhaps not seriously) we fork Soya. By the time he returned to IRC the next day the PySoy domain (pysoy.org) was registered and the fork was officially announced.

After a heated few days the 2 Summer of Code students were divided as per their mentors; Jiba's local friend Pierre-Yves (marmoute) stayed with Soya and Buddha's (blaisepascal) student Palle (praabjerg) with PySoy.

This unfortunately required employees of Google to sort out as Jiba was demanding both students and organized many of his friends to "rally" for him. Blog entries were written, #PySoy started getting flooded by friends of Jiba, and they petitioned support "in defense" of Soya from the larger Python community.

Issues spanned the range of our use of the word "Soy" being disambiguous with "Soya", who mentored the Summer of Code students, our retention of the Soya3D.org domain for forwarding ticket URLs to pysoy.org, our copying of wiki content and documentation (Jiba claims it was copyright violation since he never freely licensed it), and accusations of malicious server infiltration and domain hijacking.

This became serious enough for several Freenode IRC admins (mainly Rob Levin) to get involved ending in most of France (entire netblocks) banned from #PySoy and moderated on our mailing lists. These policies have continued to date.

With the fork issue finally calming down we went to work. Tasks were divided, discussions were held, issues were sorted out. Several former Soya developers, hearing about the fork from all the noise Jiba's friends were kicking up, joined us.

Zach became the maintainer and I, after getting distracted with other projects, stayed on to manage the website and any business matters that came up.

In my perspective, Zach was a bit intimidated by the lack of structure. We run by democracy, great, but who votes? How? When? Where? Does this mean some guy just joining is counted equally as someone who's poured their heart and soul into the project for a year? Or does it mean only people we hand-pick get to vote?

The result of this, as I see it, was a move toward insularization. Democracy, yes, but only within a closed group with membership limited to trusted developers. IRC stopped happening, the website stopped being used, the mailing list went silent, subversion updates slowed, even blogs were only updated when I asked something be posted.

This wasn't what any of us wanted to see but it kept chaos at bay in lack of any other structure, and productive work was being done, so this is the way it's been for months now.

The team Zach organized reached the same assessment (unanimously?) that Dunk reached the previous year; it would be easier to write a new engine from scratch than fix Soya even after most of the API was cleaned up. Doing so would also clear up a sticky legal situation with Jiba holding a copyright on much of the code and being unwilling/unable to defend the GPL in court.

We like the general idea of Soya - Jiba was brilliant in some of his ideas on how to abstract OpenGL behavior to a high level language such as Python. It's implementation that he fell short on.

Building an engine from scratch was a much larger scope than we started with. On one hand every release goal was getting thrown away, it has taken months (and will take many more) to get a working release out. On the other hand, PySoy will be faster and easier to work with as a result.

Although Zach was in support of this he was obviously becoming overwhelmed. On top of being the chief architect and maintainer of PySoy he had people emailing him, wanting to get involved, and helping them get started. He apparently also had several new people making commitments, contributing some code, then disappearing before they finished.

In the last two weeks alone I've seen 8 versions of the phrase "it would have been faster/easier for me do it myself" in emails from Zach as entire source files contributed by absent developers got nixed or rewritten. I'd have to say he's contributed roughly 3/4 of the current codebase himself, much of that code a new developer promised to do and dropped the ball on.

Three days go I got an email from Zach that sounded like he was quitting as maintainer, two others CC'ed on that email thought the same thing. In response to my request for clarification he proposed what we're now calling the "Rank" system.





While I have reservations with some elements of the rank proposal (and some of my concerns have been addressed), overall, I think this is a really healthy direction to move in. It's certainly in the spirit that PySoy was created.

I think a bit of perspective is needed, however, as we discuss it.

Even once implemented it's not set in stone. If we implement it and it doesn't work, or stops working, we'll modify it or try something entirely different. The question is not "will this work for us forever". We don't need to craft a structure that will scale from a dozen developers to several hundred.

The question is, simply, "will this work for us right now".

The PySoy community has adapted, and will continue to adapt, to our current needs just as it has over the past 3+ years. All we need for now is a structure to get through the first release cycle. We'll adapt it as needed.

Wednesday, December 13, 2006

site fixed

A trac upgrade rendered much of the PySoy site unusable for a time, all fixed up now.

There's a fix to the pydoc plugin pending the new docs load. Everyone should know by now that the work Zach and the team are doing obsoletes what's currently there.

Let me know if there's any further issues with the site.

Tuesday, September 12, 2006

Reorganization

I've been talking to Zach about PySoy and we've reached a solution to continue from.

First, work will continue on subversion/trac as it has. Zach's been given admin access to the website and has accepted responsibility to update the ticket system, roadmap, and post design docs on the wiki.

Next, I've passed the role of "maintainer" to Zach. We'll continue to operate democratically for major design decisions and releases but someone has to be able to make trivial choices about checkins, branches, etc.

Finally, being as some of us just aren't cool enough for Jabber, we're going to setup a jabber-irc bridge for #PySoy and get all of his crazy Python 3D crew onto the -dev mailing list.

I'm remaining as part of the project but with the new business and related deadlines I'm unable to dedicate as much time to the PySoy project as I could before. Progeny Games will still be using Python & PySoy and will continue to fund specific PySoy enhancements.

Thursday, June 22, 2006

BSP-Tree vs Z-Buffer

The standard zbuffer which we're using now renders from front-to-back for pixels, such that if two polygons intersect it will show the intersection properly and no time is wasted drawing pixels that are blocked by other polygons.

The zbuffer has downsides - beside slowing down the GPU, it's not always accurate (especially if the range of vision is deep), it doesn't allow proper display of translucent polygons, and sees all polygons as a random "soup" with no object-level abstraction.

PySoy inherited intra-object BSP tree calculations from Soya allowing large objects (such as a whole level as one entity) to use BSP tree optimisation. This can easily be extended to all entities such that every object can have unseen polygons pruned and the faces rendered in the proper order.

If objects cannot intersect with a physics model which forbids it then objects can be sorted by a BSP-tree dynamically, then each object can be rendered individually by it's precomputed order. Further optimisations can be made to only recalculate the object-level BSP-tree when objects have moved sufficiently to justify it (with a dynamic accuracy based on CPU load).

Just some ideas to chew on, nothing that's going to get into 1.0...

1.0 focus

With the target a week away and still quite a bit away from being ready, I've started ripping up the API and trying to put together something more coherent. Soya has accumulated *SO* much extra baggage in it's namespace: copies of root level classes with only slight difference in function (ie, Shape, SolidShape, CellShadingMesh, etc), convience values such as WHITE, BLACK, and TRANSPARENT mixed into the root namespace, oh and let's not even touch the ODE issue yet.

The renaming process is also the opprotunity to clean the code up. Many of these copies remove redundant code, thus decreasing the amount that will need to be debugged later, and making things generally easier to read.

At the very least I'd like to see a beta released by the target date, though we should keep in mind that the API in 1.0 will haunt us for months (or even years) to come.

Tuesday, June 13, 2006

New Physics

Maybe I'm insane, but I've been mulling over some ideas to fully integrate ODE with PySoy, not for collision detection, but for rigid body physics.

This new model would let ODE handle velocity, rotation, etc for all Nodes and Entities directly in the Scene (Nodes treated as a group of joined entities) with the controller applying forces to various objects to affect them.

This new model, which includes Cal3D characters, would allow much more dynamic characters (adjusting walk cycles to terrain, able to trip over things, etc) than was feasable previously while allowing character speed to be set just with animation speed (ie, no more animation+movement syncing needed) since the animation of their feet would propel them forward.

I don't think this has ever been done before, certainly not in a free software game.

Wednesday, June 07, 2006

Forking Flurry

It seems I was the last one to see it, but it became nessesary for a Soya3d fork to be created in order to build it into a community project.

I've spent the last 28 hours sorting the website business out. Redirects from the various parts of the old soya3d.org now point to their respective elements at pysoy.org, some of which are in read-only states until everything is done. Having external links working has been my top priority.

On the PySoy site we have permissions issues in the DB being sorted through. I'm working through them as quickly as possible but it's going to take me a few more days to get everything working stable again. It's a large website, after all, with many plugins added.

I'm also working to get the mailing lists up. GNU Mailman has changed the way it wants to be installed (-D MAILMAN flag to apache?). I'm getting this done in my free time, between work for paid clients, so it'll be done as soon as it can.

Monday, June 05, 2006

.soy progress

The new 2-tier format is written out now, seperating object level from object-data level. This was pseudo-done in the former draft (blockid 0-127 == object level, 128+ object-data level) but this is far, far cleaner.

This allows objects to be quickly skipped-over on read/import, adds a much larger id-space both for types of objects and types of data within each object, and cleaner discard/recovery methods based on Palle's experience implementing the importer. We should have the blender importer/exporter up to something resembling this spec by tonight.

Check the updated .soy file format spec and give constructive criticism.