svn.pysoy.org reset

Our subversion server should be ready to go now. The old codebase (that was dumped several months ago) is gone, the blender plugin Palle wrote is copied over as is the media directory, and a /trunk/pysoy directory is awaiting the code Zach's team has been working on.

Oh, yea, we're posting to the CIA bot now. Hopefully we can get the bot in #PySoy soon.

another re-org update

I've collected comments/concerns from 9 people now and, after a discussion with Zach on these, I've published a revised PySoy Rank proposal

The new proposal eliminates Initiate/Intern votes to remove the "leads/directors can get a bunch of non-involved friends as initiates to swing a vote" loophole. It also changes the ratio between developers to make Leads more powerful and the Maintainer less powerful as follows:

• contributor -> lead vote ratio from 150% to 200%


• lead -> maintainer vote ratio from 166% to 150%


• board -> maintainer vote ratio from 125% to 100%

Director term limits are now proposed for 1 year, previously directors could only voluntarily step down or removed through 2/3rd majority, thus giving contributing/lead developers more power to replace directors who do not represent them.

The powers granted to the Maintainer has also been reduced by delegating them to other Directors (which now have named seats and defined responsibility). Ie, the Maintainer cannot unilaterally spend money, handle code packaging, schedule monthly meetings, etc. In many ways the role of Maintainer has been changed to just being an inter-Lead mediator rather than being responsible for the entire codebase.

I think we're getting close to voting on this. Continue to send me comments, comment on the wiki, speak up on IRC, etc.

more on the re-org

I'm facilitating the PySoy re-org discussion as Zach has noted his non-neutrality..

In addition to comments from Zach and myself I've heard from 3 developers, two in Zach's current dev team and one who has not been able to contribute recently due to work not being on svn.

Planet PySoy readers click here for more.


In no order, their questions/comments and responses:

how does this address the issue of one person (maintainer) running everything? (answer from Zach, summarized)

The maintainer has only limited exclusive control and can be removed if he/she acts irresponsibly. As a member of the board, the maintainer is like a president. The maintainer's vote < (2 * board members | 2 * lead developers | 3 * contributing developers). Zach's idea is to make the maintainer's voice louder than others but not over-poweringly so.

who decides who will fill the ranks (or how will current developers be ranked)? (answered myself)

This will not change what role anyone is in, only formalize it. We'll send out an all-call to everyone who's signifigantly contributed to PySoy over the past 6 months to vote on the re-org proposal. If it passes those who voted (yay, nay, or obstain) will be included, those who did not will not be. Those currently interested in contributing, but have not really gotten involved yet, will be paired with a mentor once the re-org is complete.

why so many ranks (or why not make everyone equal)? (answer from Zach, verbatim)

I'm afraid that would be counter-productive to a stable, long-lasting pySoy. Ranks are a numerical representation of each developer's contribution level. It's a balence between allowing every developer to be involved in decisions, not having every minor decision have to go up for vote, and having the project remain stable with a single maintainer. It does not mean I view myself as better than others. :)

why can't an outside group just join to absorb us? (answered myself)

The mentorship period lets us screen people who are not really interested PySoy beyond it's name recognition

can't board members cheat votes by mentoring friends just before an election? (answered myself)

In theory, yes, but this is still better than the current situation where all developers have equal vote but Zach exclusivly controls who can be a developer

how can a new structure be implemented if/when this one stops working? (from myself, verbatim answer from Zach)

A `Vote for Change' can be made to replace the Rank structure.

Anyone have more questions or concerns? Throw them at me (Arc on Freenode or arc at pysoy) or publically to the pysoy-dev (@ pysoy) list. Don't be shy, you don't have to be a current developer to comment!

PySoy, a Historical Perspective

With the reorganization discussion underway and so many people joining over the last few weeks I'd like to present a historical perspective of PySoy and where I see things going.

If you're reading Planet PySoy you can access the full text here. It's far too lengthy for syndication.

---

So it all starts with a quirky vegetarian college kid in France using the moniker "Jiba".

Jiba likes to design video games. He isn't good at math or programming and wants a way to build 3d games without having to be good at either.

At some point, perhaps in the late 90's, he knows Visual Basic so he writes a 3d engine (in VB) called Soya. Then he learns Java and writes a new engine called Opale.Soya (Soya 0.2?) for Java with his brother, then in 2003 he and his brother rewrote it again for Python calling it Soya 0.3.

This trend continues, Soya is redesigned and rewritten (in various mixes of C, Python, and PyRex) several times with it's API radically changing in each version. Jiba gets a lot of help from his older brother and friends at school with various parts and Jiba releases a few games that use it.

The story would likely end there if Soya didn't attract interest from developers from other parts of Europe and the US. In 2004-2005 one such developer from the UK, going by the moniker "Dunk" or d1223m, joined the Soya group by adding a new widget system called Pudding. During the same time I (Arc Riley) got involved with Soya as I was using it for my GNU project Aetherspace and to teach 3d programming at a community center in upstate NY.

I organized Team Soya during the first PyWeek competition which created a lot of new energy in the Soya project (including Soya's 10.1 release). Shortly after the competition Dunk and Jiba had a falling out after a number of disagreements where Dunk suggested that Soya was designed so haphazardly and so flawed that it needed to be rewritten from scratch.

He took several of the newer developers with him, including all those Soya picked up during the PyWeek competition, and started a new 3d engine project (which fizzled out after a few weeks). His earlier work toward this effort was called Soya 2 or "Evil Soya".

I and a few of the developers who stayed with Soya worked out an agreement with Jiba to transform Soya into a community project to allow Soya to turn into something awesome. This agreement was in exchange for not joining Dunk's project, something Jiba fought to prevent.

Fast forward about 8 months.

We saw no new Soya releases, little contributed code was accepted, and we barely saw or heard from Jiba. It was seeming like the only way for things to move forward was for the rest of us to take a leadership role and make them happen. I registered Soya3d.org as a Trac site (wiki, subversion, tickets, etc) and many of us started using it.

I organized Soya's participation in the Google Summer of Code and, once again, Soya gets a flood of interest from developers (including Jiba). This flood brought a lot of new ideas, many of which Jiba disagreed with, and the "community leadership" agreement we made a year prior turned to dust as Jiba struggled to maintain control against what he perceived as a hijacking of his project.

Emotions were really strong over this. On IRC one night, after an argument, Jiba suggested (perhaps not seriously) we fork Soya. By the time he returned to IRC the next day the PySoy domain (pysoy.org) was registered and the fork was officially announced.

After a heated few days the 2 Summer of Code students were divided as per their mentors; Jiba's local friend Pierre-Yves (marmoute) stayed with Soya and Buddha's (blaisepascal) student Palle (praabjerg) with PySoy.

This unfortunately required employees of Google to sort out as Jiba was demanding both students and organized many of his friends to "rally" for him. Blog entries were written, #PySoy started getting flooded by friends of Jiba, and they petitioned support "in defense" of Soya from the larger Python community.

Issues spanned the range of our use of the word "Soy" being disambiguous with "Soya", who mentored the Summer of Code students, our retention of the Soya3D.org domain for forwarding ticket URLs to pysoy.org, our copying of wiki content and documentation (Jiba claims it was copyright violation since he never freely licensed it), and accusations of malicious server infiltration and domain hijacking.

This became serious enough for several Freenode IRC admins (mainly Rob Levin) to get involved ending in most of France (entire netblocks) banned from #PySoy and moderated on our mailing lists. These policies have continued to date.

With the fork issue finally calming down we went to work. Tasks were divided, discussions were held, issues were sorted out. Several former Soya developers, hearing about the fork from all the noise Jiba's friends were kicking up, joined us.

Zach became the maintainer and I, after getting distracted with other projects, stayed on to manage the website and any business matters that came up.

In my perspective, Zach was a bit intimidated by the lack of structure. We run by democracy, great, but who votes? How? When? Where? Does this mean some guy just joining is counted equally as someone who's poured their heart and soul into the project for a year? Or does it mean only people we hand-pick get to vote?

The result of this, as I see it, was a move toward insularization. Democracy, yes, but only within a closed group with membership limited to trusted developers. IRC stopped happening, the website stopped being used, the mailing list went silent, subversion updates slowed, even blogs were only updated when I asked something be posted.

This wasn't what any of us wanted to see but it kept chaos at bay in lack of any other structure, and productive work was being done, so this is the way it's been for months now.

The team Zach organized reached the same assessment (unanimously?) that Dunk reached the previous year; it would be easier to write a new engine from scratch than fix Soya even after most of the API was cleaned up. Doing so would also clear up a sticky legal situation with Jiba holding a copyright on much of the code and being unwilling/unable to defend the GPL in court.

We like the general idea of Soya - Jiba was brilliant in some of his ideas on how to abstract OpenGL behavior to a high level language such as Python. It's implementation that he fell short on.

Building an engine from scratch was a much larger scope than we started with. On one hand every release goal was getting thrown away, it has taken months (and will take many more) to get a working release out. On the other hand, PySoy will be faster and easier to work with as a result.

Although Zach was in support of this he was obviously becoming overwhelmed. On top of being the chief architect and maintainer of PySoy he had people emailing him, wanting to get involved, and helping them get started. He apparently also had several new people making commitments, contributing some code, then disappearing before they finished.

In the last two weeks alone I've seen 8 versions of the phrase "it would have been faster/easier for me do it myself" in emails from Zach as entire source files contributed by absent developers got nixed or rewritten. I'd have to say he's contributed roughly 3/4 of the current codebase himself, much of that code a new developer promised to do and dropped the ball on.

Three days go I got an email from Zach that sounded like he was quitting as maintainer, two others CC'ed on that email thought the same thing. In response to my request for clarification he proposed what we're now calling the "Rank" system.

---

While I have reservations with some elements of the rank proposal (and some of my concerns have been addressed), overall, I think this is a really healthy direction to move in. It's certainly in the spirit that PySoy was created.

I think a bit of perspective is needed, however, as we discuss it.

Even once implemented it's not set in stone. If we implement it and it doesn't work, or stops working, we'll modify it or try something entirely different. The question is not "will this work for us forever". We don't need to craft a structure that will scale from a dozen developers to several hundred.

The question is, simply, "will this work for us right now".

The PySoy community has adapted, and will continue to adapt, to our current needs just as it has over the past 3+ years. All we need for now is a structure to get through the first release cycle. We'll adapt it as needed.

site fixed

A trac upgrade rendered much of the PySoy site unusable for a time, all fixed up now.

There's a fix to the pydoc plugin pending the new docs load. Everyone should know by now that the work Zach and the team are doing obsoletes what's currently there.

Let me know if there's any further issues with the site.

Reorganization

I've been talking to Zach about PySoy and we've reached a solution to continue from.

First, work will continue on subversion/trac as it has. Zach's been given admin access to the website and has accepted responsibility to update the ticket system, roadmap, and post design docs on the wiki.

Next, I've passed the role of "maintainer" to Zach. We'll continue to operate democratically for major design decisions and releases but someone has to be able to make trivial choices about checkins, branches, etc.

Finally, being as some of us just aren't cool enough for Jabber, we're going to setup a jabber-irc bridge for #PySoy and get all of his crazy Python 3D crew onto the -dev mailing list.

I'm remaining as part of the project but with the new business and related deadlines I'm unable to dedicate as much time to the PySoy project as I could before. Progeny Games will still be using Python & PySoy and will continue to fund specific PySoy enhancements.

BSP-Tree vs Z-Buffer

The standard zbuffer which we're using now renders from front-to-back for pixels, such that if two polygons intersect it will show the intersection properly and no time is wasted drawing pixels that are blocked by other polygons.

The zbuffer has downsides - beside slowing down the GPU, it's not always accurate (especially if the range of vision is deep), it doesn't allow proper display of translucent polygons, and sees all polygons as a random "soup" with no object-level abstraction.

PySoy inherited intra-object BSP tree calculations from Soya allowing large objects (such as a whole level as one entity) to use BSP tree optimisation. This can easily be extended to all entities such that every object can have unseen polygons pruned and the faces rendered in the proper order.

If objects cannot intersect with a physics model which forbids it then objects can be sorted by a BSP-tree dynamically, then each object can be rendered individually by it's precomputed order. Further optimisations can be made to only recalculate the object-level BSP-tree when objects have moved sufficiently to justify it (with a dynamic accuracy based on CPU load).

Just some ideas to chew on, nothing that's going to get into 1.0...

1.0 focus

With the target a week away and still quite a bit away from being ready, I've started ripping up the API and trying to put together something more coherent. Soya has accumulated *SO* much extra baggage in it's namespace: copies of root level classes with only slight difference in function (ie, Shape, SolidShape, CellShadingMesh, etc), convience values such as WHITE, BLACK, and TRANSPARENT mixed into the root namespace, oh and let's not even touch the ODE issue yet.

The renaming process is also the opprotunity to clean the code up. Many of these copies remove redundant code, thus decreasing the amount that will need to be debugged later, and making things generally easier to read.

At the very least I'd like to see a beta released by the target date, though we should keep in mind that the API in 1.0 will haunt us for months (or even years) to come.

New Physics

Maybe I'm insane, but I've been mulling over some ideas to fully integrate ODE with PySoy, not for collision detection, but for rigid body physics.

This new model would let ODE handle velocity, rotation, etc for all Nodes and Entities directly in the Scene (Nodes treated as a group of joined entities) with the controller applying forces to various objects to affect them.

This new model, which includes Cal3D characters, would allow much more dynamic characters (adjusting walk cycles to terrain, able to trip over things, etc) than was feasable previously while allowing character speed to be set just with animation speed (ie, no more animation+movement syncing needed) since the animation of their feet would propel them forward.

I don't think this has ever been done before, certainly not in a free software game.

Forking Flurry

It seems I was the last one to see it, but it became nessesary for a Soya3d fork to be created in order to build it into a community project.

I've spent the last 28 hours sorting the website business out. Redirects from the various parts of the old soya3d.org now point to their respective elements at pysoy.org, some of which are in read-only states until everything is done. Having external links working has been my top priority.

On the PySoy site we have permissions issues in the DB being sorted through. I'm working through them as quickly as possible but it's going to take me a few more days to get everything working stable again. It's a large website, after all, with many plugins added.

I'm also working to get the mailing lists up. GNU Mailman has changed the way it wants to be installed (-D MAILMAN flag to apache?). I'm getting this done in my free time, between work for paid clients, so it'll be done as soon as it can.

.soy progress

The new 2-tier format is written out now, seperating object level from object-data level. This was pseudo-done in the former draft (blockid 0-127 == object level, 128+ object-data level) but this is far, far cleaner.

This allows objects to be quickly skipped-over on read/import, adds a much larger id-space both for types of objects and types of data within each object, and cleaner discard/recovery methods based on Palle's experience implementing the importer. We should have the blender importer/exporter up to something resembling this spec by tonight.

Check the updated .soy file format spec and give constructive criticism.

Exit xml, Enter .soy

I did a number of tests last night, and even in pure Python (where XML is highly optimized) it was far, far too slow. It was also several orders of magnitude larger than it should be, even gzipped, which is bad when transfering over the network in realtime.

Another problem, too, was the Mesh API. If the Blender exporter was to have the ability to send pre-optimized data (from Blender itself) into a soya.Mesh, it dictated a very complex API which would allow both editing and exporting to be done at reasonable speeds.

So it's all ripped up now (literally, I took the newsprint draft pages and ripped them up) and I started work on an extensible binary block format comperable to AIFF, RIFF, TIFF, PNG, etc. This will allow new features to be added while both newer files compatable with older software and newer software compatable with older files. IE, full backward and forward compatability. The draft spec can be found at http://www.soya3d.org/wiki/Formats/soy

The format is being designed with full optimizations already made; not only the various data structures already seperated, indexed, and linked, but ordered. That is, it can take advantage of vertex buffer optimization (with no specific software/format support for it).

It's structure overhead is very low, I'm guessing real world uses will pit it lower than 1% for size. With support for it implemented in C/Pyrex loading and saving should be simple. It also allows a much more relational data layout tree than Soya currently uses; it'll be possible to include multiple data objects (UNIX cat command can do it, just like Ogg) in a single file and reference them with path/filename#name just as you do in HTML.

While still in heavy development (and unfinished), comments and feedback are requested.

Drafting an XML format

I'm a bit suprised at myself, just last year arguing that XML was being over-used, now attempting to implement an optimized XML format for Soya.

The job isn't simple, since I'm working on soya.Mesh at the same time and these are intricatly linked. On one side we have Blender/etc which already have neighbors, normals, etc computed. On the other we want the data to be user-editable in it's text format. And lastly, Soya needs to be able to load and save this format more efficiently than Cerializer.

Blender importers/exporters should be able to drop data into this format without using Soya. It should survive extension for future Soya features and be permissable both in a file by itself and bundled with other data.

Subversion mirroring

I have a full Soya trunk mirror running (updated every 5 minutes) at www.soya3d.org now. This is useful for both the source browser and branching (for those without GNA! access).

Unfortunetly, because this isn't a full mirror the revision numbers are out of sync. A full mirror would require setup from GNA!'s side as well as some directory rearrangement on both ends so all the data can exist harmoniously.

SoC Python ranking finished

Well, Soya got 2 projects this year. They are:

• Soya3D Collision API : Improving ODE integration in the core by David Pierre-Yves


• Soya3D exporting/importing tools for Blender by Palle Raabjerg

Of the previously mentioned interesting non-Soya proposals, Base multidimensional array type for Python core and Python interface for writing Mozilla (NPAPI) plugins (which mentions a Soya interface as a goal) also made it into the set of 25 proposals to be accepted.

We're awaiting Google's confirmation of these, but unless one of the applicants is found ineligable, all four of these should be approved.

There's a bit of disappointment in the air over the other 4 Soya proposals that didn't make it due to the unexpected low number of project slots offered by Google, but 2 is better than none.

The Cut

I think we just finished one of the most hard decidions we've had so far. Out of the remaining 5, only 2-3 are likely to be approved. All of the remaining proposals are excellent, and each of the students very qualified, so the choice comes down to which projects are most needed for Soya.

We had a public Soya meeting where students had a chance to discuss their projects in more depth, then us mentors discussed them for another hour. I won't post what the outcome was, but we've ranked the proposals in preference. From here, it's up to the other PSF mentors and admins to decide how many slots we'll receive and possibly even re-order them.

Tomorrow at approximately 8PM EDT we'll have the tenative results. Tuesday Google will be posting the winning proposals. Only then will the final results be known (after all, one of the students could be disqualified/etc).

SoC final cut begins...

"N"'s are out. PSF was allocated 25 proposals, tied with Apache for the maximum.

One of the Soya proposals has dropped out of the range due to negetive feedback from last year's mentors re: the student.

The remaining 5 Soya proposals are hovering around the #25 marker and may move radically over the next 48 hours.

Also, it's possible the #25 may be boosted to 26 or 27 before the end.

It's all up to the other Python mentors now. Wether they vote to downgrade the Soya apps or promote others in their slots will detirmine which get approved.

SoC update

We're still waiting for numbers from Google. They're apparently still working on the tools to generate the number of projects each organization will receive (tweaking variables/etc).

There's been some dire numbers being thrown around, such as 75% of the orgs receiving 5 or fewer projects, but the PSF should certainly be in the upper 25%. The lowest Soya app is now rated at #28.

Note, however, if the number PSF receives is very low - 30 or 35, a chaotic re-ordering is likely and anything is possible. I'm still betting on a higher number; it'd be entirely unfair for smaller orgs with 2-4 mentors to receive a roughly 1:1 student:mentor ratio while larger orgs to have fewer than half as many projects as mentors. We'd end up with a good number of mentors jaded from the time-consuming last few weeks not producing anything productive.

Summer of Code project ranking

We're expecting our nearly-official "N" (number of projects) from Google tomarrow. Python Software Foundation is requesting 70 proposals, it's very unlikely we'll receive fewer than 50.

Out of over 200 applications, the lowest Soya app we're considering is ranked at 31. 6 out of 7 isn't bad at all. One application for a water-wave-effects class was removed from consideration when we received a much stronger proposal for roughly the same project.

Currently ranked at 39 is an extremely interesting project by Marcin Pertek for building a Mozilla/Firefox plugin interface for Python. Basically, if completed, we could write Firefox plugins using PyGame, Soya, or any number of other gui/widget modules.

Of course, for security, we need a .xml file format for 3d objects before Soya is used for building Plugins. Another great example for why we need to stop bundling code with data via Python pickles.

Andrew Bouchard has another interesting proposal up (currently ranked 34) for an artificial intelligence suite in Python. His focus is primarily robotics though there's only a marginal difference between game ai and robotics ai when you work with cal3d directly.

The most interesting PSF proposal, ranking solid at 9, is Karol Langner's "Base multidimensional array type for Python core". Anyone who's needed to use an array[x][y][z] for 3d data storage can surely see the value of having these types replaced by an optimized multidimensional list in Python core. Prehaps we'll see this with Python 2.5.

I think these are all a sure bet to be approved. We'll know for sure on Monday.