subversion vs git

I stayed up late last night working to install a shared git repository for the Pyrex replacement project. First of all, copious thanks to johnw and Ilari on #git for their help on this.

I'm very much an enthusiast of Trac, it's an excellent engine for project websites and has all the essential tools, builtin and via plugins, that any project should need. Ok, so there's a few things a project could need that isn't already written, but new plugins are easy enough (it's all Python, after all).

As such, I like to leave user administration to Trac, and it maintains a .htusers and .access file local to each project for that purpose. Likewise, for SVN, we use WebDAV on Apache, whereas the ssh+svn method would require adding those users to the local system or PAM hacking.

Thus, the most direct path would seem to install GIT via WebDAV and run it from there. As Ilari pointed out, GIT's DAV has many problems over SVN, from lockups to a failed push sometimes corrupting the shared repository. I tried anyway, with hours of help from joshw, and at 5am I had to throw in the towel. I'm sure it's possible, but not with my current lack of GIT knowledge.

The other available solutions could work, but first I'd want to write some new Trac plugins for accepting developer ssh keys and using these on the backend to govern access or figure out the PAM configuration for .htusers to have restricted SSH access for GIT pushes.

There are certainly advantages to GIT, but server setup difficulty and added complications for Windows developers leads me to sticking with subversion servers with git-svn client-side for now.

a radical redirection

We've been at a deadlock for a few weeks now, our previous coding pace has turned into a lot of contemplation and side discussions about the greater issues with the PySoy codebase.

I think we have a roadmap forward, or enough of one to post these in a blog entry:

First, PySoy 1.0 will not be released until after Python 3.0 this Fall, and it would be pointless to continue developing with 2.x in target. Python 3.0 also has a large number of useful changes for our project. Our target platform should therefore be 3.0 (and it's alpha/beta's for now).

Second, with the release of Pyrex 0.9.7 Greg has, once again, introduced a major language change (the "for" statement) without even soliciting input from the community. This has sealed the deal for us, even if our immediate issues are fixed we cannot continue to subject ourselves to a language that changes in incompatible ways between 2 micro versions (0.9.6 -> 0.9.8, when the old syntax will not be supported).

We earlier started a Cython variant to support PySoy's codebase, but given recent developments in their community, the timelines of our projects, and their choice to stick with Python 2.x for now, we're moving forward on a Pyrex replacement written from scratch and targeting Python 3.0.

We're going to use a similar language style to Pyrex, Greg certainly had some good ideas, and will keep language porting in mind, but there will be changes. We will also not be using any of his code including his Plex module, Pyrex and Cython's lexical analyzer which is the core of those packages.

PySoy Beta3, earlier aimed for release in "early Spring", has been indefinably postponed as we work on that. The subversion repository is of course open, and development can and will continue, but we cannot release until the new build system is complete.

initial impressions of Trac 0.11

Trac 0.11rc1 was released this week, marking a feature freeze for 0.11.

One of the major improvements over 0.10.4 is the Genshi template system allowing complete control over a site's HTML. For example, you can copy the default layout.html to your site's templates/ directory to remove the obnoxious "Trac" string at the end of every page's title and the other Edgewall advertising splayed all over a default Trac site.

Of course, this is also means a Trac site can be radically redesigned in structure without editing the source code. Hopefully this leads to a greater diversity in project site design.

Another major feature is ticket workflows are now customizable, allowing projects using Trac to decide how their tickets will be processed. For example, a "testing" state could be added to signify a ticket as "though to be complete, but waiting for verification".

I plan on upgrading pysoy.org this weekend along with a few new Trac plugins and a minor site redesign.

some race conditions found

Well after many weeks we found the source of the race conditions on multicore CPUs; they were not in PySoy at all but in the Pyrex compiler.

Throughout the generated .c we found INCREF/DECREF where it didn't belong, namely in every cdef method for __pyx_v_self. In other words, even though the calling function must already hold a reference to instance in order to access it's C methods, thus preventing the instance from being garbage collected during method execution, the Pyrex.Compiler inserts redundant INCREF/DECREF calls.

These would normally just waste CPU cycles, but since our 3d engine has non-GIL background threads these refcount adjusting calls are accessed in a thread-unsafe manner and will inevitably cause either a segfault or a memory leak.

The solution is to not consider "self" a normal Python argument, which would also allow these C methods to be called with nogil: set, provided checking is done to ensure no Python methods, properties, or variables of self are accessed.

We'll be fixing this problem with our new source compiler, to be announced soon.

1.0_beta3 release and a Sprint this weekend

There is a rumor going around that because our 1.0_beta3 milestone is marked as due today (3/21) we're planning a release today. This is false, we'll be releasing sometime in early spring as advertised - the due dates marked for our milestones are our preferred date only.

We're still tracking down one or more nasty race conditions which only show up on multicore systems, we still have a bit more work to do before the next release, and a good deal more testing is needed after these bugs are found to demonstrate stability. If you're experience with debugging we could use some help - some of our tests seem to indicate mutex locks being ignored.

This weekend is a good time, too. Starting today (Friday) we're running a pre-Beta3 online sprint. In our IRC channel (#PySoy on irc.freenode.net) you can find a good number of developers working almost around the clock on various parts of the 3D engine and a few working in Gobby for remote pair programming.

If you're not into debugging, or don't have a multicore system, there are many other tasks - large and small - that need doing.

State of the Soy @ r1000

Last night our humble 3D game engine reached 1000 commits to it's Subversion repository.

PySoy has been contributed to by 20 developers, 7 of which are currently active. The project's IRC channel is in nearly constant use by developers working on different tasks.

The external API has drawn considerable praise by users, despite it's state as beta, and the constructive feedback we've received has led many of the post-Beta2 API changes.

The internal API, while having some rough edges, has been commented on by several new developers as remarkably clean. We've passed the point which the COMPLETED list exceeds the TODO list, despite the latter constantly growing, and focus has largely shifted toward debugging, documentation, and polish.

Most notably, we've reached the point of needing a legal entity for the project. Using the Software Freedom Conservancy as a rough template, we've started the process to form a non-profit corporation with a focus on copyleft game projects. An announcement about this will be posted as more details are firmed up.

PySoy goes from 3 threads to 5

We're doing the final work now to replace CoreLoop with TransportLoop, WindowLoop, and SceneLoop. All three are instances of soy._internals.LoopThread which (should) play well with Python's threading system and should never hold the GIL.

The increased performance this has brought for multi-core users is amazing, some have reported more than twice the framerate!

To be fair, an equal number of subversion Trunk users are experiencing frequent segfaults since we've not finished adding mutexes and rearranging the internal API post-move. Obviously these issues are being resolved prior to the Beta-3 release next month.

why we're blocking Microsoft Live

While going through server logs I noticed something funny, Microsoft Live has surpassed Google in the number of hits we've received. Weird, eh? I didn't think anyone actually used it.

The terms people arriving at our site through search.live.com are just... weird, though. Most are outright vulgar, searching for obscure pornography or celebrity names, drugs, sex aids... Here's an few examples:

65.55.165.36 http://search.live.com/result.aspx?q=valtrex&mrt=en-us&FORM=LVSP
/log/trunk Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR
1.1.4322)
131.107.0.95 http://search.live.com/result.aspx?q=breast+enhancement&mrt=en-us&FORM=LVSP /
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; Win64; x64; SV1)
65.55.165.122 http://search.live.com/result.aspx?q=ferarri&mrt=en-us&FORM=LVSP
/browser/media/tutorials Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2;
.NET CLR 1.1.4322)

These hits were to small obscure pages on our site, such as svn changelogs, and then I noticed the source IP addresses were in just a few IP ranges, so I ran a whois to see if one ISP tied connected them all;

arc@sobek ~/work/pysoy $ whois 65.55.165.83

OrgName:    Microsoft Corp
OrgID:      MSFT
Address:    One Microsoft Way
City:       Redmond
StateProv:  WA
PostalCode: 98052
Country:    US

NetRange:   65.52.0.0 - 65.55.255.255
CIDR:       65.52.0.0/14
NetName:    MICROSOFT-1BLK

You read that correctly. Microsoft, in a desperate attempt to make themselves seem more important, or perhaps just to flood free software project's websites with unwanted traffic, is running bots which act like normal web crawlers. Indeed, over 97% of the hits we got from search.live.com were from Microsoft's own IP subnets. Searching Google, I found this story was previously covered by others more observant of their logs.

In response, I'm adding a special rule to block all future traffic from the offending netblocks, including MICROSOFT 131.107.0.0 - 131.107.255.255 and MICROSOFT-1BLK 65.52.0.0 - 65.55.255.255.

libjingle disappointments

Last year we chalked libjingle as our choice for a XMPP library, since it handled all the networking we needed (ICE-UDP, HTTP, etc) tightly integrated with XMPP through a standard protocol.

On closer inspection, we found why libjingle hasn't really been adopted by the community yet. For starters it requires patches to build the latest release with GCC 4, and no release has been made to include these patches in months. Not a good sign.

Next we find there are no dynamic libjingle libraries, it builds as static only, and doesn't build for C programs (only C++). Unless we transformed our project to build as C++ this isn't an immediate option.

Looking over the libjingle code structure, it's in a severe state of poor maintenance. The internal name for it appears to be "libcricket", with naming confusion all over the place, code format inconsistencies, and extremely poorly documented.

Thus, libjingle integration is unlikely until after PySoy's Beta-3 release when we'll have time to write the C shim code as a dynamic library.

All this said, libjingle appears to have great potential for us and the entire free software community, it's just in need of attention from a few detail-oriented developers.

consolidated _core-*/ to _core/

Last year Pyrex, then version 0.9.5.1a, did not support compile-time conditions for things such as platform specific code. To work around this used different _core directories, one for each platform, and a platform test in setup.py would determine which directory soy._core was compiled from.

While functional, this created duplicated work and a fairly messy source tree with four different _core-* directories. With Pyrex 0.9.6 release came support for code such as this:

IF UNAME_SYSNAME == "Windows":
  include "icky_definitions.pxi"
ELIF UNAME_SYSNAME == "Darwin":
  include "nice_definitions.pxi"
ELIF UNAME_SYSNAME == "Linux":
include "penguin_definitions.pxi"
ELSE:
    include "other_definitions.pxi"

... and, thus, we could finally merge those _core-* directories into src/_core and remove kludge from setup.py. Tonight that's exactly what I did. Behold, http://svn.pysoy.org/trunk/pysoy/src