Several times this month I've found www.xmpp.org/, the organization responsible for the Jabber protocol and it's many extensions, non-responsive. Their site has been down for at least 7 hours tonight, the longest yet (to my knowledge).
For our purposes I'll be mirroring their XEP's on pysoy.org as soon as they're back up. It's seriously frustrating to be working on this when the documents you need are often unavailable.
If anyone knows of an existing mirror we could use, please reply to this posting.
Saturday, May 17, 2008
Friday, May 16, 2008
some real progress toward beta-3
With Pyrex 0.9.8 released yesterday, including a number of key changes we needed to move forward, we're back in action for PySoy belated Beta-3 release.
The new forward declarations are a bit kludgy (example), and I don't think it'll be very maintainable in the long term. I think changing a class's inheritance should require editing one file, not two (the .pxd) or more (every .pxd that uses it).
It works for now, until we can wipe out the entire
The nogil allowance for cdef methods is a massive step toward stomping out the multicore race conditions. One of the developers testing just the fixes to
One of the roadblocks I hit is the nogil fix can't be used with our
While our usage is GIL-safe, the current version of Pyrex still wants to throw in an
In the meantime, I've called for an IRC sprint this weekend for the pending cleanup, refactoring, documentation, and testing that needs doing before beta-3. Join
The new forward declarations are a bit kludgy (example), and I don't think it'll be very maintainable in the long term. I think changing a class's inheritance should require editing one file, not two (the .pxd) or more (every .pxd that uses it).
It works for now, until we can wipe out the entire
include/ directory, and to be fair it's only so cumbersome due to PySoy's size and how the classes are interconnected.The nogil allowance for cdef methods is a massive step toward stomping out the multicore race conditions. One of the developers testing just the fixes to
soy.bodies noticed the examples often running longer than before, though there's a lot more cleanup to be done before this is package-wide.One of the roadblocks I hit is the nogil fix can't be used with our
Children class, which stores pointers to Python objects of the same type and uses those pointers (through a typecast) to access their cdef methods in a for loop.While our usage is GIL-safe, the current version of Pyrex still wants to throw in an
INCREF just to be extra-cautious, which makes it not. I've let Greg know about this, we'll see if it can't be fixed soon.In the meantime, I've called for an IRC sprint this weekend for the pending cleanup, refactoring, documentation, and testing that needs doing before beta-3. Join
#PySoy on irc.freenode.net if you want to pitch-in!
Tuesday, May 13, 2008
Building a better Pyrex - part 1
With our recent decision to drop the Pyrex codebase and start from scratch, we find ourselves repeating a process from 2 years ago;
When PySoy first forked off of Soya, we had a small list of things that needed to be fixed based on our negative experiences working with that project. The API needed to be cleaned up, pydocs added, Shape needed to be mutable and implemented with vertex arrays or VBOs rather than display lists, etc.
As we worked on it, that list grew longer, including rewriting large parts of the codebase, until we found ourselves at a point where starting over would be more expedient. Of course now there's little doubt that was a good move, learning from both Soya's mistakes and our own failed early attempts we now have a pretty rocking architecture.
The first and most foundational change we're making from Pyrex, as previously stated, is in the lexical scanner and parser. Both Pyrex and Cython are based on Plex, which was Greg's answer to processing a Python-like language.
In contrast, we'll be using Python's own
This way we have less code to debug and a solid foundation to start with. With lexing and parsing almost "for free" we'll be able to focus on the important bits and get the platform able to support the kind of features we need quickly.
We've also reached a rough consensus on the following syntax changes from Pyrex:
There's more to come, which I'll post in additional installments as work progresses on this project.
When PySoy first forked off of Soya, we had a small list of things that needed to be fixed based on our negative experiences working with that project. The API needed to be cleaned up, pydocs added, Shape needed to be mutable and implemented with vertex arrays or VBOs rather than display lists, etc.
As we worked on it, that list grew longer, including rewriting large parts of the codebase, until we found ourselves at a point where starting over would be more expedient. Of course now there's little doubt that was a good move, learning from both Soya's mistakes and our own failed early attempts we now have a pretty rocking architecture.
The first and most foundational change we're making from Pyrex, as previously stated, is in the lexical scanner and parser. Both Pyrex and Cython are based on Plex, which was Greg's answer to processing a Python-like language.
In contrast, we'll be using Python's own
tokenize module for our lexical scanner and an ASDL-based parser akin to the parser used in Python 3.0. We're extending Python.asdl with cdef, ctypedef, cimport, cinclude, etc.This way we have less code to debug and a solid foundation to start with. With lexing and parsing almost "for free" we'll be able to focus on the important bits and get the platform able to support the kind of features we need quickly.
We've also reached a rough consensus on the following syntax changes from Pyrex:
- unicode, generators, decorators, etc will be supported
- all custom
forsyntaxes dropped in favor of Python's standard with nogil:replaced with fullwithsupport- special methods will always have
selfas first argument (ie, both __sub__ and __rsub__ used) cincludeadded for direct C header parsing
There's more to come, which I'll post in additional installments as work progresses on this project.
setup-pyrex.py works with 0.9.7.2
Greg fixed the latest bug that caused PySoy to raise a dict assignment error on import with 0.9.7.
Pyrex 0.9.7.2 was released a few hours ago with this fix, and I've updated
Beta-3 is still blocked by the multicore race condition caused by redundant incref/decref calls on Python arguments to class methods in this latest version of Pyrex.
Pyrex 0.9.7.2 was released a few hours ago with this fix, and I've updated
setup-pyrex.py to work with either 0.9.6.4 or >=0.9.7.2.Beta-3 is still blocked by the multicore race condition caused by redundant incref/decref calls on Python arguments to class methods in this latest version of Pyrex.
Monday, May 12, 2008
setup-pyrex.py now requires EXACTLY Pyrex 0.9.6.4
16:02 < maacl> import soy
16:02 < maacl> File "LoopThread.pym", line 51, in
soy._internals.LoopThread.__cinit__
16:02 < maacl> TypeError: 'dict' object does not support item assignment
16:03 < Arc> from threading import _active, _active_limbo_lock
16:03 < Arc> self._active = _active
16:03 < Arc> [...] and line 51:
16:03 < Arc> self._active[self._threadID] = self
I'm not going to even look into this further. Pyrex 0.9.7, for a reason I'm not going to even contemplate, raises an error about dict item assignment on the last line. None of us use it due to it's barfing of depreciated lines for all our
for statements, but this brave tested did and reported the problem today.I just committed an update to
setup-pyrex.py that fails when any Pyrex version besides 0.9.6.4 is used.
Sunday, May 11, 2008
subversion vs git
I stayed up late last night working to install a shared git repository for the Pyrex replacement project. First of all, copious thanks to
I'm very much an enthusiast of Trac, it's an excellent engine for project websites and has all the essential tools, builtin and via plugins, that any project should need. Ok, so there's a few things a project could need that isn't already written, but new plugins are easy enough (it's all Python, after all).
As such, I like to leave user administration to Trac, and it maintains a .htusers and .access file local to each project for that purpose. Likewise, for SVN, we use WebDAV on Apache, whereas the ssh+svn method would require adding those users to the local system or PAM hacking.
Thus, the most direct path would seem to install GIT via WebDAV and run it from there. As
The other available solutions could work, but first I'd want to write some new Trac plugins for accepting developer ssh keys and using these on the backend to govern access or figure out the PAM configuration for .htusers to have restricted SSH access for GIT pushes.
There are certainly advantages to GIT, but server setup difficulty and added complications for Windows developers leads me to sticking with subversion servers with
johnw and Ilari on #git for their help on this.I'm very much an enthusiast of Trac, it's an excellent engine for project websites and has all the essential tools, builtin and via plugins, that any project should need. Ok, so there's a few things a project could need that isn't already written, but new plugins are easy enough (it's all Python, after all).
As such, I like to leave user administration to Trac, and it maintains a .htusers and .access file local to each project for that purpose. Likewise, for SVN, we use WebDAV on Apache, whereas the ssh+svn method would require adding those users to the local system or PAM hacking.
Thus, the most direct path would seem to install GIT via WebDAV and run it from there. As
Ilari pointed out, GIT's DAV has many problems over SVN, from lockups to a failed push sometimes corrupting the shared repository. I tried anyway, with hours of help from joshw, and at 5am I had to throw in the towel. I'm sure it's possible, but not with my current lack of GIT knowledge.The other available solutions could work, but first I'd want to write some new Trac plugins for accepting developer ssh keys and using these on the backend to govern access or figure out the PAM configuration for .htusers to have restricted SSH access for GIT pushes.
There are certainly advantages to GIT, but server setup difficulty and added complications for Windows developers leads me to sticking with subversion servers with
git-svn client-side for now.
Saturday, May 10, 2008
a radical redirection
We've been at a deadlock for a few weeks now, our previous coding pace has turned into a lot of contemplation and side discussions about the greater issues with the PySoy codebase.
I think we have a roadmap forward, or enough of one to post these in a blog entry:
First, PySoy 1.0 will not be released until after Python 3.0 this Fall, and it would be pointless to continue developing with 2.x in target. Python 3.0 also has a large number of useful changes for our project. Our target platform should therefore be 3.0 (and it's alpha/beta's for now).
Second, with the release of Pyrex 0.9.7 Greg has, once again, introduced a major language change (the "for" statement) without even soliciting input from the community. This has sealed the deal for us, even if our immediate issues are fixed we cannot continue to subject ourselves to a language that changes in incompatible ways between 2 micro versions (0.9.6 -> 0.9.8, when the old syntax will not be supported).
We earlier started a Cython variant to support PySoy's codebase, but given recent developments in their community, the timelines of our projects, and their choice to stick with Python 2.x for now, we're moving forward on a Pyrex replacement written from scratch and targeting Python 3.0.
We're going to use a similar language style to Pyrex, Greg certainly had some good ideas, and will keep language porting in mind, but there will be changes. We will also not be using any of his code including his Plex module, Pyrex and Cython's lexical analyzer which is the core of those packages.
PySoy Beta3, earlier aimed for release in "early Spring", has been indefinably postponed as we work on that. The subversion repository is of course open, and development can and will continue, but we cannot release until the new build system is complete.
I think we have a roadmap forward, or enough of one to post these in a blog entry:
First, PySoy 1.0 will not be released until after Python 3.0 this Fall, and it would be pointless to continue developing with 2.x in target. Python 3.0 also has a large number of useful changes for our project. Our target platform should therefore be 3.0 (and it's alpha/beta's for now).
Second, with the release of Pyrex 0.9.7 Greg has, once again, introduced a major language change (the "for" statement) without even soliciting input from the community. This has sealed the deal for us, even if our immediate issues are fixed we cannot continue to subject ourselves to a language that changes in incompatible ways between 2 micro versions (0.9.6 -> 0.9.8, when the old syntax will not be supported).
We earlier started a Cython variant to support PySoy's codebase, but given recent developments in their community, the timelines of our projects, and their choice to stick with Python 2.x for now, we're moving forward on a Pyrex replacement written from scratch and targeting Python 3.0.
We're going to use a similar language style to Pyrex, Greg certainly had some good ideas, and will keep language porting in mind, but there will be changes. We will also not be using any of his code including his Plex module, Pyrex and Cython's lexical analyzer which is the core of those packages.
PySoy Beta3, earlier aimed for release in "early Spring", has been indefinably postponed as we work on that. The subversion repository is of course open, and development can and will continue, but we cannot release until the new build system is complete.
Friday, May 02, 2008
initial impressions of Trac 0.11
Trac 0.11rc1 was released this week, marking a feature freeze for 0.11.
One of the major improvements over 0.10.4 is the Genshi template system allowing complete control over a site's HTML. For example, you can copy the default
Of course, this is also means a Trac site can be radically redesigned in structure without editing the source code. Hopefully this leads to a greater diversity in project site design.
Another major feature is ticket workflows are now customizable, allowing projects using Trac to decide how their tickets will be processed. For example, a "testing" state could be added to signify a ticket as "though to be complete, but waiting for verification".
I plan on upgrading pysoy.org this weekend along with a few new Trac plugins and a minor site redesign.
One of the major improvements over 0.10.4 is the Genshi template system allowing complete control over a site's HTML. For example, you can copy the default
layout.html to your site's templates/ directory to remove the obnoxious "Trac" string at the end of every page's title and the other Edgewall advertising splayed all over a default Trac site.Of course, this is also means a Trac site can be radically redesigned in structure without editing the source code. Hopefully this leads to a greater diversity in project site design.
Another major feature is ticket workflows are now customizable, allowing projects using Trac to decide how their tickets will be processed. For example, a "testing" state could be added to signify a ticket as "though to be complete, but waiting for verification".
I plan on upgrading pysoy.org this weekend along with a few new Trac plugins and a minor site redesign.
Wednesday, April 02, 2008
some race conditions found
Well after many weeks we found the source of the race conditions on multicore CPUs; they were not in PySoy at all but in the Pyrex compiler.
Throughout the generated .c we found INCREF/DECREF where it didn't belong, namely in every cdef method for __pyx_v_self. In other words, even though the calling function must already hold a reference to instance in order to access it's C methods, thus preventing the instance from being garbage collected during method execution, the Pyrex.Compiler inserts redundant INCREF/DECREF calls.
These would normally just waste CPU cycles, but since our 3d engine has non-GIL background threads these refcount adjusting calls are accessed in a thread-unsafe manner and will inevitably cause either a segfault or a memory leak.
The solution is to not consider "self" a normal Python argument, which would also allow these C methods to be called with nogil: set, provided checking is done to ensure no Python methods, properties, or variables of self are accessed.
We'll be fixing this problem with our new source compiler, to be announced soon.
Throughout the generated .c we found INCREF/DECREF where it didn't belong, namely in every cdef method for __pyx_v_self. In other words, even though the calling function must already hold a reference to instance in order to access it's C methods, thus preventing the instance from being garbage collected during method execution, the Pyrex.Compiler inserts redundant INCREF/DECREF calls.
These would normally just waste CPU cycles, but since our 3d engine has non-GIL background threads these refcount adjusting calls are accessed in a thread-unsafe manner and will inevitably cause either a segfault or a memory leak.
The solution is to not consider "self" a normal Python argument, which would also allow these C methods to be called with nogil: set, provided checking is done to ensure no Python methods, properties, or variables of self are accessed.
We'll be fixing this problem with our new source compiler, to be announced soon.
Friday, March 21, 2008
1.0_beta3 release and a Sprint this weekend
There is a rumor going around that because our 1.0_beta3 milestone is marked as due today (3/21) we're planning a release today. This is false, we'll be releasing sometime in early spring as advertised - the due dates marked for our milestones are our preferred date only.
We're still tracking down one or more nasty race conditions which only show up on multicore systems, we still have a bit more work to do before the next release, and a good deal more testing is needed after these bugs are found to demonstrate stability. If you're experience with debugging we could use some help - some of our tests seem to indicate mutex locks being ignored.
This weekend is a good time, too. Starting today (Friday) we're running a pre-Beta3 online sprint. In our IRC channel (#PySoy on irc.freenode.net) you can find a good number of developers working almost around the clock on various parts of the 3D engine and a few working in Gobby for remote pair programming.
If you're not into debugging, or don't have a multicore system, there are many other tasks - large and small - that need doing.
We're still tracking down one or more nasty race conditions which only show up on multicore systems, we still have a bit more work to do before the next release, and a good deal more testing is needed after these bugs are found to demonstrate stability. If you're experience with debugging we could use some help - some of our tests seem to indicate mutex locks being ignored.
This weekend is a good time, too. Starting today (Friday) we're running a pre-Beta3 online sprint. In our IRC channel (#PySoy on irc.freenode.net) you can find a good number of developers working almost around the clock on various parts of the 3D engine and a few working in Gobby for remote pair programming.
If you're not into debugging, or don't have a multicore system, there are many other tasks - large and small - that need doing.
Subscribe to:
Posts (Atom)