Well after many weeks we found the source of the race conditions on multicore CPUs; they were not in PySoy at all but in the Pyrex compiler.
Throughout the generated .c we found INCREF/DECREF where it didn't belong, namely in every cdef method for __pyx_v_self. In other words, even though the calling function must already hold a reference to instance in order to access it's C methods, thus preventing the instance from being garbage collected during method execution, the Pyrex.Compiler inserts redundant INCREF/DECREF calls.
These would normally just waste CPU cycles, but since our 3d engine has non-GIL background threads these refcount adjusting calls are accessed in a thread-unsafe manner and will inevitably cause either a segfault or a memory leak.
The solution is to not consider "self" a normal Python argument, which would also allow these C methods to be called with nogil: set, provided checking is done to ensure no Python methods, properties, or variables of self are accessed.
We'll be fixing this problem with our new source compiler, to be announced soon.
Wednesday, April 02, 2008
Subscribe to:
Posts (Atom)