Well after many weeks we found the source of the race conditions on multicore CPUs; they were not in PySoy at all but in the Pyrex compiler.
Throughout the generated .c we found INCREF/DECREF where it didn't belong, namely in every cdef method for __pyx_v_self. In other words, even though the calling function must already hold a reference to instance in order to access it's C methods, thus preventing the instance from being garbage collected during method execution, the Pyrex.Compiler inserts redundant INCREF/DECREF calls.
These would normally just waste CPU cycles, but since our 3d engine has non-GIL background threads these refcount adjusting calls are accessed in a thread-unsafe manner and will inevitably cause either a segfault or a memory leak.
The solution is to not consider "self" a normal Python argument, which would also allow these C methods to be called with nogil: set, provided checking is done to ensure no Python methods, properties, or variables of self are accessed.
We'll be fixing this problem with our new source compiler, to be announced soon.