We've made some massive headway in the last two weeks;
First, we have physics processing back. We're using ODE for now just to get us through the next release cycle or two, but we've modified its headers (ABI-compatible) to work better with libsoy. Done are the days when we stress over API compatibility, since we've decided to write our own fully integrated (and scaled down) physics processing with Orc we have a certain freedom to throw their conventions to the wind.
How this is going to play out is a slow migration of values to arrays. Body data, for example, will be stored an array inside a scene so SIMD processing can be used more efficiently. The Soy Body class will represent a pointer to that array and include properties and methods for working with its data. The actual storage of that data will be governed by the Scene, giving it the freedom to sort bodies per "island" (groups of bodies connected by group). This is similar to how ODE processes bodies already and we'll be likely using their code as a starting place.
Second, we have the basic types for rendering. At the moment this is just a rotating cube, but that involved roughly 6 classes operating in both GObject and Python domains. The next step is soy.models.Mesh for rendering more complex shapes.
Third and most exciting, our humble game engine can now run as a Firefox plugin! There's some bugs to be worked out still, but "embedded" support has been added to our soy.widgets.Window class that allows it to run inside virtually any application.
With luck and determination we'll have the next release out by the end of the month. I've dialed back our goals considerably given that its a rewrite from our last beta release but our release cycle will be much shorter now that the rewrite is finishing up.
Monday, February 28, 2011
Saturday, February 19, 2011
New import (and export) mechanism
I've started writing a new import mechanism for .soy files, replacing the old
With the old method, "transport" objects acted as dicts of objects in the .soy files and could be loaded and saved as such. The resulting code looked like this:
Now, the new .soy importer is added to sys.path_hooks when PySoy is first initialized and applys to any directory added after. When finished, the above code could be rewritten as:
No "etc" needed because in that one import command data/MyGame.soy is loaded and made available directly in a module called MyGame. Much more pythonic, IMHO.
A separate class will be written next called
Similarly, two archives could be combined like this:
The reasoning behind this is in most cases .soy files will be written by GIMP or Blender, so the most common use case for the
As always, constructive/critical feedback welcome and appreciated.
soy.transports module with a importer that allows .soy files to be loaded as modules with the standard Python import command.With the old method, "transport" objects acted as dicts of objects in the .soy files and could be loaded and saved as such. The resulting code looked like this:
import soy
MyGame = soy.transports.File('data/MyGame.soy')
window = MyGame['window']
redblock = MyGame['redblock']
Now, the new .soy importer is added to sys.path_hooks when PySoy is first initialized and applys to any directory added after. When finished, the above code could be rewritten as:
import soy
sys.path.append('data/')
import MyGame
No "etc" needed because in that one import command data/MyGame.soy is loaded and made available directly in a module called MyGame. Much more pythonic, IMHO.
A separate class will be written next called
soy.Exporter which PySoy objects can be added and removed from, then saved to a file such as this example which modifies the previously imported MyGame module:
archive = soy.Exporter(MyGame)
archive.grasstex = grass_texture
del(archive.bricktex)
archive('data/MyGame2.soy')
Similarly, two archives could be combined like this:
archive = soy.Exporter(MyGame)
archive2 = soy.Exporter(CharacterFoo)
archive += archive2
archive('data/MyGame3.soy')
The reasoning behind this is in most cases .soy files will be written by GIMP or Blender, so the most common use case for the
soy.Exporter class is for combining and pruning the .soy files created by these tools.As always, constructive/critical feedback welcome and appreciated.
Saturday, February 05, 2011
libsoy migration milestone
After a great deal of time (I'd rather not admit how long) we've finally managed to link the new libsoy code in with Python.
This afternoon (around 4:30) the following code opened a PySoy window on the screen:
While incomplete and, well, its only a window, this is the first time we've gotten the two pieces to work together. David Czech and I have spent the rest of the evening fixing both build systems (libsoy and PySoy) and getting ready to copy the window binding code for the rest of the engine.
Our largest failure in this has been diverting so much energy into GObject Introspection. While its a really great idea and I'd love to see it evolve, its still so deep alpha that I can't imagine anyone but Gnome developers getting much practical use out of it. I would like to finish up GTypes at some point, but either the XML format for .gir or any level of documentation on typelib needs to be written first. I've wasted far too much time already trying to reverse engineer these from examples and their source that should have been put into my own projects.
This afternoon (around 4:30) the following code opened a PySoy window on the screen:
arc@khonsu ~/work/pysoy $ python3
Python 3.1.3 (r313:86834, Dec 5 2010, 09:55:24)
[GCC 4.5.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import soy.widgets
>>> w = soy.widgets.Window()
>>>
(soy:16042): Gdk-CRITICAL **: IA__gdk_window_set_title: assertion `title != NULL' failed
IRQ's not enabled, falling back to busy waits: 2 0
OpenGL version 2.1 Mesa 7.10
GL_ARB_vertex_buffer_object: Yes
While incomplete and, well, its only a window, this is the first time we've gotten the two pieces to work together. David Czech and I have spent the rest of the evening fixing both build systems (libsoy and PySoy) and getting ready to copy the window binding code for the rest of the engine.
Our largest failure in this has been diverting so much energy into GObject Introspection. While its a really great idea and I'd love to see it evolve, its still so deep alpha that I can't imagine anyone but Gnome developers getting much practical use out of it. I would like to finish up GTypes at some point, but either the XML format for .gir or any level of documentation on typelib needs to be written first. I've wasted far too much time already trying to reverse engineer these from examples and their source that should have been put into my own projects.
Thursday, December 30, 2010
common Orc opcodes
I've been going through liboil's 0.3 source to rewrite the oil_yuv2rgbx_sub2_u8 function we use for Theora decoding to Orc pseudo-assembly code.
Because the Orc opcode documentation splits opcode description and processor support between two tables, for reference I wrote a quick Python script to build a table of Orc opcodes common to SSE (x86), Altivec (PPC/Cell), and NEON (Arm Cortex) processors.
Here's that table for reference, at least until I put the time to format it for a wiki:
Because the Orc opcode documentation splits opcode description and processor support between two tables, for reference I wrote a quick Python script to build a table of Orc opcodes common to SSE (x86), Altivec (PPC/Cell), and NEON (Arm Cortex) processors.
Here's that table for reference, at least until I put the time to format it for a wiki:
| opcode | dst | src1 | src2 | description | pseudo code |
|---|---|---|---|---|---|
| absb | 1 | 1 | absolute value | (a < 0) ? -a : a | |
| addb | 1 | 1 | 1 | add | a + b |
| addssb | 1 | 1 | 1 | add with signed saturate | clamp(a + b) |
| addusb | 1 | 1 | 1 | add with unsigned saturate | clamp(a + b) |
| andb | 1 | 1 | 1 | bitwise AND | a & b |
| andnb | 1 | 1 | 1 | bitwise AND NOT | a & (~b) |
| avgsb | 1 | 1 | 1 | signed average | (a + b + 1)>>1 |
| avgub | 1 | 1 | 1 | unsigned average | (a + b + 1)>>1 |
| cmpeqb | 1 | 1 | 1 | compare equal | (a == b) ? (~0) : 0 |
| cmpgtsb | 1 | 1 | 1 | compare greater than | (a > b) ? (~0) : 0 |
| copyb | 1 | 1 | copy | a | |
| loadb | 1 | 1 | load from memory | array[i] | |
| loadpb | 1 | 1 | load parameter or constant | scalar | |
| maxsb | 1 | 1 | 1 | signed maximum | (a > b) ? a : b |
| maxub | 1 | 1 | 1 | unsigned maximum | (a > b) ? a : b |
| minsb | 1 | 1 | 1 | signed minimum | (a < b) ? a : b |
| minub | 1 | 1 | 1 | unsigned minimum | (a < b) ? a : b |
| mullb | 1 | 1 | 1 | low bits of multiply | a * b |
| mulhsb | 1 | 1 | 1 | high bits of signed multiply | (a * b) >> 8 |
| mulhub | 1 | 1 | 1 | high bits of unsigned multiply | (a * b) >> 8 |
| orb | 1 | 1 | 1 | bitwise or | a | b |
| shlb | 1 | 1 | 1S | shift left | a << b |
| shrsb | 1 | 1 | 1S | signed shift right | a >> b |
| shrub | 1 | 1 | 1S | unsigned shift right | a >> b |
| signb | 1 | 1 | sign | sign(a) | |
| storeb | 1 | 1 | store to memory | special | |
| subb | 1 | 1 | 1 | subtract | a - b |
| subssb | 1 | 1 | 1 | subtract with signed saturate | clamp(a - b) |
| subusb | 1 | 1 | 1 | subtract with unsigned saturate | clamp(a - b) |
| xorb | 1 | 1 | 1 | bitwise XOR | a ^ b |
| absw | 2 | 2 | absolute value | (a < 0) ? -a : a | |
| addw | 2 | 2 | 2 | add | a + b |
| addssw | 2 | 2 | 2 | add with signed saturate | clamp(a + b) |
| addusw | 2 | 2 | 2 | add with unsigned saturate | clamp(a + b) |
| andw | 2 | 2 | 2 | bitwise AND | a & b |
| andnw | 2 | 2 | 2 | bitwise AND NOT | a & (~b) |
| avgsw | 2 | 2 | 2 | signed average | (a + b + 1)>>1 |
| avguw | 2 | 2 | 2 | unsigned average | (a + b + 1)>>1 |
| cmpeqw | 2 | 2 | 2 | compare equal | (a == b) ? (~0) : 0 |
| cmpgtsw | 2 | 2 | 2 | compare greater than | (a > b) ? (~0) : 0 |
| copyw | 2 | 2 | copy | a | |
| div255w | 2 | 2 | divide by 255 | a/255 | |
| loadw | 2 | 2 | load from memory | array[i] | |
| loadpw | 2 | 2 | load parameter or constant | scalar | |
| maxsw | 2 | 2 | 2 | signed maximum | (a > b) ? a : b |
| maxuw | 2 | 2 | 2 | unsigned maximum | (a > b) ? a : b |
| minsw | 2 | 2 | 2 | signed minimum | (a < b) ? a : b |
| minuw | 2 | 2 | 2 | unsigned minimum | (a < b) ? a : b |
| mullw | 2 | 2 | 2 | low bits of multiply | a * b |
| mulhsw | 2 | 2 | 2 | high bits of signed multiply | (a * b) >> 8 |
| mulhuw | 2 | 2 | 2 | high bits of unsigned multiply | (a * b) >> 8 |
| orw | 2 | 2 | 2 | bitwise or | a | b |
| shlw | 2 | 2 | 2S | shift left | a << b |
| shrsw | 2 | 2 | 2S | signed shift right | a >> b |
| shruw | 2 | 2 | 2S | unsigned shift right | a >> b |
| signw | 2 | 2 | sign | sign(a) | |
| storew | 2 | 2 | store to memory | special | |
| subw | 2 | 2 | 2 | subtract | a - b |
| subssw | 2 | 2 | 2 | subtract with signed saturate | clamp(a - b) |
| subusw | 2 | 2 | 2 | subtract with unsigned saturate | clamp(a - b) |
| xorw | 2 | 2 | 2 | bitwise XOR | a ^ b |
| absl | 4 | 4 | absolute value | (a < 0) ? -a : a | |
| addl | 4 | 4 | 4 | add | a + b |
| addssl | 4 | 4 | 4 | add with signed saturate | clamp(a + b) |
| addusl | 4 | 4 | 4 | add with unsigned saturate | clamp(a + b) |
| andl | 4 | 4 | 4 | bitwise AND | a & b |
| andnl | 4 | 4 | 4 | bitwise AND NOT | a & (~b) |
| avgsl | 4 | 4 | 4 | signed average | (a + b + 1)>>1 |
| avgul | 4 | 4 | 4 | unsigned average | (a + b + 1)>>1 |
| cmpeql | 4 | 4 | 4 | compare equal | (a == b) ? (~0) : 0 |
| cmpgtsl | 4 | 4 | 4 | compare greater than | (a > b) ? (~0) : 0 |
| copyl | 4 | 4 | copy | a | |
| loadl | 4 | 4 | load from memory | array[i] | |
| loadpl | 4 | 4 | load parameter or constant | scalar | |
| maxsl | 4 | 4 | 4 | signed maximum | (a > b) ? a : b |
| maxul | 4 | 4 | 4 | unsigned maximum | (a > b) ? a : b |
| minsl | 4 | 4 | 4 | signed minimum | (a < b) ? a : b |
| minul | 4 | 4 | 4 | unsigned minimum | (a < b) ? a : b |
| orl | 4 | 4 | 4 | bitwise or | a | b |
| shll | 4 | 4 | 4S | shift left | a << b |
| shrsl | 4 | 4 | 4S | signed shift right | a >> b |
| shrul | 4 | 4 | 4S | unsigned shift right | a >> b |
| signl | 4 | 4 | sign | sign(a) | |
| storel | 4 | 4 | store to memory | special | |
| subl | 4 | 4 | 4 | subtract | a - b |
| subssl | 4 | 4 | 4 | subtract with signed saturate | clamp(a - b) |
| subusl | 4 | 4 | 4 | subtract with unsigned saturate | clamp(a - b) |
| xorl | 4 | 4 | 4 | bitwise XOR | a ^ b |
| loadq | 8 | 8 | load from memory | array[i] | |
| storeq | 8 | 8 | store to memory | special | |
| splatw3q | 8 | 8 | duplicates high 16-bits to lower 48 bits | special | |
| convsbw | 2 | 1 | convert signed | a | |
| convubw | 2 | 1 | convert unsigned | a | |
| splatbw | 2 | 1 | duplicates 8 bits to both halfs of 16 bits | special | |
| splatbl | 4 | 1 | duplicates 8 bits to all parts of 32 bits | special | |
| convswl | 4 | 2 | convert signed | a | |
| convuwl | 4 | 2 | convert unsigned | a | |
| convslq | 8 | 4 | signed convert | a | |
| convulq | 8 | 4 | unsigned convert | a | |
| convwb | 1 | 2 | convert | a | |
| convhwb | 1 | 2 | shift and convert | a>>8 | |
| convssswb | 1 | 2 | convert signed to signed with saturation | clamp(a) | |
| convsuswb | 1 | 2 | convert signed to unsigned with saturation | clamp(a) | |
| convuuswb | 1 | 2 | convert unsigned to unsigned with saturation | clamp(a) | |
| convlw | 2 | 4 | convert | a | |
| convhlw | 2 | 4 | shift and convert | a>>16 | |
| convssslw | 2 | 4 | convert signed to signed with saturation | clamp(a) | |
| convql | 4 | 8 | convert | a | |
| mulsbw | 2 | 1 | 1 | multiply signed | a * b |
| mulubw | 2 | 1 | 1 | multiply unsigned | a * b |
| mulswl | 4 | 2 | 2 | multiply signed | a * b |
| muluwl | 4 | 2 | 2 | multiply unsigned | a * b |
| accl | 4 | 4 | accumulate | += a | |
| swapw | 2 | 2 | endianness swap | special | |
| swapl | 4 | 4 | endianness swap | special | |
| select0wb | 1 | 2 | select first half | special | |
| select1wb | 1 | 2 | select second half | special | |
| select0lw | 2 | 4 | select first half | special | |
| select1lw | 2 | 4 | select second half | special | |
| mergewl | 4 | 2 | 2 | merge halves | special |
| mergebw | 2 | 1 | 1 | merge halves | special |
| splitlw | 2 | 4 | split first/second words | special | |
| splitwb | 1 | 2 | split first/second bytes | special | |
| addf | 4 | 4 | 4 | add | a + b |
| subf | 4 | 4 | 4 | subtract | a - b |
| mulf | 4 | 4 | 4 | multiply | a * b |
| maxf | 4 | 4 | 4 | maximum | max(a,b) |
| minf | 4 | 4 | 4 | minimum | min(a,b) |
| cmpeqf | 4 | 4 | 4 | compare equal | (a == b) ? (~0) : 0 |
| convfl | 4 | 4 | convert float point to integer | a | |
| convlf | 4 | 4 | convert integer to floating point | a |
Wednesday, December 29, 2010
stuck, back to PySoy
I've been working on a streaming XML parser for Python, but need a break. At this point there's no way Concordance is getting out Jan 1st, but certainly by the end of Winter.
Our libsoy migration process PySoy got pretty far. We were migrating from Pyrex to Genie, essentially moving the core engine from PyObject to GObject to remove Python dependency in game clients and enable further multicore processing on both client and servers. Much of the rendering area of the engine has been migrated, but the process has been held up in two areas;
First, while libsoy is in pretty good shape, we still lack Python bindings - aka PySoy itself, which is what we intend games to be written and run with. Our original plan to use GObject Introspection failed in a horrible mess that I've documented in previous postings, we've looked at using SWIG and even building our own bindings generation with little measurable success. In order to get us moving forward again I'm going to just drop out some .c templates and write the custom wrapper classes by hand. The time it'd take to write and maintain these cannot possibly be greater than the time we've wasted talking about a more elegant solution that only exists conceptually.
When GObject Introspection reaches a state of even remote maturity, where it can offer a Pythonic API, we'll look at it again. We'd even help get it there if the current GIR developers would just document the .gir XML schema or typelib format so we wouldn't have to refer to their source code as the sole definition of these.
Second is our physics code. As I've posted, ODE worked for us in the past but has numerous issues with packaging for various Linux distros (and poor features, slow, and extremely difficult to port to mobile devices). We attempted to migrate to Bullet but this burned us out - virtually no work has gone into that in the past 6 months. We're all pretty frustrated with Bullet's haphazard API (whereas ODE is fairly clean) and the C++ only API doesn't play well with GObject (or anything other than c++ for that matter). Bullet's C API is minimal at best.
When it comes right down to it, the biggest barrier we face with physics is processing power on mobile devices, an issue that using Bullet would not solve. Most of the devices we're interested in include ARM6/7 processors from Qualcomm or TI. Many do not include a FPU (floating point unit), but they all seem to offer a fairly powerful DSP used extensively for processing multimedia. We do not, however, want to rewrite and maintain our physics processing for each platform.
A solution I've come up with is to write our physics, greatly simplified from even what ODE offers, using Orc. It's yet another metalanguage (first Pyrex, then Vala/Genie, now this..), but the successor to liboil (which we and much of the Gnome community use) and already supports many interesting platforms.
My plan is to first migrate our liboil-based YUV-RGB conversion code to Orc to get my feet wet, then implement a greatly simplified collision system using it, and expect the next release (or two) to still use ODE for at least rigid body physics with the plan to eventually replace even that with our own physics solver. It should be much faster, and the same Orc code we write now should be able to compile to DSP code for Android handsets and other mobile devices in the future.
Orc already supports ARM Cortex (NEON), so if we were to finish this work today we'd be able to run PySoy clients on more modern Android handsets without touching DSP code. DSP support in Orc would also be very useful for future hardware for PySoy game servers.
While we'd all really like to get the next PySoy release out ASAP, we'd also like to avoid rewriting the engine again down the road.
Our libsoy migration process PySoy got pretty far. We were migrating from Pyrex to Genie, essentially moving the core engine from PyObject to GObject to remove Python dependency in game clients and enable further multicore processing on both client and servers. Much of the rendering area of the engine has been migrated, but the process has been held up in two areas;
First, while libsoy is in pretty good shape, we still lack Python bindings - aka PySoy itself, which is what we intend games to be written and run with. Our original plan to use GObject Introspection failed in a horrible mess that I've documented in previous postings, we've looked at using SWIG and even building our own bindings generation with little measurable success. In order to get us moving forward again I'm going to just drop out some .c templates and write the custom wrapper classes by hand. The time it'd take to write and maintain these cannot possibly be greater than the time we've wasted talking about a more elegant solution that only exists conceptually.
When GObject Introspection reaches a state of even remote maturity, where it can offer a Pythonic API, we'll look at it again. We'd even help get it there if the current GIR developers would just document the .gir XML schema or typelib format so we wouldn't have to refer to their source code as the sole definition of these.
Second is our physics code. As I've posted, ODE worked for us in the past but has numerous issues with packaging for various Linux distros (and poor features, slow, and extremely difficult to port to mobile devices). We attempted to migrate to Bullet but this burned us out - virtually no work has gone into that in the past 6 months. We're all pretty frustrated with Bullet's haphazard API (whereas ODE is fairly clean) and the C++ only API doesn't play well with GObject (or anything other than c++ for that matter). Bullet's C API is minimal at best.
When it comes right down to it, the biggest barrier we face with physics is processing power on mobile devices, an issue that using Bullet would not solve. Most of the devices we're interested in include ARM6/7 processors from Qualcomm or TI. Many do not include a FPU (floating point unit), but they all seem to offer a fairly powerful DSP used extensively for processing multimedia. We do not, however, want to rewrite and maintain our physics processing for each platform.
A solution I've come up with is to write our physics, greatly simplified from even what ODE offers, using Orc. It's yet another metalanguage (first Pyrex, then Vala/Genie, now this..), but the successor to liboil (which we and much of the Gnome community use) and already supports many interesting platforms.
My plan is to first migrate our liboil-based YUV-RGB conversion code to Orc to get my feet wet, then implement a greatly simplified collision system using it, and expect the next release (or two) to still use ODE for at least rigid body physics with the plan to eventually replace even that with our own physics solver. It should be much faster, and the same Orc code we write now should be able to compile to DSP code for Android handsets and other mobile devices in the future.
Orc already supports ARM Cortex (NEON), so if we were to finish this work today we'd be able to run PySoy clients on more modern Android handsets without touching DSP code. DSP support in Orc would also be very useful for future hardware for PySoy game servers.
While we'd all really like to get the next PySoy release out ASAP, we'd also like to avoid rewriting the engine again down the road.
Wednesday, December 08, 2010
XML parsing in Python
Its been a couple months, so I'm going to give a brief update on what I've been working on.
Concordance is getting close to release, I plan to have the first release (0.1) out January 1st. More on this toward the end of December.
One of the roadblocks I've hit (again and again) is the lack of a decent XML parsing package for Python. The standard library is a shame when it comes to XML; at least four different modules (expat, sax, dom, etree) to choose from and none of them support even XPath. The most popular option, etree (or ElementTree), cannot even process an XML file with the namespace prefix intact.
There's lxml, which offers an etree-compatible API and fixes many of ElementTree's major faults (namespace prefix preservation, xpath/xslt support) but still cannot handle stream processing and, due to ElementTree's API, does not expose multiple text nodes broken up by a child element such as "<div>first string <br/> second string</div>".
To support XMPP streams we need to use expat or sax to handle the stream event-by-event, since the full XML document is only available once the root element closes at the end of the stream, but the direct children of the root element (what we call "stanzas" in XMPP) need to be processed as complete objects. While we may be able to hack something together using lxml, it would likely be less work than to implement a new XML parsing package. As long as the resulting API doesn't diverge very greatly etree the work necessary to switch should be minimal.
Beside this I've been working on a host of different packages around Concordance, from getting a javascript BOSH/XMPP library together to getting distutils2 ready for Python 3. I've even managed to ship a pitiful little serial library for Python 3, PyTTY that we're using to interface with some Arduinos.
Concordance is getting close to release, I plan to have the first release (0.1) out January 1st. More on this toward the end of December.
One of the roadblocks I've hit (again and again) is the lack of a decent XML parsing package for Python. The standard library is a shame when it comes to XML; at least four different modules (expat, sax, dom, etree) to choose from and none of them support even XPath. The most popular option, etree (or ElementTree), cannot even process an XML file with the namespace prefix intact.
There's lxml, which offers an etree-compatible API and fixes many of ElementTree's major faults (namespace prefix preservation, xpath/xslt support) but still cannot handle stream processing and, due to ElementTree's API, does not expose multiple text nodes broken up by a child element such as "<div>first string <br/> second string</div>".
To support XMPP streams we need to use expat or sax to handle the stream event-by-event, since the full XML document is only available once the root element closes at the end of the stream, but the direct children of the root element (what we call "stanzas" in XMPP) need to be processed as complete objects. While we may be able to hack something together using lxml, it would likely be less work than to implement a new XML parsing package. As long as the resulting API doesn't diverge very greatly etree the work necessary to switch should be minimal.
Beside this I've been working on a host of different packages around Concordance, from getting a javascript BOSH/XMPP library together to getting distutils2 ready for Python 3. I've even managed to ship a pitiful little serial library for Python 3, PyTTY that we're using to interface with some Arduinos.
Friday, July 02, 2010
Diaspora: the $200k scam on free software
After a month of work, its bluntly clear the community has been scammed for $200k.
Four NYU students posted a video and solicited funding on the kickstarter website, promising to build an open source, pro-privacy alternative to Facebook. The community response was astonishing. Its too bad the people pledging didn't research these students, for example, checking to see if they have Ohloh pages with their past contributions to other projects.
The first and most severe, after a month of work, they have posted that they have running code but will not allow the community (including the people who paid for it) to view until its "complete". This is commonly referred to as the Cathedral model. People have invested a great deal of trust in them to develop the code they have promised to, the least they could offer is read only access to their VCS. Instead they posted a video demonstrating it, which poses the second problem.
Their claim to use the "latest and greatest in web standards" fell flat on its face in their choice of method to post their video. Instead of using HTML/5 <video>, which they could have even used Ogg with the Cortado java player as fallback for MSIE and Safari, they used a flash player. Perhaps they don't realize Flash is not a web standard, or that HTML/5 has built-in video support, in either case its a really bad sign.
The most blunt demonstration of their lack of skill and commitment is in their use of websockets. At this point, websockets is still in heavy draft state with the few browsers/servers implementing it all running a slightly different, incompatible variants, none of which will be the final draft which is (at least) months off. Moreso, they propose to implement chat and gaming over their own websockets-based protocol rather than using existing standards such as XMPP.
XMPP is being used by a wide number of social network services already including Google, Livejournal, and Facebook. Its an established, widely implemented standard recognized by the IETF and the community at large for federated, real-time exchange of chat and presence (status). Beyond chat, XMPP has support for PubSub, Data Forms, and Ad-Hoc Commands. There is virtually nothing Facebook, or Diaspora, aims to accomplish that cannot be implemented using these existing standards.
What these four students are building is lock-in, if they finish at all. Instead of replacing the walled gardens of Facebook and other proprietary social network sites, they aim to build a network of little sites running the same software, only able to federate with each other, and only able to be extended with their custom APIs. These sites will not federate with other distributed social network efforts being built such as movim and concordance.
Hopefully the community will learn a lesson from this. Pledge money to experienced developers with demonstratable code, not charismatic students pledging to work on free software only if they're paid to do it.
Four NYU students posted a video and solicited funding on the kickstarter website, promising to build an open source, pro-privacy alternative to Facebook. The community response was astonishing. Its too bad the people pledging didn't research these students, for example, checking to see if they have Ohloh pages with their past contributions to other projects.
The first and most severe, after a month of work, they have posted that they have running code but will not allow the community (including the people who paid for it) to view until its "complete". This is commonly referred to as the Cathedral model. People have invested a great deal of trust in them to develop the code they have promised to, the least they could offer is read only access to their VCS. Instead they posted a video demonstrating it, which poses the second problem.
Their claim to use the "latest and greatest in web standards" fell flat on its face in their choice of method to post their video. Instead of using HTML/5 <video>, which they could have even used Ogg with the Cortado java player as fallback for MSIE and Safari, they used a flash player. Perhaps they don't realize Flash is not a web standard, or that HTML/5 has built-in video support, in either case its a really bad sign.
The most blunt demonstration of their lack of skill and commitment is in their use of websockets. At this point, websockets is still in heavy draft state with the few browsers/servers implementing it all running a slightly different, incompatible variants, none of which will be the final draft which is (at least) months off. Moreso, they propose to implement chat and gaming over their own websockets-based protocol rather than using existing standards such as XMPP.
XMPP is being used by a wide number of social network services already including Google, Livejournal, and Facebook. Its an established, widely implemented standard recognized by the IETF and the community at large for federated, real-time exchange of chat and presence (status). Beyond chat, XMPP has support for PubSub, Data Forms, and Ad-Hoc Commands. There is virtually nothing Facebook, or Diaspora, aims to accomplish that cannot be implemented using these existing standards.
What these four students are building is lock-in, if they finish at all. Instead of replacing the walled gardens of Facebook and other proprietary social network sites, they aim to build a network of little sites running the same software, only able to federate with each other, and only able to be extended with their custom APIs. These sites will not federate with other distributed social network efforts being built such as movim and concordance.
Hopefully the community will learn a lesson from this. Pledge money to experienced developers with demonstratable code, not charismatic students pledging to work on free software only if they're paid to do it.
Monday, June 28, 2010
Using Android without Google (part 1)
As magicfab on identi.ca requested, I'm starting a journal of my experience using Android without Google's proprietary apps.
I own an HTC Eris phone which originally came with Android 1.5. A few months ago a firmware was released by a 3rd party which allows users to gain root on their phone (about time) and I've been experimenting with this since. My primary interests were removing HTC's "Sense" UI so I could use a different soft keyboard and removing the backdoors HTC, Verizon, and Google installed in the phone.
After some disturbing problems with the VanillaDroid firmware, I switched to CyanogenEris 3.0 last week. Not only did this new build not come with HTC's Sense, thanks to Google hitting them with a threatening DMCA letter, the firmware came without Google's apps either. This includes GMail, GTalk, Google Maps and Market.
Everything works on the firmware except for USB tethering (which is a feature of the firmware). I started by using the browser to download the Barcode Scanner and then used that to install other apps via QR code:
The Foursquare app I was previously using refuses to install (likely because it depends on Google Maps app) and I haven't figured out how to install the dvorak keyboard add-on to soft keyboard.
I have the standard Android Contacts app, but without Google it does not sync to my Google contacts. I plan to remove my contacts from Google and write a small app that provides ContactsProvider2 via XMPP in an effort to decentralize and federate contact syncing.
I found the web version of GMail superior to the old app, though I do miss the Google Maps app (there are various OpenStreetMaps alternatives but they don't have the same features). The biggest thing I miss is the GTalk app, though its pitiful as far as XMPP clients go and we should be able to do a lot better.
Most importantly, my phone is now 99% free software with only a few of HTC's drivers left to be reverse engineered. Even for an advertising company Google is really stretching the truth when they call Android "open source" - but it should be an attainable goal.
I own an HTC Eris phone which originally came with Android 1.5. A few months ago a firmware was released by a 3rd party which allows users to gain root on their phone (about time) and I've been experimenting with this since. My primary interests were removing HTC's "Sense" UI so I could use a different soft keyboard and removing the backdoors HTC, Verizon, and Google installed in the phone.
After some disturbing problems with the VanillaDroid firmware, I switched to CyanogenEris 3.0 last week. Not only did this new build not come with HTC's Sense, thanks to Google hitting them with a threatening DMCA letter, the firmware came without Google's apps either. This includes GMail, GTalk, Google Maps and Market.
Everything works on the firmware except for USB tethering (which is a feature of the firmware). I started by using the browser to download the Barcode Scanner and then used that to install other apps via QR code:
The Foursquare app I was previously using refuses to install (likely because it depends on Google Maps app) and I haven't figured out how to install the dvorak keyboard add-on to soft keyboard.
I have the standard Android Contacts app, but without Google it does not sync to my Google contacts. I plan to remove my contacts from Google and write a small app that provides ContactsProvider2 via XMPP in an effort to decentralize and federate contact syncing.
I found the web version of GMail superior to the old app, though I do miss the Google Maps app (there are various OpenStreetMaps alternatives but they don't have the same features). The biggest thing I miss is the GTalk app, though its pitiful as far as XMPP clients go and we should be able to do a lot better.
Most importantly, my phone is now 99% free software with only a few of HTC's drivers left to be reverse engineered. Even for an advertising company Google is really stretching the truth when they call Android "open source" - but it should be an attainable goal.
Sunday, June 27, 2010
concordance progress
While dredging through framework API design, its sometimes difficult to sense how close you are to finish. Tonight the Concordance "pure Python" redesign reached a critical point - being able to add an extension and respond to an XMPP <iq> stanza:
That's a correct response to an XMPP Ping request, one of the simplest XMPP extensions. Many of these basic extensions will be combined into a single extension module and a ready-made Service class, this is just proof that it works.
>>> import concordance
>>> import concordance.extensions.ping
>>> s = concordance.Service('host')
>>> s.append(concordance.extensions.ping.Ping)
>>> import xml.etree.ElementTree as et
>>> st = et.fromstring('<iq from="arc@host" id="f00" to="host" type="get"><ping xmlns="urn:xmpp:ping" /></iq>')
>>> s.stream(st)
<iq from="host" id="f00" to="arc@host" type="result" />
>>>
That's a correct response to an XMPP Ping request, one of the simplest XMPP extensions. Many of these basic extensions will be combined into a single extension module and a ready-made Service class, this is just proof that it works.
Wednesday, June 16, 2010
Concordance stickers ordered
I just ordered 1000 2" x 2" vinyl laptop stickers with the Concordance logo on white. They should arrive by the end of the month. Contributors can send me their mailing address to receive one along with one of the new Python 3 stickers from the PSF.
New PySoy stickers will be made as soon as we have a new logo.
Subscribe to:
Posts (Atom)