common Orc opcodes

I've been going through liboil's 0.3 source to rewrite the oil_yuv2rgbx_sub2_u8 function we use for Theora decoding to Orc pseudo-assembly code.

Because the Orc opcode documentation splits opcode description and processor support between two tables, for reference I wrote a quick Python script to build a table of Orc opcodes common to SSE (x86), Altivec (PPC/Cell), and NEON (Arm Cortex) processors.

Here's that table for reference, at least until I put the time to format it for a wiki:

opcode	dst	src1	src2	description	pseudo code
absb	1	1		absolute value	(a < 0) ? -a : a
addb	1	1	1	add	a + b
addssb	1	1	1	add with signed saturate	clamp(a + b)
addusb	1	1	1	add with unsigned saturate	clamp(a + b)
andb	1	1	1	bitwise AND	a & b
andnb	1	1	1	bitwise AND NOT	a & (~b)
avgsb	1	1	1	signed average	(a + b + 1)>>1
avgub	1	1	1	unsigned average	(a + b + 1)>>1
cmpeqb	1	1	1	compare equal	(a == b) ? (~0) : 0
cmpgtsb	1	1	1	compare greater than	(a > b) ? (~0) : 0
copyb	1	1		copy	a
loadb	1	1		load from memory	array[i]
loadpb	1	1		load parameter or constant	scalar
maxsb	1	1	1	signed maximum	(a > b) ? a : b
maxub	1	1	1	unsigned maximum	(a > b) ? a : b
minsb	1	1	1	signed minimum	(a < b) ? a : b
minub	1	1	1	unsigned minimum	(a < b) ? a : b
mullb	1	1	1	low bits of multiply	a * b
mulhsb	1	1	1	high bits of signed multiply	(a * b) >> 8
mulhub	1	1	1	high bits of unsigned multiply	(a * b) >> 8
orb	1	1	1	bitwise or	a | b
shlb	1	1	1S	shift left	a << b
shrsb	1	1	1S	signed shift right	a >> b
shrub	1	1	1S	unsigned shift right	a >> b
signb	1	1		sign	sign(a)
storeb	1	1		store to memory	special
subb	1	1	1	subtract	a - b
subssb	1	1	1	subtract with signed saturate	clamp(a - b)
subusb	1	1	1	subtract with unsigned saturate	clamp(a - b)
xorb	1	1	1	bitwise XOR	a ^ b
absw	2	2		absolute value	(a < 0) ? -a : a
addw	2	2	2	add	a + b
addssw	2	2	2	add with signed saturate	clamp(a + b)
addusw	2	2	2	add with unsigned saturate	clamp(a + b)
andw	2	2	2	bitwise AND	a & b
andnw	2	2	2	bitwise AND NOT	a & (~b)
avgsw	2	2	2	signed average	(a + b + 1)>>1
avguw	2	2	2	unsigned average	(a + b + 1)>>1
cmpeqw	2	2	2	compare equal	(a == b) ? (~0) : 0
cmpgtsw	2	2	2	compare greater than	(a > b) ? (~0) : 0
copyw	2	2		copy	a
div255w	2	2		divide by 255	a/255
loadw	2	2		load from memory	array[i]
loadpw	2	2		load parameter or constant	scalar
maxsw	2	2	2	signed maximum	(a > b) ? a : b
maxuw	2	2	2	unsigned maximum	(a > b) ? a : b
minsw	2	2	2	signed minimum	(a < b) ? a : b
minuw	2	2	2	unsigned minimum	(a < b) ? a : b
mullw	2	2	2	low bits of multiply	a * b
mulhsw	2	2	2	high bits of signed multiply	(a * b) >> 8
mulhuw	2	2	2	high bits of unsigned multiply	(a * b) >> 8
orw	2	2	2	bitwise or	a | b
shlw	2	2	2S	shift left	a << b
shrsw	2	2	2S	signed shift right	a >> b
shruw	2	2	2S	unsigned shift right	a >> b
signw	2	2		sign	sign(a)
storew	2	2		store to memory	special
subw	2	2	2	subtract	a - b
subssw	2	2	2	subtract with signed saturate	clamp(a - b)
subusw	2	2	2	subtract with unsigned saturate	clamp(a - b)
xorw	2	2	2	bitwise XOR	a ^ b
absl	4	4		absolute value	(a < 0) ? -a : a
addl	4	4	4	add	a + b
addssl	4	4	4	add with signed saturate	clamp(a + b)
addusl	4	4	4	add with unsigned saturate	clamp(a + b)
andl	4	4	4	bitwise AND	a & b
andnl	4	4	4	bitwise AND NOT	a & (~b)
avgsl	4	4	4	signed average	(a + b + 1)>>1
avgul	4	4	4	unsigned average	(a + b + 1)>>1
cmpeql	4	4	4	compare equal	(a == b) ? (~0) : 0
cmpgtsl	4	4	4	compare greater than	(a > b) ? (~0) : 0
copyl	4	4		copy	a
loadl	4	4		load from memory	array[i]
loadpl	4	4		load parameter or constant	scalar
maxsl	4	4	4	signed maximum	(a > b) ? a : b
maxul	4	4	4	unsigned maximum	(a > b) ? a : b
minsl	4	4	4	signed minimum	(a < b) ? a : b
minul	4	4	4	unsigned minimum	(a < b) ? a : b
orl	4	4	4	bitwise or	a | b
shll	4	4	4S	shift left	a << b
shrsl	4	4	4S	signed shift right	a >> b
shrul	4	4	4S	unsigned shift right	a >> b
signl	4	4		sign	sign(a)
storel	4	4		store to memory	special
subl	4	4	4	subtract	a - b
subssl	4	4	4	subtract with signed saturate	clamp(a - b)
subusl	4	4	4	subtract with unsigned saturate	clamp(a - b)
xorl	4	4	4	bitwise XOR	a ^ b
loadq	8	8		load from memory	array[i]
storeq	8	8		store to memory	special
splatw3q	8	8		duplicates high 16-bits to lower 48 bits	special
convsbw	2	1		convert signed	a
convubw	2	1		convert unsigned	a
splatbw	2	1		duplicates 8 bits to both halfs of 16 bits	special
splatbl	4	1		duplicates 8 bits to all parts of 32 bits	special
convswl	4	2		convert signed	a
convuwl	4	2		convert unsigned	a
convslq	8	4		signed convert	a
convulq	8	4		unsigned convert	a
convwb	1	2		convert	a
convhwb	1	2		shift and convert	a>>8
convssswb	1	2		convert signed to signed with saturation	clamp(a)
convsuswb	1	2		convert signed to unsigned with saturation	clamp(a)
convuuswb	1	2		convert unsigned to unsigned with saturation	clamp(a)
convlw	2	4		convert	a
convhlw	2	4		shift and convert	a>>16
convssslw	2	4		convert signed to signed with saturation	clamp(a)
convql	4	8		convert	a
mulsbw	2	1	1	multiply signed	a * b
mulubw	2	1	1	multiply unsigned	a * b
mulswl	4	2	2	multiply signed	a * b
muluwl	4	2	2	multiply unsigned	a * b
accl	4	4		accumulate	+= a
swapw	2	2		endianness swap	special
swapl	4	4		endianness swap	special
select0wb	1	2		select first half	special
select1wb	1	2		select second half	special
select0lw	2	4		select first half	special
select1lw	2	4		select second half	special
mergewl	4	2	2	merge halves	special
mergebw	2	1	1	merge halves	special
splitlw	2	4		split first/second words	special
splitwb	1	2		split first/second bytes	special
addf	4	4	4	add	a + b
subf	4	4	4	subtract	a - b
mulf	4	4	4	multiply	a * b
maxf	4	4	4	maximum	max(a,b)
minf	4	4	4	minimum	min(a,b)
cmpeqf	4	4	4	compare equal	(a == b) ? (~0) : 0
convfl	4	4		convert float point to integer	a
convlf	4	4		convert integer to floating point	a

stuck, back to PySoy

I've been working on a streaming XML parser for Python, but need a break. At this point there's no way Concordance is getting out Jan 1st, but certainly by the end of Winter.

Our libsoy migration process PySoy got pretty far. We were migrating from Pyrex to Genie, essentially moving the core engine from PyObject to GObject to remove Python dependency in game clients and enable further multicore processing on both client and servers. Much of the rendering area of the engine has been migrated, but the process has been held up in two areas;

First, while libsoy is in pretty good shape, we still lack Python bindings - aka PySoy itself, which is what we intend games to be written and run with. Our original plan to use GObject Introspection failed in a horrible mess that I've documented in previous postings, we've looked at using SWIG and even building our own bindings generation with little measurable success. In order to get us moving forward again I'm going to just drop out some .c templates and write the custom wrapper classes by hand. The time it'd take to write and maintain these cannot possibly be greater than the time we've wasted talking about a more elegant solution that only exists conceptually.

When GObject Introspection reaches a state of even remote maturity, where it can offer a Pythonic API, we'll look at it again. We'd even help get it there if the current GIR developers would just document the .gir XML schema or typelib format so we wouldn't have to refer to their source code as the sole definition of these.

Second is our physics code. As I've posted, ODE worked for us in the past but has numerous issues with packaging for various Linux distros (and poor features, slow, and extremely difficult to port to mobile devices). We attempted to migrate to Bullet but this burned us out - virtually no work has gone into that in the past 6 months. We're all pretty frustrated with Bullet's haphazard API (whereas ODE is fairly clean) and the C++ only API doesn't play well with GObject (or anything other than c++ for that matter). Bullet's C API is minimal at best.

When it comes right down to it, the biggest barrier we face with physics is processing power on mobile devices, an issue that using Bullet would not solve. Most of the devices we're interested in include ARM6/7 processors from Qualcomm or TI. Many do not include a FPU (floating point unit), but they all seem to offer a fairly powerful DSP used extensively for processing multimedia. We do not, however, want to rewrite and maintain our physics processing for each platform.

A solution I've come up with is to write our physics, greatly simplified from even what ODE offers, using Orc. It's yet another metalanguage (first Pyrex, then Vala/Genie, now this..), but the successor to liboil (which we and much of the Gnome community use) and already supports many interesting platforms.

My plan is to first migrate our liboil-based YUV-RGB conversion code to Orc to get my feet wet, then implement a greatly simplified collision system using it, and expect the next release (or two) to still use ODE for at least rigid body physics with the plan to eventually replace even that with our own physics solver. It should be much faster, and the same Orc code we write now should be able to compile to DSP code for Android handsets and other mobile devices in the future.

Orc already supports ARM Cortex (NEON), so if we were to finish this work today we'd be able to run PySoy clients on more modern Android handsets without touching DSP code. DSP support in Orc would also be very useful for future hardware for PySoy game servers.

While we'd all really like to get the next PySoy release out ASAP, we'd also like to avoid rewriting the engine again down the road.

XML parsing in Python

Its been a couple months, so I'm going to give a brief update on what I've been working on.

Concordance is getting close to release, I plan to have the first release (0.1) out January 1st. More on this toward the end of December.

One of the roadblocks I've hit (again and again) is the lack of a decent XML parsing package for Python. The standard library is a shame when it comes to XML; at least four different modules (expat, sax, dom, etree) to choose from and none of them support even XPath. The most popular option, etree (or ElementTree), cannot even process an XML file with the namespace prefix intact.

There's lxml, which offers an etree-compatible API and fixes many of ElementTree's major faults (namespace prefix preservation, xpath/xslt support) but still cannot handle stream processing and, due to ElementTree's API, does not expose multiple text nodes broken up by a child element such as "<div>first string <br/> second string</div>".

To support XMPP streams we need to use expat or sax to handle the stream event-by-event, since the full XML document is only available once the root element closes at the end of the stream, but the direct children of the root element (what we call "stanzas" in XMPP) need to be processed as complete objects. While we may be able to hack something together using lxml, it would likely be less work than to implement a new XML parsing package. As long as the resulting API doesn't diverge very greatly etree the work necessary to switch should be minimal.

Beside this I've been working on a host of different packages around Concordance, from getting a javascript BOSH/XMPP library together to getting distutils2 ready for Python 3. I've even managed to ship a pitiful little serial library for Python 3, PyTTY that we're using to interface with some Arduinos.

Diaspora: the $200k scam on free software

After a month of work, its bluntly clear the community has been scammed for $200k.

Four NYU students posted a video and solicited funding on the kickstarter website, promising to build an open source, pro-privacy alternative to Facebook. The community response was astonishing. Its too bad the people pledging didn't research these students, for example, checking to see if they have Ohloh pages with their past contributions to other projects.

The first and most severe, after a month of work, they have posted that they have running code but will not allow the community (including the people who paid for it) to view until its "complete". This is commonly referred to as the Cathedral model. People have invested a great deal of trust in them to develop the code they have promised to, the least they could offer is read only access to their VCS. Instead they posted a video demonstrating it, which poses the second problem.

Their claim to use the "latest and greatest in web standards" fell flat on its face in their choice of method to post their video. Instead of using HTML/5 <video>, which they could have even used Ogg with the Cortado java player as fallback for MSIE and Safari, they used a flash player. Perhaps they don't realize Flash is not a web standard, or that HTML/5 has built-in video support, in either case its a really bad sign.

The most blunt demonstration of their lack of skill and commitment is in their use of websockets. At this point, websockets is still in heavy draft state with the few browsers/servers implementing it all running a slightly different, incompatible variants, none of which will be the final draft which is (at least) months off. Moreso, they propose to implement chat and gaming over their own websockets-based protocol rather than using existing standards such as XMPP.

XMPP is being used by a wide number of social network services already including Google, Livejournal, and Facebook. Its an established, widely implemented standard recognized by the IETF and the community at large for federated, real-time exchange of chat and presence (status). Beyond chat, XMPP has support for PubSub, Data Forms, and Ad-Hoc Commands. There is virtually nothing Facebook, or Diaspora, aims to accomplish that cannot be implemented using these existing standards.

What these four students are building is lock-in, if they finish at all. Instead of replacing the walled gardens of Facebook and other proprietary social network sites, they aim to build a network of little sites running the same software, only able to federate with each other, and only able to be extended with their custom APIs. These sites will not federate with other distributed social network efforts being built such as movim and concordance.

Hopefully the community will learn a lesson from this. Pledge money to experienced developers with demonstratable code, not charismatic students pledging to work on free software only if they're paid to do it.

Using Android without Google (part 1)

As magicfab on identi.ca requested, I'm starting a journal of my experience using Android without Google's proprietary apps.

I own an HTC Eris phone which originally came with Android 1.5. A few months ago a firmware was released by a 3rd party which allows users to gain root on their phone (about time) and I've been experimenting with this since. My primary interests were removing HTC's "Sense" UI so I could use a different soft keyboard and removing the backdoors HTC, Verizon, and Google installed in the phone.

After some disturbing problems with the VanillaDroid firmware, I switched to CyanogenEris 3.0 last week. Not only did this new build not come with HTC's Sense, thanks to Google hitting them with a threatening DMCA letter, the firmware came without Google's apps either. This includes GMail, GTalk, Google Maps and Market.

Everything works on the firmware except for USB tethering (which is a feature of the firmware). I started by using the browser to download the Barcode Scanner and then used that to install other apps via QR code:

• wifi tether for root
• k9 mail
• connectbot (ssh client)
• mustard (identi.ca client)

The Foursquare app I was previously using refuses to install (likely because it depends on Google Maps app) and I haven't figured out how to install the dvorak keyboard add-on to soft keyboard.

I have the standard Android Contacts app, but without Google it does not sync to my Google contacts. I plan to remove my contacts from Google and write a small app that provides ContactsProvider2 via XMPP in an effort to decentralize and federate contact syncing.

I found the web version of GMail superior to the old app, though I do miss the Google Maps app (there are various OpenStreetMaps alternatives but they don't have the same features). The biggest thing I miss is the GTalk app, though its pitiful as far as XMPP clients go and we should be able to do a lot better.

Most importantly, my phone is now 99% free software with only a few of HTC's drivers left to be reverse engineered. Even for an advertising company Google is really stretching the truth when they call Android "open source" - but it should be an attainable goal.

concordance progress

While dredging through framework API design, its sometimes difficult to sense how close you are to finish. Tonight the Concordance "pure Python" redesign reached a critical point - being able to add an extension and respond to an XMPP <iq> stanza:

>>> import concordance
>>> import concordance.extensions.ping
>>> s = concordance.Service('host')
>>> s.append(concordance.extensions.ping.Ping)
>>> import xml.etree.ElementTree as et
>>> st = et.fromstring('<iq from="arc@host" id="f00" to="host" type="get"><ping xmlns="urn:xmpp:ping" /></iq>')
>>> s.stream(st)
<iq from="host" id="f00" to="arc@host" type="result" />
>>> 

That's a correct response to an XMPP Ping request, one of the simplest XMPP extensions. Many of these basic extensions will be combined into a single extension module and a ready-made Service class, this is just proof that it works.

Concordance stickers ordered

I just ordered 1000 2" x 2" vinyl laptop stickers with the Concordance logo on white. They should arrive by the end of the month. Contributors can send me their mailing address to receive one along with one of the new Python 3 stickers from the PSF.

New PySoy stickers will be made as soon as we have a new logo.

Happy 4th Birthday, PySoy

On this day, June 6th 2006, the PySoy Project was born.

We've come a long way from the small homebrew Pyrex-based game engine we started from. We've built a solid development team, spun off numerous related projects, contributed hundreds of bug reports and patches for libraries and applications, and grown close to the Python, Gnome, and XMPP communities.

Currently halfway through our most ambitious endeavour yet, moving our codebase to the GObject model and preparing the engine and related projects for cloud gaming, there's a bright future ahead for this tiny project. While its sometimes frustrating that we're still not to 1.0 release, looking back to what we started from four years ago its clear we've made tremendous progress.

Here's to another exciting year!

A killer Android app

I've been running the Vanilladroid firmware for my HTC Eris for a few weeks now. Beside the phone spontaneously rebooting itself every few hours, its run great, and with built in wifi tethering.

Tonight it proved potentially fatal for one young man. While walking home it appeared like I interupted a mugging near HacDC. The victim was being kicked by a man who ran away as I got closer. He was laying sprawled out on the concrete, face right on the sidewalk and shirt ripped off. I pulled out my phone, hit the icon for Dialer One (which comes with Vanilladroid), dialed 911, dial key, and... nothing. Dial key again, nothing. Checked, it says 911 in the number box, but the app isn't dialing it. I closed the app and tried again, notta. I can't believe it would refuse to dial 911 so I hard reset the phone and try again, still the dialer just sits there as I press the dial key.

I finally had to go on the web, look up the number for Dc Police, and have them transfer me to 911.

I take partial blame for not looking into this app and switching it out sooner. Hopefully Google removes it before someone dies while trying to call for help.

praise for Gnome's GIO

We've been (finally) getting Concordance's C code refactored into a standalone XMPP server called GNation.

Concordance used a good deal of GLib code already, but we've gone a bit further in using the GIO library for managing sockets and connections. In doing so, we were able to reduce roughly 160 lines of code for opening and monitoring listening sockets into 12 and I suspect further reductions as we continue. All of the IPv4 vs IPv6 support was cleanly abstracted, adding a socket listener to the MainLoop was done automatically, and the GIO documentation was of the same high quality we've come to expect from GLib.

Major thanks to the hard work that has gone into this!

How free is Google's VP8 (WebM)?

Thank you Google for using your wealth to purchase On2 corporation and release VP8 to the world. However, there appears to be one glaring flaw in the patent license you offer which makes it incompatible with free software licensing: it appears to only cover the VP8 bitstream itself, not improvements to it made by the community.

When On2 previously released VP3.2 to the Xiph foundation and the world, the patent license explicitly included derivatives (improvements) to VP3.2 including Ogg Theora. This was a critical step in transitioning the proprietary VP3.2 codec to the open codec Theora is today.

On2's "VP" codecs included little extensibility, why should they as On2's revenue stream depended on releasing (and offering a new license for) a new improved codec version every year or two. You cannot use their VP5 codec to decode VP6, you need to get a VP6 codec for that. In contrast, Vorbis is ten years old now and is still being improved on.

The large body of work undertaken primarily by Tim and Ralph at Xiph before Theora 1.0 could be released was extending the bitstream in a losslessly upgradable manner from VP3.2 to include the fields necessary to continually improve Theora over the next decade and beyond. You can see this by playing a VP3.2 video next to a Theora 1.1 video, and even more so in the Theora 1.2-beta9 comparisons, and they've barely scratched the surface still.

For comparison, On2 corporation's patent license for VP3.2 can be found in the Theora license file vs Google's VP8 bitstream license. As I'm not a lawyer, I would love to hear SFLC's evaluation of this in their next Oggcast.

stability, always a few versions behind

Several of our projects use waf and Vala, which I normally have nothing but good things to say about, but this is an exception. At the moment, we're stuck working with outdated versions of both.

Tonight I tried upgrading from Waf 1.5.10 to 1.5.16 and was unable to get any of our packages to build with it. As it turns out, it doesn't seem to like glib (maybe the "-" or number in the package name?) and fails to record its cflags and libs to pass to gcc during building. If 1.5.10 didn't have a few annoying bugs of its own or if we could patch and "compile" 1.5.10 we'd be fine, but even this version we had to use the "binary" (compressed Python script) from their website as we couldn't get a Python 3 version to build from source.

Despite all this, it is very nice to have a small, clean build system that both supports Genie and runs on Python 3. The small workarounds we've had to do are worth the savings in headache and frustration autotools/scons/etc would cause us.

With Vala, apparently nobody did even a casual check to ensure the Genie parser was still working properly before releasing valac 0.8.0 (now up to 0.8.1), and a basic example from the language's frontpage (command line argument parsing) compiles but fails to link. For now, we're all forced to stick with valac 0.7.10 until the problem is solved or someone suggests a workaround. Its unfortunate a few of our devs run Ubuntu 10.04 which doesn't offer valac 0.7.x as a package.

Good news is I'm almost finished debugging expat.vapi (Expat bindings for Vala/Genie) and David is sprinting ahead with xml config file parsing with it. By this weekend we could have XMPP routing with the new server being branched off from Concordance-XMPP.

dusting off pyogg2

I've started work on updating an old CMS I worked on between 2002-2005 that includes a non-linear Ogg editor and Icecast stream queue management. Part of this is updating the CMS to Python 3, including pyogg2 which it uses.

Since I no longer have working ssh keys for committing to svn.xiph.org, I copied the pyogg2 repository to hg.concordance-xmpp.org and resumed the work I left off in 2005. Pyogg2 was in fairly stable shape but lacked documentation, utilities for working with common codecs, and only had a few examples to learn it with. I'm already given its build system a major cleanup (now just "setup.py build" like any other Python package) and almost finished the Python 3 PyModule code.

I have a lot more work to do on this; the CMS needs to be updated for modern web standards (HTML/5), the homebrew MySQL ORM with SQLAlchemy, and changing from the old CGI Apache configuration setup to CherryPy.

Leaving the Ubuntu project

In response to the recent news that Canonical has stated peddling MPEG-4 patent licenses to Ubuntu OEMs, I am resigning as member of the Ubuntu project.

I have been uncomfortable by Canonical misusing the Ubuntu trademark with their proprietary Ubuntu One service, which they have recently added a proprietary music store to for MP3-only downloads fully integrated with Ubuntu's version of Rhythmbox, and bundling an ever-increasing amount of proprietary drivers with Ubuntu.

I still find the Ubuntu community an outstanding model for how the free software community should organize. Strong grassroots presence through the local community teams, democratically organized leadership, and most importantly the Ubuntu Code of Conduct are models other projects should look to emulate.

I'm certain that if put to a vote, the Ubuntu community would release the Ubuntu One server code as AGPLv3 and discontinue business practices in contrast with free software values conducted in its name, but the only way we can vote against Canonical is with our feet.

As I serve in a leadership position in a local community team, I'll be upholding my agreement with the Ubuntu Code of Conduct by gracefully stepping down after finding another team member to take my place and migrating any remaining accounts off my @ubuntu.com email address.

server migration complete

I have finished migrating all the websites I host from maat (at ServerBeach) to nut (at Hurricane Electric). Tonight I finished wiping all the passwords, ssl certs, and other private data from the old server before its shut down tomorrow.

Maat was an Athlon XP 2200+ server with 1gig ram, 80gig HD, 10meg up/down, 2TB/month transfer, and 1 IPv4 address. Nut is a Atom 330 (1.6ghz 64-bit dual core) with 2gig ram, 500gig HD, 100meg up/down, unlimited bandwidth, 6 usable IPv4 addresses and a /64 IPv6 subnet - and costs less.

I shed a tear working through maat's filesystem checking for any remaining files to be copied and finding the original Gentoo stage3 tarball used to install it: /stage3-i686-2007.0.tar. Tomorrow ServerBeach will likely disconnect maat and retire it as after 3 years the server is out of maintainability.

My experience so far with Hurricane Electric is that their remote admin tools are limited to "remote hands" (call a technician 24/7 to reboot a server, type in commands, etc), where ServerBeach offered a really nice web portal for rebooting, accessing a rescue image, etc without having to call anyone. Of course, those tools haven't been working correctly since December. The technicians at Hurricane Electric have demonstrated a far higher level of expertise than anyone I've spoken to at ServerBeach.

Unlimited 100meg bandwidth, IPv6, and 10x bandwidth (HTML/5 Ogg audio/video hosting) is the real seller. In retrospect, I'm really glad ServerBeach had the FUBAR downtime in December prompting me to look for alternatives, this is a much better deal.

working on gtypes

We've been moving forward on the PySoy rewrite using Genie in faith that a Python 3 binding for GObject would come about.

At PyCon this weekend I've brainstormed with Kirk McDonald, author of PyD which serves a similar role to use D types/objects in Python, and several other authors of FFIs for Python. A rough strategy is coming together to solve this in a dynamic manner.

The first part of the solution is libvala; using a small part of its functionality to parse .vapi and .gir files for existing GObject libraries such as GTK and Cairo as well as any library written in Vala or Genie. This gets us the C API for the library in all it's object oriented goodness, and with that we can dynamically generate bindings using Python's ctypes.

The second part is a GObject type in Python which supports subclassing in Python. Kirk detailed how this is done in PyD; each generated GObject type gets a subtype with all of its virtual methods all running Python methods which call the superclass's C method by default. When a Python subclass of a GObject class is made this GObject subclass is used allowing Python to override any of its methods and pass objects of the subclass back to GObject where they'll perform as expected. Not simple or easy, but its a strategy.

The end result is dynamic Python 3 bindings for GTK as well as PySoy and all the other tools we work on.

I'm organizing a sprint for this at PyCon. If anyone has ideas on this please comment, or if you'd like to help (whether at PyCon 2010 or not) get in touch with me.

Epson Artisan 810 on Linux (Ubuntu)

We purchased a new printer for the house last night to replace the windows-only Dell printer that ran out of ink (and would cost almost as much as a new printer for replacements).

First, some advice: plan ahead, order online, don't buy from Staples. We paid $170 for this when Epson is offering them direct for $130 with free shipping. If we were not in need of immediate replacement we could have saved $40 and waited a week. Further, the rep at the store lied when he told us the printer was going to the back to get a unit "fresh off the truck", it was a unit another customer had returned with ink spilled and dried on the inside and some of the packaging torn open. Buyer beware. Otherwise, they seemed to clean it up and repackaged it well. I have half a mind to order the printer online and return this one within 14 days for refund to save the $40.

The printer itself is fairly impressive. Two paper trays (letter/legal plus photo/cardstock) and a built-in CD printing tray that slides out when you select CD printing, scanner with auto-document feeder, and built in 10/100 and wifi so we don't need to depend on any computer in the house to properly "share" the printer. All I had to do is select our access point on the front controls and give it a static IP on our router.

Ubuntu 9.10 detects the printer over the network in seconds and I was printing the Ubuntu test page in under a minute, so more than half the systems in the house were easy. Windows setup was not hard, but time consuming as loads of bundled software installed from the CD. The only complaint I have is the color seems a little off; printing the same image in both Linux and Windows shows a fairly radical difference in quality. The prints from Linux are still much better than printing from Windows to our older printer.

Next I tried to use the scanner function. Using GIMP to launch SANE (Scanner Access Now Easy), the printer was automatically detected and let me select either the flatbed or auto-document feeder. Both options work fine until I start the scan, at which point the Epson Artisan 810 locks up and needs to be rebooted.

There is a scan function from the front panel, not supporting anywhere near the scanner's full resolution and storing in JPEG format with obvious artifacts around text/etc, but at least we don't need to use a Windows system for basic scans until SANE is fixed. Since I have a stand-alone Epson scanner that works beautifully this isn't a huge issue for me, though it would be great to be able to use the auto-document feeder from Linux and be able to put this older scanner into storage.

soy.windows.Window rendering, GLEW headache

I'm seeing "red" now with our Windows, in more ways than one. The rendering code for soy.windows.Windows is to the point that, by changing the glClear color, we're seeing red in the window rather than black. This is a big step forward given that porting to use GDK is a big part of the libsoy migration.

On the other side of "red" is anger over GLEW. I really wish I knew of a decent, mature alternative to GLEW. The project is so immature that, though one of the maintainers is a Debian developer, they do not provide the pkgconfig file for it. Without this file, which provides Waf with the location of the library and it's headers, we would have to write a search script to verify that it's installed and find it's location. None of the major distros I looked at provide the pkgconfig on their own.

It's also at least partially redundant with the functionality of gtkglext which I believe should be able to replace GLEW entirely.

libsoy running on OSX - briefly

For the first time in 3.5 years, we had soy.windows.Window running on OSX. Just the basics; opening, (re)setting the title, resizing - but it worked. This is thanks to GDK, the windowing and event backend to GTK+, running on Mac OSX via native Quartz backend (not X11).

Shortly after we moved the window code to a background thread and it broke. The code runs fine on GNU/Linux and Windows, but on OSX all we get are white rectangles where the windows would normally be rendered. The rectangles even resize properly, and window events are getting processed by our code, just nothing rendered inside (not even the window title or buttons). I filed this bug and hope the problem to be solved quickly.

The problem could be as simple as a missing NSGraphicsContext.flushGraphics call (which is only needed when rendering is done in a non-main thread). If the bug hasn't been fixed by our next release I'll look into patching this myself.

language mixing

After some work to update our ODE bindings for Vala, we discussed Bullet again last night and returned to the conclusion to drop further work on ODE and focus on Bullet for the next release.

It quickly became clear that the C bindings for Bullet are utter horse crap; praise for the effort for whoever wrote them, and I by no means intend to disparage your ability or intentions, but they're horribly incomplete and usable for little more than the two demo apps included in the source. Moreso, the C bindings are "higher level" than the C++ library, performing several steps automatically (such as calculating body inertia from mass and shape) which our 3d game engine needs more direct control over. I first wrote an alternative (ABI compatible) C header for it, and then started to rewrite it entirely.

I decided that if we're going to write our own Bullet compatibility layer, we may as well go all the way in wrapping it with GObject. The code is simple enough, but defining GObject in C involves a lot of boilerplate - in some cases more than the code itself. To get through this and retain my sanity, I started work this morning on a C++ code generator that takes the relevant functions and wraps them in the standard boilerplate.

The format I'm using is Genie-like for class and method definition, then C++ within dumped directly to the target .cpp. The code generator will generate the .vapi as it works and compile our wrapper as a static library which libsoy can link to directly.

Kudos again to Waf for making this rather eccentric use of their build system dead simple and clean.

libsoy building separately now

We've been working on the "libsoy" branch of PySoy for some time, a split of the compiled code to a GObject-based C library with Python bindings plus the pure Python code constituting PySoy itself.

Previously we were struggling to build this with Python's distutils system, which didn't support building system-wide C libraries nor using valac to do so. Jon and I started looking into Waf last week. It's pretty impressive; Python based (and works with Python 3), simple, intuitive, does everything we need with only a few dozen lines of script. Last night we setup libsoy in it's own repository and building (just a very basic soy.windows for now) on it's own.

I think we're settled to work with Genie, a comparison between the syntax of soy.windows in Vala vs in Genie makes that choice simple. The only complaint I have with Genie is needing to escape newlines inside struct {}'s (ie, line 32-35), but reportedly this is being fixed in an upcoming release to be more Python-like (ignoring newlines while inside () or {}).

After a bit of searching (through source code, I wish Waf had better API reference in it's docs) I found the gir= option for Vala building, and another hour later both soy-1.0.gir and soy-1.0.typelib were being generated, except both were empty save the "soy" namespace:


<?xml version="1.0"?>
<repository version="1.0" xmlns="http://www.gtk.org/introspection/core/1.0" xmlns:c="http://www.gtk.org/introspection/c/1.0" xmlns:glib="http://www.gtk.org/introspection/glib/1.0">
  <package name="soy"/>
  <namespace name="soy" version="1.0" c:prefix="soy">
  </namespace>
</repository>


I've hit this before with trying to generate GObject Introspection from distutils but chalked it up to not building it correctly. As it turns out, this is a known bug in GIR and a previously unknown bug in valac to fail silently.

I'm still hoping the GIR developers will come to their senses and support nested namespaces, but given that they've stubbornly rejected this for months already I'm afraid we'll have to build a cross-language bindings builder based on Vala/Genie instead of helping to update PyGI for Python 3.

state of Vala IDEs

Wasted quite a few hours today trying to get a Vala/Genie IDE working.

VTG is basically broken, crashing at random. It had problems before this; having a fairly rigid concept of a "project", not supporting non-autotools build systems or libraries very easily, not supporting Genie, and the symbol mapping having odd quirks, so this is not a huge loss.

A newer Vala plugin for gEdit, Valencia, looks promising but doesn't support recent Vala versions and attempting to build it raised pages of errors.

Anjuta has always looked interesting, but I've never managed to get the Anjuta Vala plugin to compile. At present, it doesn't appear to work with Gnome 2.28.

I don't use Eclipse or Mono, so the final option is Valide. To my surprise, it's been packaged for Ubuntu and works out of the box. It supports Genie, it seems to have features on-par with VTG and the other IDEs, and works with the latest version of Vala.

My only complaint so far is the new project wizard doesn't list the AGPLv3 as a license option, but then none of the Vala IDE options do yet.