Arc's Soy Machine

GCI Low Hanging Fruit

2011-12-16T12:06:00.002-05:00

I have to admit I'm pretty surprised that at halfway through Google Code-In 2011 only about a quarter of the tasks we posted for the first half have been completed or in-progress. Further, most of the tasks thus far haven't been coding tasks.

I'm going to publish here a list of what we consider "low hanging fruit"; coding tasks which are fairly easy to get started with. If you're a student age 13-17 these would be an easy way to earn a Google tshirt and some cash. All of these tasks deal with OpenGL rendering, usually just arrays for points and lines/triangles which connect the points. There's a lot of examples in the code already for this and numerous tutorials on the web (such as NeHe).

Simple Rendering: We have 2 new tasks listed for rendering simple models; Camera and Light. Camera is very simple, while Light can be as simple or complex as you want to make it.
Shapes Rendering: We have 3 tasks for rendering shapes, Box, Room, and Sphere. Any of these could be knocked out in a few hours even without prior OpenGL knowledge.
Joints Rendering: We have 6 new tasks up for rendering joints; Ball, Fixed, Hinge, Piston, Slider, and Universal. Joints (soy.joints) connect two bodies such that they can only move in respect to each other in a certain way, such as a door hinge or piston. These are all documented with graphic depictions. This is slightly more complex than the simple rendering tasks (above) in that there's two pieces to each joint and they can be rendered as either wireframe or solid (with provided materials).

As always, we're on IRC if a student wants to discuss these or other tasks.

PyTTY 0.3

2011-12-14T00:27:00.000-05:00

Continuing my annual end-of-year coding sprint, I just released PyTTY 0.3.

PyTTY is a Python serial communication package I started last year after a friend said he couldn't use Python 3 yet because pyserial wasn't ported. The point of writing this was to show him that he didn't need an ancient, bloated, poorly-maintained package to do something as simple as serial communication.

What I wrote over an afternoon turned out to be a little over 100 lines of fairly useful code which I've since used in quite a few microcontroller projects (eg, Arduino). Its by no means complete, the only setting is baud rate and there's no Windows support, but its done everything I've needed it to over the last year. The only problem that's been reported is poor documentation which this release aims to fix. It includes a short code example ("pydoc pytty.TTY") which runs on both legacy Python and Python 3.

PyTTY 0.3 is under 135 lines of pure Python and relies only on the Python standard library. If there's a feature you need which this doesn't have either email me a patch or a feature request so I can add it. The Mercurial repository is http://hg.pytty.org/pytty.

NodeTree 0.2 Released

2011-12-08T00:50:00.001-05:00

I just shipped NodeTree 0.2.

This version will not parse an XML stream. All it contains are some basic types representing XML nodes such as Comment, Document, and Element. As promised, text is also handled as a node but uses standard Python strings (UTF-8 strings/bytes and unicode). These should all be fairly intuitive to use.

The magic is the XML data is being managed in C using a libxml2 DOM tree but accessed through a Pythonic object-oriented API. For example, in DOM each node may have exactly one parent - in NodeTree a node may be added to any number of parents with a separate DOM node and context for each.

I started this project because the existing XML packages for Python proved too difficult to use with XMPP. Fritzy's SleekXMPP uses lxml but had to jump through several hoops to get stream parsing to work, looking over his work I certainly didn't want to repeat it with Concordance-XMPP.

Beyond this the leading XML API for Python, ElementTree, includes several unfortunate design decisions that make it frustrating to use in the best cases and unusable in others. A full list of why can be left for another time, but the difference to NodeTree can be described in their names - ElementTree is a tree of XML Element nodes with other kinds of nodes either silently dropped, mangled, or made available in bizarre ways (eg, .text and .tail). In contrast, NodeTree provides XML data as a tree of nodes starting with the Document node and includes comment and text nodes in its tree. I plan to provide 100% XML 1.0 support in a future release while maintaining a clean, simple, and intuitive API.

Storing XML data in libxml2 DOM format gives us a few advantages over other XML libraries. First, we'll have XPath, XInclude, and XSLT available without having to convert the data between formats. Second, Python objects only need to be created for nodes Python wants a reference to so when we get to parsing data this will happen much faster and with less memory.

At version 0.2 NodeTree is still in its infancy but some of its API can be demonstrated. Here's a short example:


Python 3.2.2 (default, Oct  3 2011, 00:20:58) 
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import nodetree
>>> doc = nodetree.Document()
>>> doc.append(nodetree.Comment(' Start '))
>>> doc.append(nodetree.Element('data'))
>>> doc.append(nodetree.Comment(' Fini '))
>>> doc[1].attributes['thing'] = 'normal'
>>> doc[1].append(nodetree.Element('record'))
>>> doc[1][0].append('First') 
>>> doc[1].append(nodetree.Element('record'))
>>> doc[1][1].append('Second')
>>> doc
<?xml version="1.0"?>
<!-- Start -->
<data thing="normal">
  <record>First</record>
  <record>Second</record>
</data>
<!-- Fini -->

NodeTree 0.2 is tested to work with Python 2.6, 2.7, 3.1, 3.2, and 3.3-pre. The next release is intended to support basic file and stream parsing.

XMPP on the web

2011-12-03T20:55:00.001-05:00

A short thread on G+ has prompted this longer sharing of my vision for XMPP on the web.

For XMPP use on a website we currently have BOSH and, in an extreme-alpha state, XMPP over websockets. The advantage of websockets is obvious, BOSH is a high overhead protocol that we'd all rather not have to use, however both have the same problem: you either must share your login credentials (and thus access to your account) with every website you use a single account with, or must create a new account (JID) for every XMPP-based website you use.

Oauth2 for XMPP might be a partial solution to this by requiring your authorization through a central identity site and using the resulting token for logging in, however, you're still opening yourself up to the 3rd party website accessing your roster, sending spam messages on your behalf, and potentially worse. All this really gives you is the ability to later disable access to websites who misuse your account.

This is the crux of the issue: when using an Javascript library provided by a website and using a proxy provided by that website, whether BOSH, websockets, or otherwise, you're giving that website unlimited access to your account. I have not seen a workable proposal to solve this and until this is solved XMPP cannot see widespread use on the web.

I'm proposing that we solve this by putting XMPP in the browser, either directly or through a plugin. Expose a standard javascript API for allowing websites to use an XMPP connection along with a security model which gives users control as to what a website is allowed to use their connection for. Ie, if a script on a website wants access to their roster the user will be prompted for it, if they want to join a MUC room display a standard prompt for that. Browsers can have multiple XMPP sessions at once and allow the user to select which account they'd like to use with an XMPP-enabled site.

This is just some early ideas, I'm nowhere near implementing this though I think the conversation would be useful to get started.

OpenGL ES support complete

2011-11-28T01:56:00.001-05:00

The experimental branch, where the OpenGL ES migration was being done, has just been closed and merged into the default branch of PySoy.

Thanks to Steve Anton, one of our Google Code-In 2011 students, for some of the last bits of work to complete the merge. We are now one step closer to mobile support! If you're a student ages 13-17 and would like to earn a Google tshirt and some cash by helping us with Android support, sign up for Google Code-In and claim this task.

Google Code-In 2011 is Open

2011-11-21T04:45:00.001-05:00

Google Code-In has officially begun!

From today through January 16th students age 13-17 can earn up to $500 working on small tasks for software projects such as MoinMoin, SymPy, and PySoy!

Tasks include coding, documentation, graphic design, video production, testing, translation, research, public speaking, and many other kinds of challenges of varying difficulty.

Completing just one task earns a student a Google tshirt. Every 3 tasks they complete earns them $100, and the 10 top students worldwide will earn an all-expense paid trip to Mountain View, CA to receive an award at Google.

PySoy has over 75 tasks offered for the first half of the program and another 75-100 will be made available December 16th. Our mentors are on Freenode channel #PySoy ready to help students start earning their tshirt and cash today.

Sign up today and get started!

Rugby season nearly over, getting back to work

2011-11-09T23:09:00.001-05:00

Wow its been a long time.

We wrapped up Google's Summer of Code 2011 in August. The Python Software Foundation did wonderfully overall, for PySoy 6 of our 7 students passed. A great year overall - thanks to all the mentors and students!

Rugby has been a life changer for me. My first game was in September, after floating in and out of practice for years and training pretty heavily since April. No serious injuries, but no shortage of pain; I've frequently needed to sleep in a reclining chair to keep blood from pooling in my shoulders and nurse bruised ribs, dislocated fingers and toes, shin splints, and pulled muscles everywhere. All so worth it.

These guys are like family to me. I know it sounds sappy, but I've come to trust the men in my pack with my life - in a way we all do every time we bind onto each other a scrum. Its not that big of an adjustment culturally though due to the large number of programmers, lawyers, and IT professionals on the team. When you work behind a desk all day its nice to balance it out with a physically intensive training in the evening and games on Saturday.

The climax of the season was Hellfest October 29th in Dallas, TX. Washington Renegades brought our B-side to compete and returned with the 1st place trophy. My teammate Jimbo has more pics on his blog of the tournament, I was wearing #23 as tighthead prop.

We have two more games this season before we settle in for the Winter and indoor off-season training at the gym. A group of us plan to do a 8-week program run by a professional rugby player this Winter to get ready for the Spring season and the Bingham Cup 2012 in Manchester UK next June.

Today the PySoy project was accepted to Google Code-In. We've got a number of student tasks lined up, with many more being worked on for the first batch set to release in less than two weeks. Interested students should hop on Freenode (#PySoy) and get oriented before the program starts so they're ready to jump right into their first task!

Transcoding FLAC to Ogg Vorbis

2011-07-21T11:31:00.004-04:00

Last night I hit a dilemma; a very old CD I ripped to FLAC and now can't find was playable only on certain players, not my Android phone (despite FLAC support) and behaving strange on many desktop players.

Usually I just use something like oggenc -q 4 *.flac since vorbis-tools supports FLAC as a source format (and preserves metadata like artist, title, etc). Strangely, oggenc didn't recognize the files in this album.

GStreamer to the rescue;

for file in *.flac; do gst-launch-0.10 filesrc location="$file" ! decodebin ! audioconvert ! vorbisenc quality=0.4 ! oggmux ! filesink location="$file.ogg"; done;

Even though its much slower and more complicated, this should have done the trick. It didn't, and ogginfo showed that the Ogg muxer in GStreamer has some issues;


WARNING: granulepos in stream 1 decreases from 218558 to 205632
WARNING: granulepos in stream 1 decreases from 666174 to 655680
WARNING: granulepos in stream 1 decreases from 3150206 to 3142208
WARNING: granulepos in stream 1 decreases from 3605054 to 3597376
WARNING: granulepos in stream 1 decreases from 3828670 to 3822400
WARNING: granulepos in stream 1 decreases from 4741630 to 4733312
WARNING: granulepos in stream 1 decreases from 5636158 to 5630784
WARNING: granulepos in stream 1 decreases from 5858494 to 5854208
WARNING: granulepos in stream 1 decreases from 6096126 to 6090560
WARNING: granulepos in stream 1 decreases from 6317758 to 6314048
WARNING: granulepos in stream 1 decreases from 7668798 to 7665088
WARNING: granulepos in stream 1 decreases from 7902718 to 7898240

So with GStreamer not an option, I looked at the original FLAC files and found that whatever encoder I used added ID3 tags (which are not part of the FLAC spec). A quick ID3 removal command stripped these out so oggenc would recognize the files and work;


find . -name "*.flac" -exec id3v2 --delete-all {} \;
oggenc -q 4 *.flac

Done.

Blowing up the silos (hypothermia)

2011-06-21T08:41:00.002-04:00

Its a common belief that fat is fat, whether we eat it or whether its stored in our bodies. Of course this is naive, our bodies are far more complex than skin balloons packed with nutrients. There are two kinds of fat tissue in our body;

The "fuel silos" are called White Adipose Tissue (WAT) made of simple cells; their membrane has a few receptors on it that trigger the cell to store or release energy, the nucleus is small and pressed against the cell membrane, and inside one big lipid droplet that shrinks or swells as needed. When dieting your pancreas releases Glucagon to trigger your fat to release their reserves. When your body releases insulin (which it does after almost every meal to some degree) your WAT cells swell back up. This is why its so hard to keep weight off.

Then there's the factory fat cells, Brown Adipose Tissue (BAT) which are much more complex; they contain many smaller lipid droplets and the enzimes needed to burn the energy right in them. They're especially important in infants because they're too small to shiver to get warm, so their "brown fat" burns their fat immediately to keep their body temperature regulated. Adults maintain a good amount of it in our upper back and neck, though strangely lean adults have more BAT than obese adults.

When our bodies become hypothermic, say by taking an ice water bath, BAT thermogenesis is activated depleting those cells of stored energy to produce body heat. Shivering also releases more energy to raise heat. While the sound of a ice water bath may sound horrifying, its actually not that bad after you've been running, decrease water temp slowly, and can even be meditative.

The goal is to destroy WAT (White Adipose Tissue) - not just slowly deplete it but leave the cells in a state ready to swell back up every time you eat, but actually reducing the number of white fat cells in your body for permanent weight loss.

Blowing up the silos (morning)

2011-06-16T08:55:00.003-04:00

An awesome holiday gift last December was a book, "The 4-Hour Body", a 570 page hardcover hacking guide to the human body. On Dec 20 I started the "Rapid Fat-Loss" program within with great success - in less than three months I lost nearly 5% body fat, with just with a few changes to my diet and no additional exercise.

I got looser with the diet as the weather got warmer and I started exercising more, then partially tore my archilles tendon between two different accidents confining me to a reclining chair for two months. Of course I gained all the fat back; the fuel silos (fat cells) were still there in the same number, just emptied a bit, and all of them sending out messages to adjust insulin and metabolism so that they could refill.

You can see on my bodybuilding.com profile the raise starting in mid-March. However, the experience wasn't a lost cause - I have a better idea of what works and learned a number of new recipes for making it work easier, since the book only mentions veganism and doesn't provide much in the way of meal ideas (I'll share one below).

I started training June 2nd for Fall Rugby season - something I'm using as a focus point to stay motivated. I'm calling this program "Blowing up the Silos" - not just slowly using the fuel within, but mushroom clouds on the horizon. As I sit here at 9am writing this my 640 calorie breakfast burns like napalm through my veins, barely able to sit still long enough to type. When I finish with this I'm headed out for wind sprints just to burn off the excess energy.

Upon waking I immediately downed 500mg Phosphatidylserine, 500mg L-Carnitine, 300mg Alpha-Lipoic Acid, 500mg Maca root powder with a tall glass of water with 1tbsp lemon juice in it, then green tea made from 8g whole leaf supplemented with 3 grams of powdered Cissus Quadrangularis.

From there I have 30 minutes to cook breakfast while drinking the tea and chewing on a handful (24g) of cashews. Breakfast was 1 tbsp (14g) refined coconut oil in a frying pan and a batter made from 1.5tsp (4 grams) ground cinnimon and 92g gram (chic pea) flour soaked in an airtight container with 1 cup of water the night before. Makes something not even remotely akin to a pancake, but edible. Net carbs 50 grams, 26 grams protein, and a whopping 640 calories.

Along with breakfast was another handful of pills; daily cal-mag-zinc supplement (needed to replenish whats lost from green tea causing you to urinate all day), 500mg chromium, 60mg CoQ 10, and a multivitamin.

The magic in this combination is how these various elements cascade each other. I would explain in detail here how each of these do amazing things on their own, but I can't sit any longer. I'll post more on other meals and adjustments I'm making later.

making an Android/iPhone/iPad home screen icon for your website

2011-05-31T16:21:00.007-04:00

One of the cool features of new phones and tablets is the ability to drop a website bookmark as an icon on the home screen, as if it was a normal App for that device. Many websites provide icons which are automatically used when a user bookmarks them, however, the tools and resources for doing this are not immediately obvious. I just spent Memorial Day figuring this out and will share what I found here.

First, design your logo. I use Inkscape so I can later scale the icon for tshirts and printed material, but any tool will do as long as you end up with a square flattened 512x512 PNG. Open this in GIMP.

Next download this GIMP template and open it in GIMP. Copy/paste your icon into the already selected region of the template, it should "just work". Anchor your selection and merge visible layers for your 512x512 icon.

Now that you have your glossy icon with rounded edges, scale and save the image three times for 57x57, 72x72, and 114x114 as follows:


  apple-touch-icon-57x57-precomposed.png
  apple-touch-icon-72x72-precomposed.png
  apple-touch-icon-114x114-precomposed.png

Also save your original (pre-template) logo at 57x57 as apple-touch-icon.png. These four files should cover iPhone, iPad, Android, and Blackberry. You may also want to save either the original logo or the glossy version at 16x16 for favicon.ico if you have not done so already.

Upload these files to your web server saved in the root directory of your website (like robots.txt) and add the following template to the head of your website:


  <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" />
  <link rel="apple-touch-icon-precomposed" sizes="114x114" href="http://yourdomain.com/apple-touch-icon-114x114-precomposed.png" />
  <link rel="apple-touch-icon-precomposed" sizes="72x72" href="http://yourdomain.com/apple-touch-icon-72x72-precomposed.png" />
  <link rel="apple-touch-icon-precomposed" href="http://yourdomain.com/apple-touch-icon-57x57-precomposed.png" />
  <link rel="apple-touch-icon" href="http://yourdomain.com/apple-touch-icon.png" />

That should take care of virtually every device that supports this feature. Test it out on your Android phone and, if you have any friends in the Cult of Mac, ask if you can check this on the half dozen iOS devices they own.

If you want to really encourage iPhone/iPad users to take advantage of your new, shiny icon to bookmark your webapp on their Home screen, check out this javascript library called Mobile Bookmark Bubble.

Of course you should also make sure your site works well on handheld devices, but that's a topic for another time.

Have fun!

Nodetree has Document and root Element

2011-05-24T02:02:00.002-04:00

I think an example will explain the emerging API far better than I can.

You can replicate this with the code from http://hg.concordance-xmpp.org/nodetree, though its still very early the code is completely documented in its current state.

Python 3.2 (r32:88445, Feb 28 2011, 00:50:14) 
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import nodetree
>>> doc = nodetree.Document()
>>> foo = nodetree.Element('foo')
>>> doc.root = foo
>>> len(doc)
1
>>> print(doc)
<?xml version="1.0"?>
<foo/>

>>> bar = nodetree.Element('bar')
>>> doc.root = bar
>>> print(doc)
<?xml version="1.0"?>
<bar/>

>>> print(foo)
<foo/>
>>> print(bar)
<bar/>
>>> print(doc[0])
<bar/>
>>>

Nodetree design

2011-05-21T15:48:00.003-04:00

I've spent the last two weeks charting out Nodetree, a XML data binding package for Python 3.

I'd rather not reiterate the multitude of reasons for this, but since they always get asked, I'll summarize this with needing a stream parser that supports xpath/xslt and passing chunks of an XML stream in Python, something that ElementTree and lxml do not support. SleekXMPP employs a fairly ingenious hack on lxml to achieve this, but I'd rather spend some time doing it right from the start for Concordance.

Since XPath and XSLT are best supported by libxml2 than any other free software library, and I'd rather not spend the next few months writing an implementation from scratch, the data must exist as a libxml2 DOM tree at the point of processing.

You can get a libxml2 DOM tree either by parsing a file (which gives you little control, it parses the entire file one-shot) or build the tree node by node. libxml2 also supports processing an XML stream through its SAX interface, node by node, which gives us the flexibility needed to pop and/or parse nodes from the stream as its being parsed while still having all the other tools we need by generating a DOM tree of the segments as we process them.

The problem here is that Nodetree isn't a DOM interface, its a Pythonic XML data binding interface, so such things as adding the same element to two documents (or being able to create a new document using a piece of another document while both are in memory) is a challenge. Even if we were not using libxml2/DOM, for XPath to work correctly every node must have a clear hierarchy which falls apart when your context is a node used in the same document three times and in two different documents.

One of the benefits of working on a wide variety of projects is reusing clever solutions used on one project for something completely different. For example, I recently implemented PySoy's atomic API whereas Python objects are created on the fly for underlying data structures and could be attached to multiple points of data (see ColorCanvas.py for example). This allows multiple data points to be updated with one step, and saves us from doing silly things like storing (ie) an object for every pixel in an image.

I'm implementing almost exactly the same mechanism for Nodetree, it'll limit XPath searches run from Python somewhat but should be fully XML compliant and Pythonic, something (IMHO) nobody has managed to do yet.

Rocket Fuel

2011-04-29T06:52:00.003-04:00

The gym I went to up in Manchester, NH was next door to an asian market. I knew that coconut oil is about half Lauric Acid, which is an extremely digestible medium chain saturated fat which burns body fat as part of its metabolism, and the asian market had it in water bottles that looked like they were packed with crisco. I was desperate for more energy for workout, so a disgusting squirt in the mouth meant a shot of energy lasting about 45 minutes for weight lifting.

This evolved over time into what I call Rocket Fuel which I still use for workouts, both far more effective than coconut oil alone and much better tasting. Here's the recipe;

4 whole dates
3 tbsp coconut oil
2 tbsp agave nectar
half a lemon of zest
1 tsp lemon juice
1/4 tsp sea salt
24 grams powdered gunpowder tea

Start out by warming all ingredients to room temperature, especially the coconut oil which melts at around 25C (77F) to a clearish yellow liquid. Melting temp will depend on whether its refined or unrefined, use unrefined if you like coconut flavor but it shouldn't change the recipe otherwise.

Grind whole leaf green tea in a coffee blender and filter out any sticks/etc with a wire mesh. Blend the rest of the ingredients with a stick blender until consistent, then add powdered green tea. It'll be a spinach color and tastes like candy.

For a workout gel you can keep rocket fuel in this state or freeze it in an ice cube tray if you prefer it as solid candy. I wouldn't recommend taking more than a third of this a day, but should last a few days. It takes about 5 minutes for my heart to start racing and lasts anywhere from 45 minutes to an hour, its great for sprinting, weight lifting, and pilates.

I recently learned through Rugby practice that taking this with exercise before a night of heavy drinking eliminates hangovers completely. This is counter-intuitive considering sugar, coconut oil, and green tea should dehydrate if anything, so I'd like to get feedback on if this works for you.

The Soy-Powered Mind

2011-03-26T11:30:00.006-04:00

A few weeks ago I started taking a soy-derived drug called Phosphatidylserine to help combat stress-induced high blood pressure. The results have been remarkable enough to forfeit some medical privacy to share.

First an important disclaimer; I am not a doctor, nothing in this post represents medical advice, and my experiences with this drug may be largely due to having an existing adrenal disorder, acute synesthesia, and engaging in rough sporting activities that have led to multiple concussions. As this drug is considered a dietary supplement in the US its not heavily regulated and is available in many forms which may be dangerous, such as combined with high amounts of caffeine in a diet pill. Caveat emptor, especially with things you put in your body.

My source was Vitamin Shoppe brand "Phosphatidylserine Complex", each capsule containing primarily 100mg Phosphatidylserine, 25mg Phosphatidylcholine, and a grab bag of other phosphatolipids extracted from soy lecithin. I've been taking 4 capsules a day, between meals, the first of the day with 500mg L-Carnitine before breakfast and the last of the day with 10mg Policosanol just before bed.

For the first three days the experience was shocking; colors became brighter, music richer, tastes bolder.. all of my senses increased as if I've been living in a monochromatic world and suddenly I'm in OZ surrounded by singing midgets.

On the second day I started noticing strange radial rainbow patterns around sharp contrasts of light such as my hand in front of my netbook screen. By day four I had either become accustomed to this experience or my brain compensated for it. I started noticing that I was remembering words that I had lost through repeated concussions and accidental electrocutions, as if the drug was triggering a mass repair effort on my white matter.

My mood has changed even more dramatically; in contrast it feels like I've been in a constant state of fight or flight for as long as I can remember and suddenly.. its gone. I'm calmer, happier, can concentrate better, get angry far less and have become generally nicer to everyone. Many of my friends and family have remarked about this.

The period of time I've been on this (less than a month) would have normally been extremely stressful; I've been sprinting on PySoy, went to PyCon, organizing Summer of Code, getting married (last Sunday), and working as treasurer for a local non-profit getting their bi-annual tax filings together since 2007.

Through it all my blood pressure has actually gone down, perhaps the most scientific measurement of stress, and lost 6kg of body fat while my diet has actually been fairly bad considering all the travel. As an herbalist I've seen what belief in a drug can do by itself, the changes from this drug have been far beyond what would be reasonable for a placebo effect.

While I hate the idea of being dependent on a drug, the changes to my mind, body, happiness and productivity is well worth it.

PyCon 2011 wrapup

2011-03-17T14:31:00.002-04:00

This has been an awesome week at PyCon, though I'm completely exhausted.

The most notable outcome of several meetings with Python leadership about increasing diversity in the Python community. In my role with PSF's educational outreach programs I'm calling on 3rd party projects working with us to help work to increase gender balance in our community, and as PySoy's project maintainer I'm committing us to working seriously on this issue over the next year as an example.

Several college-aged women at PyCon knew about Google's Summer of Code but believed it was only for "geniuses who've been programming since they were 13" or that they were unqualified for other reasons, even though they're clearly more skilled (and moreso, have a better attitude) than some of the male students we get.

To address this, my first step is outreach to women for applying to us for Google's Summer of Code in hopes of resolving any miscommunication over required experience and what we're looking for. This is being sent out over the next week so potential applicants leverage of early involvement with mentors that many students have already started on.

Once Summer of Code is underway I plan to continue this an internship program in attempt to get more women involved in our community in ways not supported by GSoC such as through QA testing, documentation, video tutorials, and artwork. This program will be for women ages 18+ with no college requirement, though we're still working out many of the other details.

For the PyCon sprints I worked on the new "packaging" package for Python 3.3, formerly known as distutils2. My primary interest in this is a packaging.backport module based on 3to2 (which backports code from 3.2 to 2.7) extended with further fixers for 2.7 to 2.6, 2.5, and 2.4. This module will work for packages written for Python 3.3 and above to maintain Python 2 compatibility, and also to backport "packaging" to the standalone distutils2 package for previous Python versions.

We managed to get distutils2 largely upgraded for Python 3.3 syntax and the non-PyPI tests running cleanly, though much more work is needed. We have a few months given than Python 3.2 was just released and we're working on a roughly 18 month release cycle.

Sprinting toward Beta-3

2011-02-28T10:48:00.002-05:00

We've made some massive headway in the last two weeks;

First, we have physics processing back. We're using ODE for now just to get us through the next release cycle or two, but we've modified its headers (ABI-compatible) to work better with libsoy. Done are the days when we stress over API compatibility, since we've decided to write our own fully integrated (and scaled down) physics processing with Orc we have a certain freedom to throw their conventions to the wind.

How this is going to play out is a slow migration of values to arrays. Body data, for example, will be stored an array inside a scene so SIMD processing can be used more efficiently. The Soy Body class will represent a pointer to that array and include properties and methods for working with its data. The actual storage of that data will be governed by the Scene, giving it the freedom to sort bodies per "island" (groups of bodies connected by group). This is similar to how ODE processes bodies already and we'll be likely using their code as a starting place.

Second, we have the basic types for rendering. At the moment this is just a rotating cube, but that involved roughly 6 classes operating in both GObject and Python domains. The next step is soy.models.Mesh for rendering more complex shapes.

Third and most exciting, our humble game engine can now run as a Firefox plugin! There's some bugs to be worked out still, but "embedded" support has been added to our soy.widgets.Window class that allows it to run inside virtually any application.

With luck and determination we'll have the next release out by the end of the month. I've dialed back our goals considerably given that its a rewrite from our last beta release but our release cycle will be much shorter now that the rewrite is finishing up.

New import (and export) mechanism

2011-02-19T15:54:00.003-05:00

I've started writing a new import mechanism for .soy files, replacing the old soy.transports module with a importer that allows .soy files to be loaded as modules with the standard Python import command.

With the old method, "transport" objects acted as dicts of objects in the .soy files and could be loaded and saved as such. The resulting code looked like this:


import soy
MyGame = soy.transports.File('data/MyGame.soy')
window = MyGame['window']
redblock = MyGame['redblock']

(etc)

Now, the new .soy importer is added to sys.path_hooks when PySoy is first initialized and applys to any directory added after. When finished, the above code could be rewritten as:


import soy
sys.path.append('data/')
import MyGame

No "etc" needed because in that one import command data/MyGame.soy is loaded and made available directly in a module called MyGame. Much more pythonic, IMHO.

A separate class will be written next called soy.Exporter which PySoy objects can be added and removed from, then saved to a file such as this example which modifies the previously imported MyGame module:


archive = soy.Exporter(MyGame)
archive.grasstex = grass_texture
del(archive.bricktex)
archive('data/MyGame2.soy')

Similarly, two archives could be combined like this:


archive = soy.Exporter(MyGame)
archive2 = soy.Exporter(CharacterFoo)
archive += archive2
archive('data/MyGame3.soy')

The reasoning behind this is in most cases .soy files will be written by GIMP or Blender, so the most common use case for the soy.Exporter class is for combining and pruning the .soy files created by these tools.

As always, constructive/critical feedback welcome and appreciated.

libsoy migration milestone

2011-02-05T19:39:00.002-05:00

After a great deal of time (I'd rather not admit how long) we've finally managed to link the new libsoy code in with Python.

This afternoon (around 4:30) the following code opened a PySoy window on the screen:


arc@khonsu ~/work/pysoy $ python3
Python 3.1.3 (r313:86834, Dec  5 2010, 09:55:24) 
[GCC 4.5.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import soy.widgets
>>> w = soy.widgets.Window()
>>> 
(soy:16042): Gdk-CRITICAL **: IA__gdk_window_set_title: assertion `title != NULL' failed
IRQ's not enabled, falling back to busy waits: 2 0
OpenGL version 2.1 Mesa 7.10
GL_ARB_vertex_buffer_object: Yes

While incomplete and, well, its only a window, this is the first time we've gotten the two pieces to work together. David Czech and I have spent the rest of the evening fixing both build systems (libsoy and PySoy) and getting ready to copy the window binding code for the rest of the engine.

Our largest failure in this has been diverting so much energy into GObject Introspection. While its a really great idea and I'd love to see it evolve, its still so deep alpha that I can't imagine anyone but Gnome developers getting much practical use out of it. I would like to finish up GTypes at some point, but either the XML format for .gir or any level of documentation on typelib needs to be written first. I've wasted far too much time already trying to reverse engineer these from examples and their source that should have been put into my own projects.

common Orc opcodes

2010-12-30T05:02:00.003-05:00

I've been going through liboil's 0.3 source to rewrite the oil_yuv2rgbx_sub2_u8 function we use for Theora decoding to Orc pseudo-assembly code.

Because the Orc opcode documentation splits opcode description and processor support between two tables, for reference I wrote a quick Python script to build a table of Orc opcodes common to SSE (x86), Altivec (PPC/Cell), and NEON (Arm Cortex) processors.

Here's that table for reference, at least until I put the time to format it for a wiki:

opcode	dst	src1	src2	description	pseudo code
absb	1	1		absolute value	(a < 0) ? -a : a
addb	1	1	1	add	a + b
addssb	1	1	1	add with signed saturate	clamp(a + b)
addusb	1	1	1	add with unsigned saturate	clamp(a + b)
andb	1	1	1	bitwise AND	a & b
andnb	1	1	1	bitwise AND NOT	a & (~b)
avgsb	1	1	1	signed average	(a + b + 1)>>1
avgub	1	1	1	unsigned average	(a + b + 1)>>1
cmpeqb	1	1	1	compare equal	(a == b) ? (~0) : 0
cmpgtsb	1	1	1	compare greater than	(a > b) ? (~0) : 0
copyb	1	1		copy	a
loadb	1	1		load from memory	array[i]
loadpb	1	1		load parameter or constant	scalar
maxsb	1	1	1	signed maximum	(a > b) ? a : b
maxub	1	1	1	unsigned maximum	(a > b) ? a : b
minsb	1	1	1	signed minimum	(a < b) ? a : b
minub	1	1	1	unsigned minimum	(a < b) ? a : b
mullb	1	1	1	low bits of multiply	a * b
mulhsb	1	1	1	high bits of signed multiply	(a * b) >> 8
mulhub	1	1	1	high bits of unsigned multiply	(a * b) >> 8
orb	1	1	1	bitwise or	a \| b
shlb	1	1	1S	shift left	a << b
shrsb	1	1	1S	signed shift right	a >> b
shrub	1	1	1S	unsigned shift right	a >> b
signb	1	1		sign	sign(a)
storeb	1	1		store to memory	special
subb	1	1	1	subtract	a - b
subssb	1	1	1	subtract with signed saturate	clamp(a - b)
subusb	1	1	1	subtract with unsigned saturate	clamp(a - b)
xorb	1	1	1	bitwise XOR	a ^ b
absw	2	2		absolute value	(a < 0) ? -a : a
addw	2	2	2	add	a + b
addssw	2	2	2	add with signed saturate	clamp(a + b)
addusw	2	2	2	add with unsigned saturate	clamp(a + b)
andw	2	2	2	bitwise AND	a & b
andnw	2	2	2	bitwise AND NOT	a & (~b)
avgsw	2	2	2	signed average	(a + b + 1)>>1
avguw	2	2	2	unsigned average	(a + b + 1)>>1
cmpeqw	2	2	2	compare equal	(a == b) ? (~0) : 0
cmpgtsw	2	2	2	compare greater than	(a > b) ? (~0) : 0
copyw	2	2		copy	a
div255w	2	2		divide by 255	a/255
loadw	2	2		load from memory	array[i]
loadpw	2	2		load parameter or constant	scalar
maxsw	2	2	2	signed maximum	(a > b) ? a : b
maxuw	2	2	2	unsigned maximum	(a > b) ? a : b
minsw	2	2	2	signed minimum	(a < b) ? a : b
minuw	2	2	2	unsigned minimum	(a < b) ? a : b
mullw	2	2	2	low bits of multiply	a * b
mulhsw	2	2	2	high bits of signed multiply	(a * b) >> 8
mulhuw	2	2	2	high bits of unsigned multiply	(a * b) >> 8
orw	2	2	2	bitwise or	a \| b
shlw	2	2	2S	shift left	a << b
shrsw	2	2	2S	signed shift right	a >> b
shruw	2	2	2S	unsigned shift right	a >> b
signw	2	2		sign	sign(a)
storew	2	2		store to memory	special
subw	2	2	2	subtract	a - b
subssw	2	2	2	subtract with signed saturate	clamp(a - b)
subusw	2	2	2	subtract with unsigned saturate	clamp(a - b)
xorw	2	2	2	bitwise XOR	a ^ b
absl	4	4		absolute value	(a < 0) ? -a : a
addl	4	4	4	add	a + b
addssl	4	4	4	add with signed saturate	clamp(a + b)
addusl	4	4	4	add with unsigned saturate	clamp(a + b)
andl	4	4	4	bitwise AND	a & b
andnl	4	4	4	bitwise AND NOT	a & (~b)
avgsl	4	4	4	signed average	(a + b + 1)>>1
avgul	4	4	4	unsigned average	(a + b + 1)>>1
cmpeql	4	4	4	compare equal	(a == b) ? (~0) : 0
cmpgtsl	4	4	4	compare greater than	(a > b) ? (~0) : 0
copyl	4	4		copy	a
loadl	4	4		load from memory	array[i]
loadpl	4	4		load parameter or constant	scalar
maxsl	4	4	4	signed maximum	(a > b) ? a : b
maxul	4	4	4	unsigned maximum	(a > b) ? a : b
minsl	4	4	4	signed minimum	(a < b) ? a : b
minul	4	4	4	unsigned minimum	(a < b) ? a : b
orl	4	4	4	bitwise or	a \| b
shll	4	4	4S	shift left	a << b
shrsl	4	4	4S	signed shift right	a >> b
shrul	4	4	4S	unsigned shift right	a >> b
signl	4	4		sign	sign(a)
storel	4	4		store to memory	special
subl	4	4	4	subtract	a - b
subssl	4	4	4	subtract with signed saturate	clamp(a - b)
subusl	4	4	4	subtract with unsigned saturate	clamp(a - b)
xorl	4	4	4	bitwise XOR	a ^ b
loadq	8	8		load from memory	array[i]
storeq	8	8		store to memory	special
splatw3q	8	8		duplicates high 16-bits to lower 48 bits	special
convsbw	2	1		convert signed	a
convubw	2	1		convert unsigned	a
splatbw	2	1		duplicates 8 bits to both halfs of 16 bits	special
splatbl	4	1		duplicates 8 bits to all parts of 32 bits	special
convswl	4	2		convert signed	a
convuwl	4	2		convert unsigned	a
convslq	8	4		signed convert	a
convulq	8	4		unsigned convert	a
convwb	1	2		convert	a
convhwb	1	2		shift and convert	a>>8
convssswb	1	2		convert signed to signed with saturation	clamp(a)
convsuswb	1	2		convert signed to unsigned with saturation	clamp(a)
convuuswb	1	2		convert unsigned to unsigned with saturation	clamp(a)
convlw	2	4		convert	a
convhlw	2	4		shift and convert	a>>16
convssslw	2	4		convert signed to signed with saturation	clamp(a)
convql	4	8		convert	a
mulsbw	2	1	1	multiply signed	a * b
mulubw	2	1	1	multiply unsigned	a * b
mulswl	4	2	2	multiply signed	a * b
muluwl	4	2	2	multiply unsigned	a * b
accl	4	4		accumulate	+= a
swapw	2	2		endianness swap	special
swapl	4	4		endianness swap	special
select0wb	1	2		select first half	special
select1wb	1	2		select second half	special
select0lw	2	4		select first half	special
select1lw	2	4		select second half	special
mergewl	4	2	2	merge halves	special
mergebw	2	1	1	merge halves	special
splitlw	2	4		split first/second words	special
splitwb	1	2		split first/second bytes	special
addf	4	4	4	add	a + b
subf	4	4	4	subtract	a - b
mulf	4	4	4	multiply	a * b
maxf	4	4	4	maximum	max(a,b)
minf	4	4	4	minimum	min(a,b)
cmpeqf	4	4	4	compare equal	(a == b) ? (~0) : 0
convfl	4	4		convert float point to integer	a
convlf	4	4		convert integer to floating point	a

stuck, back to PySoy

2010-12-29T04:17:00.003-05:00

I've been working on a streaming XML parser for Python, but need a break. At this point there's no way Concordance is getting out Jan 1st, but certainly by the end of Winter.

Our libsoy migration process PySoy got pretty far. We were migrating from Pyrex to Genie, essentially moving the core engine from PyObject to GObject to remove Python dependency in game clients and enable further multicore processing on both client and servers. Much of the rendering area of the engine has been migrated, but the process has been held up in two areas;

First, while libsoy is in pretty good shape, we still lack Python bindings - aka PySoy itself, which is what we intend games to be written and run with. Our original plan to use GObject Introspection failed in a horrible mess that I've documented in previous postings, we've looked at using SWIG and even building our own bindings generation with little measurable success. In order to get us moving forward again I'm going to just drop out some .c templates and write the custom wrapper classes by hand. The time it'd take to write and maintain these cannot possibly be greater than the time we've wasted talking about a more elegant solution that only exists conceptually.

When GObject Introspection reaches a state of even remote maturity, where it can offer a Pythonic API, we'll look at it again. We'd even help get it there if the current GIR developers would just document the .gir XML schema or typelib format so we wouldn't have to refer to their source code as the sole definition of these.

Second is our physics code. As I've posted, ODE worked for us in the past but has numerous issues with packaging for various Linux distros (and poor features, slow, and extremely difficult to port to mobile devices). We attempted to migrate to Bullet but this burned us out - virtually no work has gone into that in the past 6 months. We're all pretty frustrated with Bullet's haphazard API (whereas ODE is fairly clean) and the C++ only API doesn't play well with GObject (or anything other than c++ for that matter). Bullet's C API is minimal at best.

When it comes right down to it, the biggest barrier we face with physics is processing power on mobile devices, an issue that using Bullet would not solve. Most of the devices we're interested in include ARM6/7 processors from Qualcomm or TI. Many do not include a FPU (floating point unit), but they all seem to offer a fairly powerful DSP used extensively for processing multimedia. We do not, however, want to rewrite and maintain our physics processing for each platform.

A solution I've come up with is to write our physics, greatly simplified from even what ODE offers, using Orc. It's yet another metalanguage (first Pyrex, then Vala/Genie, now this..), but the successor to liboil (which we and much of the Gnome community use) and already supports many interesting platforms.

My plan is to first migrate our liboil-based YUV-RGB conversion code to Orc to get my feet wet, then implement a greatly simplified collision system using it, and expect the next release (or two) to still use ODE for at least rigid body physics with the plan to eventually replace even that with our own physics solver. It should be much faster, and the same Orc code we write now should be able to compile to DSP code for Android handsets and other mobile devices in the future.

Orc already supports ARM Cortex (NEON), so if we were to finish this work today we'd be able to run PySoy clients on more modern Android handsets without touching DSP code. DSP support in Orc would also be very useful for future hardware for PySoy game servers.

While we'd all really like to get the next PySoy release out ASAP, we'd also like to avoid rewriting the engine again down the road.

XML parsing in Python

2010-12-08T06:37:00.003-05:00

Its been a couple months, so I'm going to give a brief update on what I've been working on.

Concordance is getting close to release, I plan to have the first release (0.1) out January 1st. More on this toward the end of December.

One of the roadblocks I've hit (again and again) is the lack of a decent XML parsing package for Python. The standard library is a shame when it comes to XML; at least four different modules (expat, sax, dom, etree) to choose from and none of them support even XPath. The most popular option, etree (or ElementTree), cannot even process an XML file with the namespace prefix intact.

There's lxml, which offers an etree-compatible API and fixes many of ElementTree's major faults (namespace prefix preservation, xpath/xslt support) but still cannot handle stream processing and, due to ElementTree's API, does not expose multiple text nodes broken up by a child element such as "<div>first string <br/> second string</div>".

To support XMPP streams we need to use expat or sax to handle the stream event-by-event, since the full XML document is only available once the root element closes at the end of the stream, but the direct children of the root element (what we call "stanzas" in XMPP) need to be processed as complete objects. While we may be able to hack something together using lxml, it would likely be less work than to implement a new XML parsing package. As long as the resulting API doesn't diverge very greatly etree the work necessary to switch should be minimal.

Beside this I've been working on a host of different packages around Concordance, from getting a javascript BOSH/XMPP library together to getting distutils2 ready for Python 3. I've even managed to ship a pitiful little serial library for Python 3, PyTTY that we're using to interface with some Arduinos.

Diaspora: the $200k scam on free software

2010-07-02T15:13:00.003-04:00

After a month of work, its bluntly clear the community has been scammed for $200k.

Four NYU students posted a video and solicited funding on the kickstarter website, promising to build an open source, pro-privacy alternative to Facebook. The community response was astonishing. Its too bad the people pledging didn't research these students, for example, checking to see if they have Ohloh pages with their past contributions to other projects.

The first and most severe, after a month of work, they have posted that they have running code but will not allow the community (including the people who paid for it) to view until its "complete". This is commonly referred to as the Cathedral model. People have invested a great deal of trust in them to develop the code they have promised to, the least they could offer is read only access to their VCS. Instead they posted a video demonstrating it, which poses the second problem.

Their claim to use the "latest and greatest in web standards" fell flat on its face in their choice of method to post their video. Instead of using HTML/5 <video>, which they could have even used Ogg with the Cortado java player as fallback for MSIE and Safari, they used a flash player. Perhaps they don't realize Flash is not a web standard, or that HTML/5 has built-in video support, in either case its a really bad sign.

The most blunt demonstration of their lack of skill and commitment is in their use of websockets. At this point, websockets is still in heavy draft state with the few browsers/servers implementing it all running a slightly different, incompatible variants, none of which will be the final draft which is (at least) months off. Moreso, they propose to implement chat and gaming over their own websockets-based protocol rather than using existing standards such as XMPP.

XMPP is being used by a wide number of social network services already including Google, Livejournal, and Facebook. Its an established, widely implemented standard recognized by the IETF and the community at large for federated, real-time exchange of chat and presence (status). Beyond chat, XMPP has support for PubSub, Data Forms, and Ad-Hoc Commands. There is virtually nothing Facebook, or Diaspora, aims to accomplish that cannot be implemented using these existing standards.

What these four students are building is lock-in, if they finish at all. Instead of replacing the walled gardens of Facebook and other proprietary social network sites, they aim to build a network of little sites running the same software, only able to federate with each other, and only able to be extended with their custom APIs. These sites will not federate with other distributed social network efforts being built such as movim and concordance.

Hopefully the community will learn a lesson from this. Pledge money to experienced developers with demonstratable code, not charismatic students pledging to work on free software only if they're paid to do it.

Using Android without Google (part 1)

2010-06-28T00:01:00.002-04:00

As magicfab on identi.ca requested, I'm starting a journal of my experience using Android without Google's proprietary apps.

I own an HTC Eris phone which originally came with Android 1.5. A few months ago a firmware was released by a 3rd party which allows users to gain root on their phone (about time) and I've been experimenting with this since. My primary interests were removing HTC's "Sense" UI so I could use a different soft keyboard and removing the backdoors HTC, Verizon, and Google installed in the phone.

After some disturbing problems with the VanillaDroid firmware, I switched to CyanogenEris 3.0 last week. Not only did this new build not come with HTC's Sense, thanks to Google hitting them with a threatening DMCA letter, the firmware came without Google's apps either. This includes GMail, GTalk, Google Maps and Market.

Everything works on the firmware except for USB tethering (which is a feature of the firmware). I started by using the browser to download the Barcode Scanner and then used that to install other apps via QR code:

The Foursquare app I was previously using refuses to install (likely because it depends on Google Maps app) and I haven't figured out how to install the dvorak keyboard add-on to soft keyboard.

I have the standard Android Contacts app, but without Google it does not sync to my Google contacts. I plan to remove my contacts from Google and write a small app that provides ContactsProvider2 via XMPP in an effort to decentralize and federate contact syncing.

I found the web version of GMail superior to the old app, though I do miss the Google Maps app (there are various OpenStreetMaps alternatives but they don't have the same features). The biggest thing I miss is the GTalk app, though its pitiful as far as XMPP clients go and we should be able to do a lot better.

Most importantly, my phone is now 99% free software with only a few of HTC's drivers left to be reverse engineered. Even for an advertising company Google is really stretching the truth when they call Android "open source" - but it should be an attainable goal.

concordance progress

2010-06-27T02:08:00.002-04:00

While dredging through framework API design, its sometimes difficult to sense how close you are to finish. Tonight the Concordance "pure Python" redesign reached a critical point - being able to add an extension and respond to an XMPP <iq> stanza:


>>> import concordance
>>> import concordance.extensions.ping
>>> s = concordance.Service('host')
>>> s.append(concordance.extensions.ping.Ping)
>>> import xml.etree.ElementTree as et
>>> st = et.fromstring('<iq from="arc@host" id="f00" to="host" type="get"><ping xmlns="urn:xmpp:ping" /></iq>')
>>> s.stream(st)
<iq from="host" id="f00" to="arc@host" type="result" />
>>>

That's a correct response to an XMPP Ping request, one of the simplest XMPP extensions. Many of these basic extensions will be combined into a single extension module and a ready-made Service class, this is just proof that it works.

Concordance stickers ordered

2010-06-16T21:50:00.002-04:00

I just ordered 1000 2" x 2" vinyl laptop stickers with the Concordance logo on white. They should arrive by the end of the month. Contributors can send me their mailing address to receive one along with one of the new Python 3 stickers from the PSF.

New PySoy stickers will be made as soon as we have a new logo.

Happy 4th Birthday, PySoy

2010-06-06T04:40:00.002-04:00

On this day, June 6th 2006, the PySoy Project was born.

We've come a long way from the small homebrew Pyrex-based game engine we started from. We've built a solid development team, spun off numerous related projects, contributed hundreds of bug reports and patches for libraries and applications, and grown close to the Python, Gnome, and XMPP communities.

Currently halfway through our most ambitious endeavour yet, moving our codebase to the GObject model and preparing the engine and related projects for cloud gaming, there's a bright future ahead for this tiny project. While its sometimes frustrating that we're still not to 1.0 release, looking back to what we started from four years ago its clear we've made tremendous progress.

Here's to another exciting year!

A killer Android app

2010-05-27T03:13:00.002-04:00

I've been running the Vanilladroid firmware for my HTC Eris for a few weeks now. Beside the phone spontaneously rebooting itself every few hours, its run great, and with built in wifi tethering.

Tonight it proved potentially fatal for one young man. While walking home it appeared like I interupted a mugging near HacDC. The victim was being kicked by a man who ran away as I got closer. He was laying sprawled out on the concrete, face right on the sidewalk and shirt ripped off. I pulled out my phone, hit the icon for Dialer One (which comes with Vanilladroid), dialed 911, dial key, and... nothing. Dial key again, nothing. Checked, it says 911 in the number box, but the app isn't dialing it. I closed the app and tried again, notta. I can't believe it would refuse to dial 911 so I hard reset the phone and try again, still the dialer just sits there as I press the dial key.

I finally had to go on the web, look up the number for Dc Police, and have them transfer me to 911.

I take partial blame for not looking into this app and switching it out sooner. Hopefully Google removes it before someone dies while trying to call for help.

praise for Gnome's GIO

2010-05-21T17:39:00.002-04:00

We've been (finally) getting Concordance's C code refactored into a standalone XMPP server called GNation.

Concordance used a good deal of GLib code already, but we've gone a bit further in using the GIO library for managing sockets and connections. In doing so, we were able to reduce roughly 160 lines of code for opening and monitoring listening sockets into 12 and I suspect further reductions as we continue. All of the IPv4 vs IPv6 support was cleanly abstracted, adding a socket listener to the MainLoop was done automatically, and the GIO documentation was of the same high quality we've come to expect from GLib.

Major thanks to the hard work that has gone into this!

How free is Google's VP8 (WebM)?

2010-05-19T15:57:00.005-04:00

Thank you Google for using your wealth to purchase On2 corporation and release VP8 to the world. However, there appears to be one glaring flaw in the patent license you offer which makes it incompatible with free software licensing: it appears to only cover the VP8 bitstream itself, not improvements to it made by the community.

When On2 previously released VP3.2 to the Xiph foundation and the world, the patent license explicitly included derivatives (improvements) to VP3.2 including Ogg Theora. This was a critical step in transitioning the proprietary VP3.2 codec to the open codec Theora is today.

On2's "VP" codecs included little extensibility, why should they as On2's revenue stream depended on releasing (and offering a new license for) a new improved codec version every year or two. You cannot use their VP5 codec to decode VP6, you need to get a VP6 codec for that. In contrast, Vorbis is ten years old now and is still being improved on.

The large body of work undertaken primarily by Tim and Ralph at Xiph before Theora 1.0 could be released was extending the bitstream in a losslessly upgradable manner from VP3.2 to include the fields necessary to continually improve Theora over the next decade and beyond. You can see this by playing a VP3.2 video next to a Theora 1.1 video, and even more so in the Theora 1.2-beta9 comparisons, and they've barely scratched the surface still.

For comparison, On2 corporation's patent license for VP3.2 can be found in the Theora license file vs Google's VP8 bitstream license. As I'm not a lawyer, I would love to hear SFLC's evaluation of this in their next Oggcast.

stability, always a few versions behind

2010-05-18T05:53:00.002-04:00

Several of our projects use waf and Vala, which I normally have nothing but good things to say about, but this is an exception. At the moment, we're stuck working with outdated versions of both.

Tonight I tried upgrading from Waf 1.5.10 to 1.5.16 and was unable to get any of our packages to build with it. As it turns out, it doesn't seem to like glib (maybe the "-" or number in the package name?) and fails to record its cflags and libs to pass to gcc during building. If 1.5.10 didn't have a few annoying bugs of its own or if we could patch and "compile" 1.5.10 we'd be fine, but even this version we had to use the "binary" (compressed Python script) from their website as we couldn't get a Python 3 version to build from source.

Despite all this, it is very nice to have a small, clean build system that both supports Genie and runs on Python 3. The small workarounds we've had to do are worth the savings in headache and frustration autotools/scons/etc would cause us.

With Vala, apparently nobody did even a casual check to ensure the Genie parser was still working properly before releasing valac 0.8.0 (now up to 0.8.1), and a basic example from the language's frontpage (command line argument parsing) compiles but fails to link. For now, we're all forced to stick with valac 0.7.10 until the problem is solved or someone suggests a workaround. Its unfortunate a few of our devs run Ubuntu 10.04 which doesn't offer valac 0.7.x as a package.

Good news is I'm almost finished debugging expat.vapi (Expat bindings for Vala/Genie) and David is sprinting ahead with xml config file parsing with it. By this weekend we could have XMPP routing with the new server being branched off from Concordance-XMPP.

dusting off pyogg2

2010-05-15T03:11:00.003-04:00

I've started work on updating an old CMS I worked on between 2002-2005 that includes a non-linear Ogg editor and Icecast stream queue management. Part of this is updating the CMS to Python 3, including pyogg2 which it uses.

Since I no longer have working ssh keys for committing to svn.xiph.org, I copied the pyogg2 repository to hg.concordance-xmpp.org and resumed the work I left off in 2005. Pyogg2 was in fairly stable shape but lacked documentation, utilities for working with common codecs, and only had a few examples to learn it with. I'm already given its build system a major cleanup (now just "setup.py build" like any other Python package) and almost finished the Python 3 PyModule code.

I have a lot more work to do on this; the CMS needs to be updated for modern web standards (HTML/5), the homebrew MySQL ORM with SQLAlchemy, and changing from the old CGI Apache configuration setup to CherryPy.

Leaving the Ubuntu project

2010-05-12T03:30:00.002-04:00

In response to the recent news that Canonical has stated peddling MPEG-4 patent licenses to Ubuntu OEMs, I am resigning as member of the Ubuntu project.

While I find this latest move completely unethical, I have been previously disturbed by Canonical misusing the Ubuntu trademark with their proprietary Ubuntu One service, which they have recently added a proprietary music store to for MP3-only downloads fully integrated with Ubuntu's version of Rhythmbox, and bundling an ever-increasing amount of proprietary drivers with Ubuntu.

While I find Ubuntu's parent company ethically repugnant and have always detested the developer-unfriendly packaging system Ubuntu (Debian) uses, I still find the Ubuntu community an outstanding model for how the free software community should organize. I'm certain that if put to a vote, the Ubuntu community would release the Ubuntu One server code as AGPLv3 and discontinue unethical business practices conducted in its name, but the only way we can vote against Canonical is with our feet.

As I serve in a leadership position in a local community team, I'll be upholding my agreement with the Ubuntu Code of Conduct by gracefully stepping down after finding another team member to take my place and migrating any remaining accounts off my @ubuntu.com email address.

I've started celebrating by getting back into Gentoo packaging, starting with a CherryPy ebuild which supports both Python 2 and 3. My final Ubuntu workstation is slated for a Gentoo install this coming weekend.

server migration complete

2010-05-12T02:20:00.004-04:00

I have finished migrating all the websites I host from maat (at ServerBeach) to nut (at Hurricane Electric). Tonight I finished wiping all the passwords, ssl certs, and other private data from the old server before its shut down tomorrow.

Maat was an Athlon XP 2200+ server with 1gig ram, 80gig HD, 10meg up/down, 2TB/month transfer, and 1 IPv4 address. Nut is a Atom 330 (1.6ghz 64-bit dual core) with 2gig ram, 500gig HD, 100meg up/down, unlimited bandwidth, 6 usable IPv4 addresses and a /64 IPv6 subnet - and costs less.

I shed a tear working through maat's filesystem checking for any remaining files to be copied and finding the original Gentoo stage3 tarball used to install it: /stage3-i686-2007.0.tar. Tomorrow ServerBeach will likely disconnect maat and retire it as after 3 years the server is out of maintainability.

My experience so far with Hurricane Electric is that their remote admin tools are limited to "remote hands" (call a technician 24/7 to reboot a server, type in commands, etc), where ServerBeach offered a really nice web portal for rebooting, accessing a rescue image, etc without having to call anyone. Of course, those tools haven't been working correctly since December. The technicians at Hurricane Electric have demonstrated a far higher level of expertise than anyone I've spoken to at ServerBeach.

Unlimited 100meg bandwidth, IPv6, and 10x bandwidth (HTML/5 Ogg audio/video hosting) is the real seller. In retrospect, I'm really glad ServerBeach had the FUBAR downtime in December prompting me to look for alternatives, this is a much better deal.

working on gtypes

2010-02-20T22:46:00.002-05:00

We've been moving forward on the PySoy rewrite using Genie in faith that a Python 3 binding for GObject would come about.

At PyCon this weekend I've brainstormed with Kirk McDonald, author of PyD which serves a similar role to use D types/objects in Python, and several other authors of FFIs for Python. A rough strategy is coming together to solve this in a dynamic manner.

The first part of the solution is libvala; using a small part of its functionality to parse .vapi and .gir files for existing GObject libraries such as GTK and Cairo as well as any library written in Vala or Genie. This gets us the C API for the library in all it's object oriented goodness, and with that we can dynamically generate bindings using Python's ctypes.

The second part is a GObject type in Python which supports subclassing in Python. Kirk detailed how this is done in PyD; each generated GObject type gets a subtype with all of its virtual methods all running Python methods which call the superclass's C method by default. When a Python subclass of a GObject class is made this GObject subclass is used allowing Python to override any of its methods and pass objects of the subclass back to GObject where they'll perform as expected. Not simple or easy, but its a strategy.

The end result is dynamic Python 3 bindings for GTK as well as PySoy and all the other tools we work on.

I'm organizing a sprint for this at PyCon. If anyone has ideas on this please comment, or if you'd like to help (whether at PyCon 2010 or not) get in touch with me.

Epson Artisan 810 on Linux (Ubuntu)

2010-01-13T15:00:00.002-05:00

We purchased a new printer for the house last night to replace the windows-only Dell printer that ran out of ink (and would cost almost as much as a new printer for replacements).

First, some advice: plan ahead, order online, don't buy from Staples. We paid $170 for this when Epson is offering them direct for $130 with free shipping. If we were not in need of immediate replacement we could have saved $40 and waited a week. Further, the rep at the store lied when he told us the printer was going to the back to get a unit "fresh off the truck", it was a unit another customer had returned with ink spilled and dried on the inside and some of the packaging torn open. Buyer beware. Otherwise, they seemed to clean it up and repackaged it well. I have half a mind to order the printer online and return this one within 14 days for refund to save the $40.

The printer itself is fairly impressive. Two paper trays (letter/legal plus photo/cardstock) and a built-in CD printing tray that slides out when you select CD printing, scanner with auto-document feeder, and built in 10/100 and wifi so we don't need to depend on any computer in the house to properly "share" the printer. All I had to do is select our access point on the front controls and give it a static IP on our router.

Ubuntu 9.10 detects the printer over the network in seconds and I was printing the Ubuntu test page in under a minute, so more than half the systems in the house were easy. Windows setup was not hard, but time consuming as loads of bundled software installed from the CD. The only complaint I have is the color seems a little off; printing the same image in both Linux and Windows shows a fairly radical difference in quality. The prints from Linux are still much better than printing from Windows to our older printer.

Next I tried to use the scanner function. Using GIMP to launch SANE (Scanner Access Now Easy), the printer was automatically detected and let me select either the flatbed or auto-document feeder. Both options work fine until I start the scan, at which point the Epson Artisan 810 locks up and needs to be rebooted.

There is a scan function from the front panel, not supporting anywhere near the scanner's full resolution and storing in JPEG format with obvious artifacts around text/etc, but at least we don't need to use a Windows system for basic scans until SANE is fixed. Since I have a stand-alone Epson scanner that works beautifully this isn't a huge issue for me, though it would be great to be able to use the auto-document feeder from Linux and be able to put this older scanner into storage.

soy.windows.Window rendering, GLEW headache

2010-01-10T22:09:00.006-05:00

I'm seeing "red" now with our Windows, in more ways than one. The rendering code for soy.windows.Windows is to the point that, by changing the glClear color, we're seeing red in the window rather than black. This is a big step forward given that porting to use GDK is a big part of the libsoy migration.

On the other side of "red" is anger over GLEW. I really wish I knew of a decent, mature alternative to GLEW. The project is so immature that, though one of the maintainers is a Debian developer, they do not provide the pkgconfig file for it. Without this file, which provides Waf with the location of the library and it's headers, we would have to write a search script to verify that it's installed and find it's location. None of the major distros I looked at provide the pkgconfig on their own.

It's also at least partially redundant with the functionality of gtkglext which I believe should be able to replace GLEW entirely.

libsoy running on OSX - briefly

2010-01-08T03:16:00.003-05:00

For the first time in 3.5 years, we had soy.windows.Window running on OSX. Just the basics; opening, (re)setting the title, resizing - but it worked. This is thanks to GDK, the windowing and event backend to GTK+, running on Mac OSX via native Quartz backend (not X11).

Shortly after we moved the window code to a background thread and it broke. The code runs fine on GNU/Linux and Windows, but on OSX all we get are white rectangles where the windows would normally be rendered. The rectangles even resize properly, and window events are getting processed by our code, just nothing rendered inside (not even the window title or buttons). I filed this bug and hope the problem to be solved quickly.

The problem could be as simple as a missing NSGraphicsContext.flushGraphics call (which is only needed when rendering is done in a non-main thread). If the bug hasn't been fixed by our next release I'll look into patching this myself.

language mixing

2010-01-05T17:07:00.003-05:00

After some work to update our ODE bindings for Vala, we discussed Bullet again last night and returned to the conclusion to drop further work on ODE and focus on Bullet for the next release.

It quickly became clear that the C bindings for Bullet are utter horse crap; praise for the effort for whoever wrote them, and I by no means intend to disparage your ability or intentions, but they're horribly incomplete and usable for little more than the two demo apps included in the source. Moreso, the C bindings are "higher level" than the C++ library, performing several steps automatically (such as calculating body inertia from mass and shape) which our 3d game engine needs more direct control over. I first wrote an alternative (ABI compatible) C header for it, and then started to rewrite it entirely.

I decided that if we're going to write our own Bullet compatibility layer, we may as well go all the way in wrapping it with GObject. The code is simple enough, but defining GObject in C involves a lot of boilerplate - in some cases more than the code itself. To get through this and retain my sanity, I started work this morning on a C++ code generator that takes the relevant functions and wraps them in the standard boilerplate.

The format I'm using is Genie-like for class and method definition, then C++ within dumped directly to the target .cpp. The code generator will generate the .vapi as it works and compile our wrapper as a static library which libsoy can link to directly.

Kudos again to Waf for making this rather eccentric use of their build system dead simple and clean.

libsoy building separately now

2010-01-04T16:43:00.002-05:00

We've been working on the "libsoy" branch of PySoy for some time, a split of the compiled code to a GObject-based C library with Python bindings plus the pure Python code constituting PySoy itself.

Previously we were struggling to build this with Python's distutils system, which didn't support building system-wide C libraries nor using valac to do so. Jon and I started looking into Waf last week. It's pretty impressive; Python based (and works with Python 3), simple, intuitive, does everything we need with only a few dozen lines of script. Last night we setup libsoy in it's own repository and building (just a very basic soy.windows for now) on it's own.

I think we're settled to work with Genie, a comparison between the syntax of soy.windows in Vala vs in Genie makes that choice simple. The only complaint I have with Genie is needing to escape newlines inside struct {}'s (ie, line 32-35), but reportedly this is being fixed in an upcoming release to be more Python-like (ignoring newlines while inside () or {}).

After a bit of searching (through source code, I wish Waf had better API reference in it's docs) I found the gir= option for Vala building, and another hour later both soy-1.0.gir and soy-1.0.typelib were being generated, except both were empty save the "soy" namespace:


<?xml version="1.0"?>
<repository version="1.0" xmlns="http://www.gtk.org/introspection/core/1.0" xmlns:c="http://www.gtk.org/introspection/c/1.0" xmlns:glib="http://www.gtk.org/introspection/glib/1.0">
  <package name="soy"/>
  <namespace name="soy" version="1.0" c:prefix="soy">
  </namespace>
</repository>

I've hit this before with trying to generate GObject Introspection from distutils but chalked it up to not building it correctly. As it turns out, this is a known bug in GIR and a previously unknown bug in valac to fail silently.

I'm still hoping the GIR developers will come to their senses and support nested namespaces, but given that they've stubbornly rejected this for months already I'm afraid we'll have to build a cross-language bindings builder based on Vala/Genie instead of helping to update PyGI for Python 3.

state of Vala IDEs

2010-01-02T00:27:00.002-05:00

Wasted quite a few hours today trying to get a Vala/Genie IDE working.

VTG is basically broken, crashing at random. It had problems before this; having a fairly rigid concept of a "project", not supporting non-autotools build systems or libraries very easily, not supporting Genie, and the symbol mapping having odd quirks, so this is not a huge loss.

A newer Vala plugin for gEdit, Valencia, looks promising but doesn't support recent Vala versions and attempting to build it raised pages of errors.

Anjuta has always looked interesting, but I've never managed to get the Anjuta Vala plugin to compile. At present, it doesn't appear to work with Gnome 2.28.

I don't use Eclipse or Mono, so the final option is Valide. To my surprise, it's been packaged for Ubuntu and works out of the box. It supports Genie, it seems to have features on-par with VTG and the other IDEs, and works with the latest version of Vala.

My only complaint so far is the new project wizard doesn't list the AGPLv3 as a license option, but then none of the Vala IDE options do yet.

Python meme 2009

2009-12-29T15:54:00.000-05:00

Here’s a short, 5 questions, 2009 Python meme. Copy-paste the questions, and blog your answers !

1. What’s the coolest Python application, framework or library you have discovered in 2009 ?

SQLAlchemy, not only does it fully (properly) abstract the SQL database for you, but provides a full Object Relational Mapping. What's more, it's Python 3 ready.

2. What new programming technique did you learn in 2009 ?

Using GObject Introspection to mix several languages (ie, Python and Javascript) in the same application.

3. What’s the name of the open source project you contributed the most in 2009 ? What did you do ?

Concordance XMPP, an XMPP service framework for Python I started January 1st 2009.

4. What was the Python blog or website you read the most in 2009 ?

Python Planet

5. What are the three top things you want to learn in 2010 ?

peer1/serverbeach severe failure

2009-12-07T13:19:00.003-05:00

We've seen a greater than average share of service outages at Serverbeach; incompetent electricians blowing out the battery backup system causing an evening of downtime, network failures taking the server down for hours at a time, other random outages without explanation.

Of course these outages affect large numbers of customers so you get 500 errors when trying to access the customer portal or put on hold for up to an hour waiting to talk to someone only able to enter a ticket into the system on your behalf.

This latest outage tops them all, however. Yesterday afternoon they attempted to merge the Peer1 and Serverbeach customer portals which knocked out DNS for a "large number" of customers. They sent out an email to everyone and I fully expected it to be a short outage.

Last night at around 8pm all of our domains were still down. I attempted to log into the customer portal to find my old password not working, and after a lengthy process to recover it I found myself locked out of it unless I agreed to a lengthy new service agreement which I certainly don't have time to read through in the middle of an outage.

After sitting on hold with customer service a ticket was filed. Which I replied to, and replied to, and have only received blanket emails asking to reply again if we're still out. More than 24 hours later we are still down.

Serverbeach claims they refund customers for outages, but I have yet to receive a dime credited for any past issues and I fully expect them to make getting credit for this difficult as well. At this point I don't think it's fair to myself or the communities which rely on this server to continue hosting with them.

I've found a really decent deal with Hurricane Electric with the added perks of IPV6, more bandwidth, and newer hardware for less than I'm paying now.

Gentoo wins again with Python 3

2009-12-01T04:28:00.002-05:00

Despite being heavily involved in Ubuntu advocacy and promotion, being a member of the Ubuntu project, and using it on my personal systems, I've had to keep my servers and development machines running Gentoo.

When developing software, we need to work on what will be "mainstream" in 6 to 12 months, not what's already mainstream. When a new version of our physics library is released, it may depreciate certain methods or include new features we'll really want in our release, but that version won't be packaged until 2-8 months after the new library was released, given the typical release cycles and "feature freeze" stopping new packages from being included too close to release time.

Python 3 is a great example of this; Gnome 3 will apparently use Python 3 for many applets, and Gnome 3's release schedule sets it for inclusion in Ubuntu 10.10 (next Fall), but at present Python 3.1 is packaged but none of the 3rd party packages are available for it - even those which explicitly support Python 3 already.

While doing my periodic Gentoo upgrade, I just noticed that Portage, the package system for Gentoo, can now be built to run on Python 3 directly. When you have both Python 2 and 3 installed on your system, packages wich support both will build and install for both, and their packaged versions are often available just days after release. From what I've seen, Gentoo is one of the best distributions for Python developers right now.

I wouldn't recommend that my family run Gentoo, many of whom happily run Ubuntu, but it's lack of support for developers is concerning to me. I do not believe it should be difficult to both support new Linux users and experienced developers with the same distribution.

Dell employee discounts a joke

2009-11-30T14:00:00.002-05:00

I just learned that the Dell employee discount, which is offered to employees of numerous Dell partners, does not apply to their N-series laptops, even though many people only have this discount because of their work with Ubuntu.

The irony continues to drip as the models I compared between the employee purchase price and the Ubuntu versions of the same, such as the Inspiron 15 and Inspiron 15n, roughly discounted the added cost of Windows 7 only. It appears that the employee discount equates to free Windows 7.

The current Dell Ubuntu offerings lack in many other areas, such as not offering gigabit ethernet or 802.11n as options. While I would love to support Dell in working with the Ubuntu community more, I really can't justify buying a sub-par laptop at the prices they're offering them at.

My friend Mackenzie showed off her System76 laptop at the 9.10 release party a few weeks ago. All of their laptops include gigabit ethernet, 802.11agn wireless, and many other options Dell doesn't offer. They're a bit more pricey, but according to Mackenzie their service is significantly better.

moving to Bullet

2009-11-19T15:22:00.002-05:00

Bullet has some really great features over ODE; soft body physics, particle physics, and hardware acceleration among the top.

After talking with a Crystal Space developer about this, I put it on my list of things to look into when PySoy is a bit more mature. However, a Debian packager's decision has accelerated these plans.

ODE is not designed to be installed as a single shared library, it has many build-time options which radically change it's behaviour - such as whether the library uses single or double precision, or whether to calculate gyroscopic force. As a general rule, distributions stick to the defaults when packaging as much as possible, but Debian (and thus Ubuntu and other Debian-based distros) ship a single system-wide library with non-default compile flags (Double precision) that would result in sub-par performance, network sync issues, and in some cases bugs in games.

Debian could have shipped four or eight different versions of the ODE library, such that games could be linked against the one it was designed for and work as expected, or ODE could have been designed to build both a single and double precision library with the other options as runtime flags, but instead we game developers are caught in the middle.

We fought this battle before with Soya, for some time actually, and in the end Soya had to ship their own version of ODE embedded in the source and statically linked to get around the problem. Even then it was an issue to get the package including a static ODE included. It's a battle not worth fighting.

While Bullet also supports building as double precision, it's an option so rarely used that no distro builds with it (Gentoo doesn't even have a USE flag to switch it), and given that Bullet is currently not packaged for Debian it'd be easier to add it as a new proper package than fight Debian packages to improve their build of ODE.

I am not looking forward to working with Bullet's incomplete C-API, likely having to fill in some gaps as we work on it, nor writing the Vala bindings for Bullet, but it'd be a lot faster and fun - not to mention resulting in new features.

beta-3 roadmap updated

2009-11-05T16:45:00.002-05:00

I updated the Roadmap for 1.0-beta3 today.

In short, we're dropping ambitions for audio support in this release. Once our migration to Vala is complete and the related changes implemented we'll release without further delay. After that we'll begin work on the newer features like audio, networking, and browser embedding for beta4 while the community tests and gives feedback on beta3.

There are a lot of new features already compared to the beta2 release. We're aiming for a beta3 release by January 1st 2010, 2 years from the beta2 release.

Kudos for Mercurial going GPLv2+

2009-10-27T13:16:00.003-04:00

The largest problem with integrating Mercurial with our new web framework is being resolved; they are upgrading Mercurial's license to GPLv2 or later.

Thanks to this, our favoured dVCS will be available for integrating into GPLv3 and AGPLv3 licensed projects without any weird hacks or work-arounds.

adding full Python support to Vala

2009-06-29T12:05:00.000-04:00

We've found a way to finish full Python support for Vala; plugin support will be added to valac soon and a PyObject plugin added to our distutils extension for handling Vala and Genie extension source files.

I'll be focusing on this over the next two weeks, with a target on Python 3.1. We're going with 3.1 as baseline for PyCapsule, which will help ensure type safety when passing GType objects through Python.

In the meantime work on libsoy can continue by the rest of the team.

vala support beyond python-3.0.vapi

2009-06-27T08:56:00.002-04:00

While the vapi is still a primary need, we also need PyObject and PyTypeObject support in Vala itself to generate the correct code.

This will result in both cleaner .vala source and more readable generated .c code, so this time investment will pay off in PySoy development in the long run.

the sad state of domain registrars

2009-06-24T11:09:00.005-04:00

My first business, started when I was 17, was an ISP - run out of my bedroom with a Portmaster and Redhat servers. I offered some unused bandwidth for a Tucows mirror and was later offered to join their pilot OpenSRS program.

Of course, there's many OpenSRS resellers today. Anyone paying a nominal fee can get setup to sell $10/year domain names purchased as bulk domain credits from OpenSRS. It's a great service that has really opened up hosting and registration for small businesses.

When pysoy.org was registered, I went with an OpenSRS reseller called IPower who was advertising domain registration at only a small margin above cost. The first trouble we ran into was, while the credit card was charged and control panel access granted, the registrations didn't actually get done on OpenSRS. It took 3+ hours on the phone with a tech support rep who finally had to re-create the accounts to get the registrations through their flaky system.

Their control panel was made of a series of icons, appearing to have been made using Windows Paint, with almost all of them not working because I wasn't getting hosting from them. After quite awhile of clicking into poorly named icons I found that it would not allow me to update the nameservers, but would allow me to setup custom zone files on their nameservers - with a limited number of subdomains.

Now I will say, this has been working mostly trouble-free since 2006. Last year my former partner, who was paying for these domains on his credit card, elected to remove his card from the account and we disabled auto-renewal. This year their notices for renewal ended up in my spam folder, and they ignored or reset the auto-renew settings, charging his card once again. He calls them, where they set him up as the authorized contact (note, his name only appears as the credit card owner which was removed a year prior, he is none of the domain contacts) in the agreement that he chooses not to challenge their illegal charges to his card and they agree to remove his card from the accounts. In short, it wasn't worth $30 to fight it.

Now, everything is amicable between us, but their willingness to add someone to my account without contacting me is a bit frightening. In checking my spam folder this morning I noticed some new emails from them stating that they would now also be charging $8.99 per domain for domain privacy, which was previously free and which payment was never agreed to, unless I opted-out by tomorrow.

Of course, calling them wouldn't be so easy, after three attempts to contact billing only to get put on hold for ~10 minutes each only to get dead air and a "click". After speaking to several reps, who all demanded the security code set by my former partner when he called (which I don't have) and over an hour struggling to get their control panel to give me new passwords, I finally get a security code set for one of the reps to speak to me. His tells me "a specialist" will get back to me about removing domain privacy.

I am very tired of this, and the truth is this only slightly worse than my experience with most domain registrars. I'm going to do what I should have done in 2006, near the end of the registration periods of all my domains I'm just going to transfer them to my own OpenSRS reseller account and pay the flat $10/year.

I'll extend this to any other free software project managers out there, if you're tired of dealing with crap like what I've gone through with IPower, I'll give you at-cost pricing on your domains. If your site is Python-powered (WSGI) I'll throw in hosting for a beer next time you see me.

My email address is my full name @ubuntu.com.

pysoy.org website upgrades

2009-06-15T17:15:00.002-04:00

As some have noted, we recently added an ad to PySoy's website. This is actually only one of the many changes to prepare for the beta3 release.

I'll be putting in a good deal of work over the next few weeks upgrading the website for HTML/5 and XMPP, better separation of functionality to subdomains and standardizing the templates between these subdomains and hg.pysoy.org and planet.pysoy.org/.

Help is, of course, always sought and welcome! If you're good with CSS or want some experience working with HTML/5 drop me a line.

Vala migration continues

2009-06-13T17:23:00.002-04:00

We're pretty far into the big Vala migration now, enough to talk about where we're going with it.

The most notable new component is libsoy, a glib-based library written in Vala that integrates all our dependencies to provide the non-Python functionality of PySoy.

This abstraction ensures that all PySoy's background threads remain GIL-free and will be useful for many implementations of "cloud gaming", which I'll go into more detail on later.

The largest task remaining is python-3.0.vapi, which will allow us to write the Python extension modules for libsoy. These will not be simple Python bindings which could be provided with PyBank as PySoy includes many complex Python datatypes which the gobject model cannot provide.

summer of code students announced

2009-04-21T01:34:00.002-04:00

It's been a stressful few weeks, but we've selected 30 students to work with the Python Software Foundation for Google's Summer of Code.

I don't know what I was thinking in volunteering to be the PSF's admin this year, but I'm happy to have had the opportunity. I believe we have more students working on core Python projects this year than all previous years and certainly a good start with getting students involved in our community.

There's a lot more work to be done; getting students setup on the SoC Python planet, getting all the backup mentors and coordinators organized, making sure students are taking advantage of the community bonding period, etc.

PySoy ended up with one student, Jo Yo (aka Lucas Westine) working on character animation. He worked with the Ogre3d team last year on a similar project. He's being mentored by Eric Stein (aka Toba) who wrote PySoy's soy.fields module as a Summer of Code student in 2007.

Building a better Pyrex - part 2

2009-04-01T21:47:00.003-04:00

Vala, the C#'ish language from the Gnome project for building GObject-based applications, may not seem like the most intuitive choice for building Python extension modules.

For PySoy, she may be our long-sought salvation from Pyrex.

First to note, PySoy is based heavily on GLib. While much of GLib is redundant to types and functions available with Python's c-api, the GLib functions don't require Python's GIL to be used. This is what allows PySoy to take full advantage of multiple cores and run Python code in parallel with rendering and physics processing.

Second of note, as I wrote in my last post, Pyrex/Cython are not appropriate for PySoy as an overwhelming majority of our code is inside with nogil:. Many Pyrex/Cython language features, what makes them easy to work with, are thus unavailable to us. The language becomes a burden and "GIL minefield" that makes working with it difficult.

Like Pyrex, Vala "compiles" clean object-oriented code to C, unlike Pyrex we have full and direct access to all datatypes and classes without the GIL. Like Pyrex, Vala needs pseudo-headers to instruct it how to use C libraries, unlike Pyrex the .vapi files give us an OO API for working with those libraries.

This was the real deal maker though; Vala is flexible enough to use PyObject and family directly. Not only do we get GObject access but with python.vapi, full OO access to Python's c-api with automatic incref/decref and easy Python/C type conversion.

so long, Pyrex!

2009-03-21T20:46:00.003-04:00

Almost four years ago PySoy started as a fork of a homebrew game engine called Soya3D. Despite being rewritten four times we were unable to find a suitable replacement for Pyrex, the meta-language Soya3D was written in.

Pyrex is alright for many purposes, but not for what we were using it for. Cython (a fork of Pyrex) has improved many things but not in the areas we need. Our "development" has ground into a series of tracking down bugs generated automatically in .c code because we didn't tell Pyrex not to do something, or because Pyrex's author didn't consider a certain use case when he wrote it.

After almost four years I've found a viable escape plan all thanks glib's GObjects and PyGObject.

The challenge that we've always faced is threading in PySoy without using the GIL (Global Interpreter Lock). 90%+ of the code in PySoy is only run in threads which never hold the GIL to ensure each background thread will only ever block on it's own functionality; rendering thread blocks on the GPU, physics thread blocks on rendering a scene, IO thread blocks in poll(), audio thread blocks on the sound card, etc. We've used glib's AsyncQueue to use Python callbacks from these no-GIL threads without that callback code interrupting what they're doing.

We used to believe that Pyrex made this all easy since the nogil C code that operated on a Python object could be written in a class-like manner with C attributes accessed like Python class attributes, ie:


cdef class Foo :
  cdef int alpha
  def __init__(self, value) :
    self.alpha = value
  def __call__(self) :
    return alpha

Like Soya3D, Pyrex's simplicity and elegance lures you in and so long as you don't try to, say, build a multithreaded game engine with it, it works great. Please don't misunderstand me, Pyrex is great for most applications that use it, we just out-grew it years ago and have wasted far too much time trying to make it work.

GObjects provides an even more elegant solution, write the C code in C, and write the interface code (import soy) in Python using PyGObject. The C code never includes Python.h and thus can never have any issues with the GIL.

So here begins our 5th revision of PySoy; function by function, type by type, extension by extension, refactored into C as GObjects. It'll take a bit to shift the code over in pieces without breaking everything as we do, but when this is done we can replace all the Pyrex source files with a Python package.

No custom languages for developers to learn, no more "with nogil:" everywhere, or having to refactor every C header we want to use to a .pxd file, or having to search through the C sources Pyrex generates for bugs.

so long, Pyrex! You made PySoy possible, but also caused us all countless hours of extra work and frustration. In retrospect, I wish we never met you.

sqlite and postgres support in Python 3 - today

2009-03-18T08:56:00.003-04:00

I've been working with the SQLAlchemy package in Python 3, their pre-0.6 branch anyway.

Currently it offers sqlite (included with Py3) and postgres (via py8000) with their Object Relational Mapper and lower-level SQL Expression Language.

My needs are covered given that sqlite and postgres are the only two databases I find myself normally using. Of course there are those who will balk, "but it doesn't support {'MySQL', 'Oracle', 'MSSQL'} yet!". To the people who need those databases, it's easier to spend a few days helping to port the appropriate dbapi package than wasting energy with complaints.

For web frameworks, where the multi-db abstraction SQLAlchemy provides is vital, it's available to develop on today (svn checkout, python3 sa2to3.py). They'll provide a separate -py3 tarball with the 0.6 release planned for after PyCon.

git vs mercurial

2009-03-08T14:51:00.002-04:00

In the past we've stuck with subversion for PySoy over git because about half our team are Windows devs and git has very poor Windows support. Some of us have been using git-svn to use git locally while not making life difficult for the other half.

Last night I migrated the Concordance-XMPP project to mercurial as a trial.

First, I have to say that Mercurial's subversion migration tools need work - simple things like author lookup tables (svn username -> Full Name <email>), option to turn off auto-tagging, and a template for modifying commit messages so we can turn off marking the history up with "[rXXX]".

These things were easy enough given that Mercurial is Python-based. Two hours later I had a proper migration.

In contrast with my earlier attempts at migrating to git, mercurial was actually a joy to setup on the server. WSGI, using our existing .htusers file from Trac, a small config script, done. This contrast to git which would require some funky SSH sandboxing and/or PAM setup - yes it can be done, but such hacks shouldn't be needed.

To all our surprise "hg clone" was extremely fast even compared to subversion and especially in comparison to git. One development commented that it seemed "instant". Even for the small commit history (2 months, about 130 commits) git could take several minutes to clone and process Concordance's history.

I'm disappointed to see GPLv2-only on Mercurial as we won't be able to use it directly in any GPLv3 or AGPLv3 projects. Upward license compatability good, anti-FSF paranoia bad.

Thankfully hg's protocol is easily derived and can be implemented by other Python software without having to actually import any part of hg. Hopefully the Mercurial's license situation will be solved at some point to minimize redundant work.

Avahi to be considered harmful

2009-01-23T17:46:00.003-05:00

Let me paint a picture as it's come into focus for me recently. Apple develops a protocol called mDNS for zeroconf via multicast DNS. They call it Bonjour and are serious enough about making it a standard they release a free software library called mDNSResponder to support it which runs on GNU/Linux, OSX, and Windows.

A community project, Avahi, implements their own mDNS suite for GNU/Linux. Avahi adds some nice features, such as dbus support. To help apps use either option they support compatibility with mDNSResponder's API.

It's great to have this sort of cooperative-competition, but they took this a bit too far by printing a nag warning when their mDNSResponder compatibility API is used by an application and otherwise pressuring mDNS-using apps to drop support for mDNSResponder.

I sincerely hope Python projects aiming to implement mDNS build against the mDNSResponder API, for cross platform support if nothing else, and encourage their users to avoid using Avahi when questions about this nag page come up.

Concordance register.py

2009-01-10T00:15:00.002-05:00

I have the main Python callback in Concordance working now to handle <iq/>, <message/>, and <presence/> elements. Eventually it'll only handle these elements when they're directed to the server, otherwise the server will route the message to it's destination without Python needing to process it first.

examples/register.py demonstrates user registration handling using ElementTree (or at least the start of the process for now). The result is shown here.

Concordance lives

2009-01-07T12:09:00.002-05:00

Concordance-XMPP now listens and responds to new XMPP connections with basic XML parsing via expat.

David has been working with me to maintain Windows compatability. I'm working on SASL support and the Python callback API.

With luck the first (very early) release will be ready this weekend which can handle logging in and intra-server routing between users.

Python 3.0 - first extension module

2009-01-02T00:36:00.002-05:00

After hours of hunting down docs and debugging my build environment (turns out I had a 3.0-alpha5 lurking around mucking up setup.py)..


Python 3.0 (r30:67503, Jan  2 2009, 00:20:15) 
[GCC 4.3.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import concordance
>>> concordance.__file__
'/usr/lib/python3.0/site-packages/concordance.so'
>>>

New Year, New Project

2009-01-01T17:22:00.003-05:00

The first day of 2009, PySoy's copyright notices updated and a few tickets closed, I hit a point where I really need a decent XMPP server.

I've started a new project called Concordance, a XMPP/Jingle server written in C (message routing and RTP mixing) and Python 3.0 (authentication, login, MUC handling, etc).

The goal is a framework giving Python full control over the server, for nearly unlimited configurability, while having the speed to serve a large user base with a minimal of CPU use.

great cheese frustration

2008-12-30T18:25:00.002-05:00

I overall like Gnome, but the guys working on Cheese appear to be making some horrible design choices.

Such as, they've removed any possible way to set the video input device, either via gstreamer's properties, or command line, or within the cheese app itself.

Now I'm hunting for another app that does video capture from a video4linux device, or possibly writing one using PySoy.

Beta3, new features on Ohloh

2008-09-13T11:03:00.002-04:00

So we're trying to get Beta3 out the door. No date yet, there's still a lot of cleanup work to do.

For smaller updates, Ohloh has a new "Journal" feature that resembles Twitter. What's really cool is once you associate your XMPP id with your Ohloh account you can use XMPP (via your normal IM client) to post to your journal.

Check PySoy's journal for smaller, more frequent updates on what we're doing.

bike accident

2008-05-22T14:19:00.002-04:00

I got side-swiped by a car Tuesday. Only injuries are some scraped skin and sprained wrist (I'm either really lucky or all too experienced in surviving bike accidents), but the latter has dramatically slowed by dev pace due to having to type with only one hand.

The sprain is mending well and i should be able to do light stuff (like typing) with it this weekend, provided I keep it immobile.

In other news, bump mapping is working again, but our blocks.py lacks proper normals, texcoords, and tangents. If anyone is willing to spend an hour or two working out the calcs for these please get in touch.

hacking at 3am, seeing rainbows

2008-05-20T03:35:00.003-04:00

There are some real advantages to hacking into the wee hours of the morning, this gem of a screenshot is one of the best of late.

Thing is, I'm sitting here after refactoring several hundred lines of code and fixing two typos that was causing our normalisation cubemaps to be bugged, staring at this window, and unable to understand why.

The rainbow effect is obviously from the normalisation cubemap, a data construct that maps tangent-space bumpmaps to object-space on the video card (without shaders). I just can't determine at this late hour why it's showing up instead of rendering the bumpmap as it used to.

I'm sure it'll make much more sense after 8 hours sleep. In the meantime, enjoy the screenshot, or compile PySoy from subversion and run TexBlocks.py yourself!

Mesh code refactoring

2008-05-17T02:26:00.002-04:00

I spent most of the day fixing, shifting, folding, documenting, and otherwise refactoring soy.models.Mesh and it's related datatypes and atoms. I'll likely spend most of tomorrow finishing this up.

In short, the render process we used was crude, lacked some really easy features, looped through 6 different classes for each Mesh (Mesh, MaterialList, Material, Texture, FaceList, VertexList) and had data management code in these plus the two related atom classes (Face and Vertex).

In short, not only was the code difficult to grok (even for me and I wrote much of it) but was slower that need be and increasingly difficult to expand. Game engines need both rendering speed (fps) and easy expandability, so this is a high priority ticket.

The new structure moves all of the actual Mesh data into the Mesh objects themselves; MaterialList is gone (to be replaced by MaterialDict later), FaceList and VertexList will now only store a reference to their parent Mesh to draw data from.

I'm going to be centralizing data management functions in Mesh as well, so the atoms can be much simpler and we can actually talk about optimizing the VBO layout. We'll also be doing dynamic passes, based on how many textures/pass a GL implementation supports, and supporting more textures targets (ie, specularity, emissive)

Again, sprint via IRC this weekend complete with remote peer programming via Gobby. Join us in #PySoy on irc.freenode.net

why can't XMPP keep their website up?

2008-05-17T00:39:00.002-04:00

Several times this month I've found www.xmpp.org/, the organization responsible for the Jabber protocol and it's many extensions, non-responsive. Their site has been down for at least 7 hours tonight, the longest yet (to my knowledge).

For our purposes I'll be mirroring their XEP's on pysoy.org as soon as they're back up. It's seriously frustrating to be working on this when the documents you need are often unavailable.

If anyone knows of an existing mirror we could use, please reply to this posting.

some real progress toward beta-3

2008-05-16T02:07:00.003-04:00

With Pyrex 0.9.8 released yesterday, including a number of key changes we needed to move forward, we're back in action for PySoy belated Beta-3 release.

The new forward declarations are a bit kludgy (example), and I don't think it'll be very maintainable in the long term. I think changing a class's inheritance should require editing one file, not two (the .pxd) or more (every .pxd that uses it).

It works for now, until we can wipe out the entire include/ directory, and to be fair it's only so cumbersome due to PySoy's size and how the classes are interconnected.

The nogil allowance for cdef methods is a massive step toward stomping out the multicore race conditions. One of the developers testing just the fixes to soy.bodies noticed the examples often running longer than before, though there's a lot more cleanup to be done before this is package-wide.

One of the roadblocks I hit is the nogil fix can't be used with our Children class, which stores pointers to Python objects of the same type and uses those pointers (through a typecast) to access their cdef methods in a for loop.

While our usage is GIL-safe, the current version of Pyrex still wants to throw in an INCREF just to be extra-cautious, which makes it not. I've let Greg know about this, we'll see if it can't be fixed soon.

In the meantime, I've called for an IRC sprint this weekend for the pending cleanup, refactoring, documentation, and testing that needs doing before beta-3. Join #PySoy on irc.freenode.net if you want to pitch-in!

Building a better Pyrex - part 1

2008-05-13T23:30:00.004-04:00

With our recent decision to drop the Pyrex codebase and start from scratch, we find ourselves repeating a process from 2 years ago;

When PySoy first forked off of Soya, we had a small list of things that needed to be fixed based on our negative experiences working with that project. The API needed to be cleaned up, pydocs added, Shape needed to be mutable and implemented with vertex arrays or VBOs rather than display lists, etc.

As we worked on it, that list grew longer, including rewriting large parts of the codebase, until we found ourselves at a point where starting over would be more expedient. Of course now there's little doubt that was a good move, learning from both Soya's mistakes and our own failed early attempts we now have a pretty rocking architecture.

The first and most foundational change we're making from Pyrex, as previously stated, is in the lexical scanner and parser. Both Pyrex and Cython are based on Plex, which was Greg's answer to processing a Python-like language.

In contrast, we'll be using Python's own tokenize module for our lexical scanner and an ASDL-based parser akin to the parser used in Python 3.0. We're extending Python.asdl with cdef, ctypedef, cimport, cinclude, etc.

This way we have less code to debug and a solid foundation to start with. With lexing and parsing almost "for free" we'll be able to focus on the important bits and get the platform able to support the kind of features we need quickly.

We've also reached a rough consensus on the following syntax changes from Pyrex:

unicode, generators, decorators, etc will be supported
all custom for syntaxes dropped in favor of Python's standard
with nogil: replaced with full with support
special methods will always have self as first argument (ie, both __sub__ and __rsub__ used)
cinclude added for direct C header parsing

There's more to come, which I'll post in additional installments as work progresses on this project.

setup-pyrex.py works with 0.9.7.2

2008-05-13T23:22:00.002-04:00

Greg fixed the latest bug that caused PySoy to raise a dict assignment error on import with 0.9.7.

Pyrex 0.9.7.2 was released a few hours ago with this fix, and I've updated setup-pyrex.py to work with either 0.9.6.4 or >=0.9.7.2.

Beta-3 is still blocked by the multicore race condition caused by redundant incref/decref calls on Python arguments to class methods in this latest version of Pyrex.

setup-pyrex.py now requires EXACTLY Pyrex 0.9.6.4

2008-05-12T16:16:00.002-04:00


16:02 < maacl> import soy
16:02 < maacl> File "LoopThread.pym", line 51, in 
               soy._internals.LoopThread.__cinit__
16:02 < maacl> TypeError: 'dict' object does not support item assignment
16:03 < Arc>     from threading import _active, _active_limbo_lock
16:03 < Arc>     self._active = _active
16:03 < Arc> [...] and line 51:
16:03 < Arc>     self._active[self._threadID] = self

I'm not going to even look into this further. Pyrex 0.9.7, for a reason I'm not going to even contemplate, raises an error about dict item assignment on the last line. None of us use it due to it's barfing of depreciated lines for all our for statements, but this brave tested did and reported the problem today.

I just committed an update to setup-pyrex.py that fails when any Pyrex version besides 0.9.6.4 is used.

subversion vs git

2008-05-11T11:59:00.002-04:00

I stayed up late last night working to install a shared git repository for the Pyrex replacement project. First of all, copious thanks to johnw and Ilari on #git for their help on this.

I'm very much an enthusiast of Trac, it's an excellent engine for project websites and has all the essential tools, builtin and via plugins, that any project should need. Ok, so there's a few things a project could need that isn't already written, but new plugins are easy enough (it's all Python, after all).

As such, I like to leave user administration to Trac, and it maintains a .htusers and .access file local to each project for that purpose. Likewise, for SVN, we use WebDAV on Apache, whereas the ssh+svn method would require adding those users to the local system or PAM hacking.

Thus, the most direct path would seem to install GIT via WebDAV and run it from there. As Ilari pointed out, GIT's DAV has many problems over SVN, from lockups to a failed push sometimes corrupting the shared repository. I tried anyway, with hours of help from joshw, and at 5am I had to throw in the towel. I'm sure it's possible, but not with my current lack of GIT knowledge.

The other available solutions could work, but first I'd want to write some new Trac plugins for accepting developer ssh keys and using these on the backend to govern access or figure out the PAM configuration for .htusers to have restricted SSH access for GIT pushes.

There are certainly advantages to GIT, but server setup difficulty and added complications for Windows developers leads me to sticking with subversion servers with git-svn client-side for now.

a radical redirection

2008-05-10T19:28:00.002-04:00

We've been at a deadlock for a few weeks now, our previous coding pace has turned into a lot of contemplation and side discussions about the greater issues with the PySoy codebase.

I think we have a roadmap forward, or enough of one to post these in a blog entry:

First, PySoy 1.0 will not be released until after Python 3.0 this Fall, and it would be pointless to continue developing with 2.x in target. Python 3.0 also has a large number of useful changes for our project. Our target platform should therefore be 3.0 (and it's alpha/beta's for now).

Second, with the release of Pyrex 0.9.7 Greg has, once again, introduced a major language change (the "for" statement) without even soliciting input from the community. This has sealed the deal for us, even if our immediate issues are fixed we cannot continue to subject ourselves to a language that changes in incompatible ways between 2 micro versions (0.9.6 -> 0.9.8, when the old syntax will not be supported).

We earlier started a Cython variant to support PySoy's codebase, but given recent developments in their community, the timelines of our projects, and their choice to stick with Python 2.x for now, we're moving forward on a Pyrex replacement written from scratch and targeting Python 3.0.

We're going to use a similar language style to Pyrex, Greg certainly had some good ideas, and will keep language porting in mind, but there will be changes. We will also not be using any of his code including his Plex module, Pyrex and Cython's lexical analyzer which is the core of those packages.

PySoy Beta3, earlier aimed for release in "early Spring", has been indefinably postponed as we work on that. The subversion repository is of course open, and development can and will continue, but we cannot release until the new build system is complete.

initial impressions of Trac 0.11

2008-05-02T13:26:00.002-04:00

Trac 0.11rc1 was released this week, marking a feature freeze for 0.11.

One of the major improvements over 0.10.4 is the Genshi template system allowing complete control over a site's HTML. For example, you can copy the default layout.html to your site's templates/ directory to remove the obnoxious "Trac" string at the end of every page's title and the other Edgewall advertising splayed all over a default Trac site.

Of course, this is also means a Trac site can be radically redesigned in structure without editing the source code. Hopefully this leads to a greater diversity in project site design.

Another major feature is ticket workflows are now customizable, allowing projects using Trac to decide how their tickets will be processed. For example, a "testing" state could be added to signify a ticket as "though to be complete, but waiting for verification".

I plan on upgrading pysoy.org this weekend along with a few new Trac plugins and a minor site redesign.

some race conditions found

2008-04-02T02:01:00.002-04:00

Well after many weeks we found the source of the race conditions on multicore CPUs; they were not in PySoy at all but in the Pyrex compiler.

Throughout the generated .c we found INCREF/DECREF where it didn't belong, namely in every cdef method for __pyx_v_self. In other words, even though the calling function must already hold a reference to instance in order to access it's C methods, thus preventing the instance from being garbage collected during method execution, the Pyrex.Compiler inserts redundant INCREF/DECREF calls.

These would normally just waste CPU cycles, but since our 3d engine has non-GIL background threads these refcount adjusting calls are accessed in a thread-unsafe manner and will inevitably cause either a segfault or a memory leak.

The solution is to not consider "self" a normal Python argument, which would also allow these C methods to be called with nogil: set, provided checking is done to ensure no Python methods, properties, or variables of self are accessed.

We'll be fixing this problem with our new source compiler, to be announced soon.

1.0_beta3 release and a Sprint this weekend

2008-03-21T03:43:00.000-04:00

There is a rumor going around that because our 1.0_beta3 milestone is marked as due today (3/21) we're planning a release today. This is false, we'll be releasing sometime in early spring as advertised - the due dates marked for our milestones are our preferred date only.

We're still tracking down one or more nasty race conditions which only show up on multicore systems, we still have a bit more work to do before the next release, and a good deal more testing is needed after these bugs are found to demonstrate stability. If you're experience with debugging we could use some help - some of our tests seem to indicate mutex locks being ignored.

This weekend is a good time, too. Starting today (Friday) we're running a pre-Beta3 online sprint. In our IRC channel (#PySoy on irc.freenode.net) you can find a good number of developers working almost around the clock on various parts of the 3D engine and a few working in Gobby for remote pair programming.

If you're not into debugging, or don't have a multicore system, there are many other tasks - large and small - that need doing.

State of the Soy @ r1000

2008-02-27T06:59:00.002-05:00

Last night our humble 3D game engine reached 1000 commits to it's Subversion repository.

PySoy has been contributed to by 20 developers, 7 of which are currently active. The project's IRC channel is in nearly constant use by developers working on different tasks.

The external API has drawn considerable praise by users, despite it's state as beta, and the constructive feedback we've received has led many of the post-Beta2 API changes.

The internal API, while having some rough edges, has been commented on by several new developers as remarkably clean. We've passed the point which the COMPLETED list exceeds the TODO list, despite the latter constantly growing, and focus has largely shifted toward debugging, documentation, and polish.

Most notably, we've reached the point of needing a legal entity for the project. Using the Software Freedom Conservancy as a rough template, we've started the process to form a non-profit corporation with a focus on copyleft game projects. An announcement about this will be posted as more details are firmed up.

PySoy goes from 3 threads to 5

2008-02-26T09:11:00.005-05:00

We're doing the final work now to replace CoreLoop with TransportLoop, WindowLoop, and SceneLoop. All three are instances of soy._internals.LoopThread which (should) play well with Python's threading system and should never hold the GIL.

The increased performance this has brought for multi-core users is amazing, some have reported more than twice the framerate!

To be fair, an equal number of subversion Trunk users are experiencing frequent segfaults since we've not finished adding mutexes and rearranging the internal API post-move. Obviously these issues are being resolved prior to the Beta-3 release next month.

why we're blocking Microsoft Live

2008-02-20T14:43:00.003-05:00

While going through server logs I noticed something funny, Microsoft Live has surpassed Google in the number of hits we've received. Weird, eh? I didn't think anyone actually used it.

The terms people arriving at our site through search.live.com are just... weird, though. Most are outright vulgar, searching for obscure pornography or celebrity names, drugs, sex aids... Here's an few examples:


65.55.165.36 http://search.live.com/result.aspx?q=valtrex&mrt=en-us&FORM=LVSP
/log/trunk Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR
1.1.4322)
131.107.0.95 http://search.live.com/result.aspx?q=breast+enhancement&mrt=en-us&FORM=LVSP /
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; Win64; x64; SV1)
65.55.165.122 http://search.live.com/result.aspx?q=ferarri&mrt=en-us&FORM=LVSP
/browser/media/tutorials Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2;
.NET CLR 1.1.4322)

These hits were to small obscure pages on our site, such as svn changelogs, and then I noticed the source IP addresses were in just a few IP ranges, so I ran a whois to see if one ISP tied connected them all;


arc@sobek ~/work/pysoy $ whois 65.55.165.83

OrgName:    Microsoft Corp
OrgID:      MSFT
Address:    One Microsoft Way
City:       Redmond
StateProv:  WA
PostalCode: 98052
Country:    US

NetRange:   65.52.0.0 - 65.55.255.255
CIDR:       65.52.0.0/14
NetName:    MICROSOFT-1BLK

You read that correctly. Microsoft, in a desperate attempt to make themselves seem more important, or perhaps just to flood free software project's websites with unwanted traffic, is running bots which act like normal web crawlers. Indeed, over 97% of the hits we got from search.live.com were from Microsoft's own IP subnets. Searching Google, I found this story was previously covered by others more observant of their logs.

In response, I'm adding a special rule to block all future traffic from the offending netblocks, including MICROSOFT 131.107.0.0 - 131.107.255.255 and MICROSOFT-1BLK 65.52.0.0 - 65.55.255.255.

libjingle disappointments

2008-02-14T20:23:00.002-05:00

Last year we chalked libjingle as our choice for a XMPP library, since it handled all the networking we needed (ICE-UDP, HTTP, etc) tightly integrated with XMPP through a standard protocol.

On closer inspection, we found why libjingle hasn't really been adopted by the community yet. For starters it requires patches to build the latest release with GCC 4, and no release has been made to include these patches in months. Not a good sign.

Next we find there are no dynamic libjingle libraries, it builds as static only, and doesn't build for C programs (only C++). Unless we transformed our project to build as C++ this isn't an immediate option.

Looking over the libjingle code structure, it's in a severe state of poor maintenance. The internal name for it appears to be "libcricket", with naming confusion all over the place, code format inconsistencies, and extremely poorly documented.

Thus, libjingle integration is unlikely until after PySoy's Beta-3 release when we'll have time to write the C shim code as a dynamic library.

All this said, libjingle appears to have great potential for us and the entire free software community, it's just in need of attention from a few detail-oriented developers.

consolidated _core-*/ to _core/

2008-02-11T02:19:00.000-05:00

Last year Pyrex, then version 0.9.5.1a, did not support compile-time conditions for things such as platform specific code. To work around this used different _core directories, one for each platform, and a platform test in setup.py would determine which directory soy._core was compiled from.

While functional, this created duplicated work and a fairly messy source tree with four different _core-* directories. With Pyrex 0.9.6 release came support for code such as this:


IF UNAME_SYSNAME == "Windows":
  include "icky_definitions.pxi"
ELIF UNAME_SYSNAME == "Darwin":
  include "nice_definitions.pxi"
ELIF UNAME_SYSNAME == "Linux":
include "penguin_definitions.pxi"
ELSE:
    include "other_definitions.pxi"

... and, thus, we could finally merge those _core-* directories into src/_core and remove kludge from setup.py. Tonight that's exactly what I did. Behold, http://svn.pysoy.org/trunk/pysoy/src

website makeover and new logo

2008-02-06T01:58:00.000-05:00

We've been working with the same website design (www.pysoy.org) for a year now. While it got the job done, we're in need of a makeover - and that's exactly what's going on. The basic layout from Planet PySoy is going site-wide with a new docs section, info on how to get involved with development, separate bug and task/idea/etc filing forms, and a ton of new features that will make the site more community-oriented.

In the spirit of community inclusion, I'm also opening the new logo for collaboration and competition for. Draft logos can be worked on in http://svn.pysoy.org/media/logos with the final designs voted on by all active developers near the end of this month. We'll aim to include this in the beta-3 release next month as a special mesh and texture.

normals (dot3) texture

2008-02-01T22:39:00.000-05:00

With a few minor bugs left to be worked out, we now have normals mapping. This was written by Jaroslaw Tworek, a high school student in Poland who also added recently volumetric fog and billboards. These features and more will be included in the Beta-3 release next month.

Bump Mapping; yet another feature PySoy has over it's predecessor.

downtime and new server

2008-02-01T05:27:00.000-05:00

As many people have noticed, *.pysoy.org was down yesterday. Since our former server is still offline (thanks, Verizon) I've paid out of pocket for a new colocated server hosted at ServerBeach.

After an all-nighter getting it setup, www.pysoy.org is back up, as is svn, and hopefully everything else. I'll resolve any problems tomorrow, as I'm in serious need of sleep now.

PySoy Beta-2 response!

2008-01-02T15:11:00.000-05:00

12 hours after the PySoy Beta2 release we've had over 120 downloads, about half of which for Windows.

Some users were quick to point out we forgot three dll's in the installer; glew, iconv, and zlib1. We're rebuilding the installers for Windows to include these and testing to ensure we didn't forget anything else.

PySoy Beta-2 Released!

2008-01-02T04:30:00.000-05:00

We finished up the spit-n-polish work tonight and shipped the second "beta" of our 3D game engine for Python!

Included is support for Microsoft Windows (with nice bdist graphical installer), keyboard input and an early draft of the controller-action system, text rendering thanks to Cairo, support for Joints, and other enhancements and fixes too numerous to list.

While over a dozen developers contributed to this release I'd like to especially thank Kirk McDonald for porting the engine to Windows API, Derek Rhodes for getting the Cairo support in, Eric Stein for his work from Summer of Code and since, and Piet Delport who provided countless hours of help to all of us with Pyrex-related issues.

We're far from done so please download, enjoy, give feedback, and if so inspired help us reach 1.0!

r600

2007-12-29T00:07:00.000-05:00

A quick update to INSTALL (dependencies list) brings us up to r600. This makes about 50 commits in the past month.

... of course with the sprint ongoing we're likely to reach r750 by PySoy's Beta-2 release next week.

fresh soymilk!

2007-12-27T14:18:00.000-05:00

While this is a blog about PySoy, a 3d engine for Python, this post is about a soy related product I recently fell in love with.

It's called the SoyQuick soymilk machine. Basically a pitcher, heating element, grinder, filter screen, and some sensors in one package.

You throw in 2/3 cup of soaked soy beans, fill water to a line in the pitcher, plug in a standard power chord (same as computers use), press the start button, and 15 minutes later fresh soymilk in the pitcher and okara in the filter basket.

This awesome machine plus an ample supply of Special Gunpowder Green Tea is what's fueling me through the PySoy Beta-2 release process.

Beta-2 task assignments

2007-12-25T21:09:00.000-05:00

Looking at beta2 tickets there's an extremely disproportionate number of tickets being handled by yours truly.

If you can participate later this week or as part of the sprint, please take a look through what needs to be done and volunteer yourself for some of the workload. Many of these unassigned tickets almost anyone can do without previous experience with PySoy.

Sprinting to Beta-2

2007-12-20T14:59:00.000-05:00

What a better way to kick off 2008 than with the long-awaited Beta-2 release of PySoy?

We've been lazy though, very little has been committed since Beta-1's release a few months ago. To play catchup and get us focused enough to reach this goal a sprint has been scheduled for the last weekend of 2007 (Dec 28-30).

The physical site for the sprint is in Manchester, NH. A bank of public IPs (no NAT), caffeine, desktop space, and inflatable crash mats await those able to make it. RSVP for the address.

For those working from home we'll have IRC and video conferencing throughout the weekend. If you're planning to participate join #PySoy or get your webcam working with Ekiga.

Even if you have no experience with C programming or OpenGL we need people for documentation, writing examples, and unit testing. Being inexperienced is an advantage for locating inadequately documented classes and filling in the gaps!

Every serious participant will be mailed a silk-screened "PySoy Sprint 2007" patch. Sews easily to a backpack, jacket, etc to let the world know that you helped make the PySoy revolution!

dependencies upgraded

2007-11-09T22:08:00.000-05:00

Pyrex 0.9.6

There's a few API changes which result in warnings, we're going to keep things the way they are now until Ubuntu and a few other distributions have this version available, or 0.9.7 is released, whichever comes soonest.

ODE 0.9

ODE's new version comes without an updated changelog, however, a diff on the include directories shows very few API changes from 0.8.

Because Ubuntu 7.10 just came out with ODE 0.8 we're going to stick with ODE 0.8 compatibility for the time being. Those previously trying to install PySoy on Ubuntu may remember having to manually install ODE 0.8 as the last packaged version available for Ubuntu was 0.5.

Theora

We may need to update for libtheora beta2 since there were numerous changes since alpha7, however, beta2 is not packaged for most distros yet (ie, both Gentoo and Ubuntu only run alpha7). It would seem prudent to stick with alpha7 for now.

Server move

2007-11-04T02:39:00.000-05:00

The server hosting *.pysoy.org has been moved from Ithaca, NY to my new home in Manchester, NH.

It's now on a much faster connection (I've clocked over 850k up) and a decent UPS to reduce downtime to near-zero. I used the switchover time to upgrade the software and tighten security a bit, too.

Sorry for the downtime, it'll be worth it in the long run.

Maintainer's Update

2007-10-02T15:46:00.000-04:00

It's been awhile since I've posted so a quick update;

We've got two new developers, KirkMcDonald and DerekRhodes. Kirk has taken over the Win32 porting and Derek is working on materials/textures.

Toba has remained active after Summer of Code. He's also the only student to complete his SoC project and thus receive full payment.

I've been in the process of moving so my free time for PySoy has been limited. I'm generally available via IRC for questions and, now living in Manchester NH, will be able to pound out some code again once part of my studio is setup here. It's going to take me months more to move it all.

We'll have Beta-2 out by the end of the year at the current rate with Win32 compatibility, input handling, and model+texture loading/saving.

beautiful mistakes

2007-07-13T14:17:00.000-04:00

Fullscreen and splash window (ie borderless) support is in. I ran several systems overnight with a script which flips between fullscreen and windowed every second to ensure there's no threading issues with the new code.

In the process I generated this great mistake, basically what happens when you don't clear the background while rendering the collision blocks. We're going to have to do more complicated mojo to make splash windows "transparent".

Overall work on the engine is on-track for Beta-2.

Now on to dynamic text textures

Beta-1 Success & Bugs

2007-07-12T02:23:00.000-04:00

A mere hours after Beta-1's release bugs started getting found. This was entirely expected.

The largest problem people have had is with installing ODE 0.8 on Ubuntu. For whatever reason the last packaged version available to Ubuntu users is 5. Installing from source means editing /etc/ld.so.conf to add /usr/local/lib and running ldconfig, something a bit scary to new users.

Overall positive comments from testers though. It's great to get this sort of confirmation even with the engine so incomplete. Of course Beta-2 with input support and mesh loading will be much better.

A month off from that point, here's another screenshot.

power outage

2007-07-10T01:45:00.000-04:00

Ack. Thanks to a long term power outage the svn + website server was down for several hours today delaying work. With luck we should have that release out tomorrow (7/10).

Three tickets remain

2007-07-09T12:33:00.000-04:00

The last of the beta-1 tickets have been delegated. Hopefully they're easy ones.

Then we start with the final check to make sure nothing obvious segfaults.

the day before beta-1

2007-07-09T04:41:00.000-04:00

Today we saw one of the largest recent API changes come to pass, "mesh" is now something that can be applied to a Body, not a type of Body itself.

soy.bodies.Pyramid is gone, there is now a pyramid.py in tests/ which provides it for the test scripts. I also copied over the blocks.py file from Blockdropper (one of my old Soya games) and updated it's API for our engine. That is the screenshot I included here.

There are a few tickets remaining for Beta-1, but barring some last minute critical bug discovery we should be set in the next 16 hours.