XMPP on the web

A short thread on G+ has prompted this longer sharing of my vision for XMPP on the web.

For XMPP use on a website we currently have BOSH and, in an extreme-alpha state, XMPP over websockets. The advantage of websockets is obvious, BOSH is a high overhead protocol that we'd all rather not have to use, however both have the same problem: you either must share your login credentials (and thus access to your account) with every website you use a single account with, or must create a new account (JID) for every XMPP-based website you use.

Oauth2 for XMPP might be a partial solution to this by requiring your authorization through a central identity site and using the resulting token for logging in, however, you're still opening yourself up to the 3rd party website accessing your roster, sending spam messages on your behalf, and potentially worse. All this really gives you is the ability to later disable access to websites who misuse your account.

This is the crux of the issue: when using an Javascript library provided by a website and using a proxy provided by that website, whether BOSH, websockets, or otherwise, you're giving that website unlimited access to your account. I have not seen a workable proposal to solve this and until this is solved XMPP cannot see widespread use on the web.

I'm proposing that we solve this by putting XMPP in the browser, either directly or through a plugin. Expose a standard javascript API for allowing websites to use an XMPP connection along with a security model which gives users control as to what a website is allowed to use their connection for. Ie, if a script on a website wants access to their roster the user will be prompted for it, if they want to join a MUC room display a standard prompt for that. Browsers can have multiple XMPP sessions at once and allow the user to select which account they'd like to use with an XMPP-enabled site.

This is just some early ideas, I'm nowhere near implementing this though I think the conversation would be useful to get started.

OpenGL ES support complete

The experimental branch, where the OpenGL ES migration was being done, has just been closed and merged into the default branch of PySoy.

Thanks to Steve Anton, one of our Google Code-In 2011 students, for some of the last bits of work to complete the merge. We are now one step closer to mobile support! If you're a student ages 13-17 and would like to earn a Google tshirt and some cash by helping us with Android support, sign up for Google Code-In and claim this task.

Google Code-In 2011 is Open

Google Code-In has officially begun!

From today through January 16th students age 13-17 can earn up to $500 working on small tasks for software projects such as MoinMoin, SymPy, and PySoy!

Tasks include coding, documentation, graphic design, video production, testing, translation, research, public speaking, and many other kinds of challenges of varying difficulty.

Completing just one task earns a student a Google tshirt. Every 3 tasks they complete earns them $100, and the 10 top students worldwide will earn an all-expense paid trip to Mountain View, CA to receive an award at Google.

PySoy has over 75 tasks offered for the first half of the program and another 75-100 will be made available December 16th. Our mentors are on Freenode channel #PySoy ready to help students start earning their tshirt and cash today.

Sign up today and get started!

Rugby season nearly over, getting back to work

Wow its been a long time.

We wrapped up Google's Summer of Code 2011 in August. The Python Software Foundation did wonderfully overall, for PySoy 6 of our 7 students passed. A great year overall - thanks to all the mentors and students!

Rugby has been a life changer for me. My first game was in September, after floating in and out of practice for years and training pretty heavily since April. No serious injuries, but no shortage of pain; I've frequently needed to sleep in a reclining chair to keep blood from pooling in my shoulders and nurse bruised ribs, dislocated fingers and toes, shin splints, and pulled muscles everywhere. All so worth it.

These guys are like family to me. I know it sounds sappy, but I've come to trust the men in my pack with my life - in a way we all do every time we bind onto each other a scrum. Its not that big of an adjustment culturally though due to the large number of programmers, lawyers, and IT professionals on the team. When you work behind a desk all day its nice to balance it out with a physically intensive training in the evening and games on Saturday.

The climax of the season was Hellfest October 29th in Dallas, TX. Washington Renegades brought our B-side to compete and returned with the 1st place trophy. My teammate Jimbo has more pics on his blog of the tournament, I was wearing #23 as tighthead prop.

We have two more games this season before we settle in for the Winter and indoor off-season training at the gym. A group of us plan to do a 8-week program run by a professional rugby player this Winter to get ready for the Spring season and the Bingham Cup 2012 in Manchester UK next June.

Today the PySoy project was accepted to Google Code-In. We've got a number of student tasks lined up, with many more being worked on for the first batch set to release in less than two weeks. Interested students should hop on Freenode (#PySoy) and get oriented before the program starts so they're ready to jump right into their first task!

Transcoding FLAC to Ogg Vorbis

Last night I hit a dilemma; a very old CD I ripped to FLAC and now can't find was playable only on certain players, not my Android phone (despite FLAC support) and behaving strange on many desktop players.

Usually I just use something like oggenc -q 4 *.flac since vorbis-tools supports FLAC as a source format (and preserves metadata like artist, title, etc). Strangely, oggenc didn't recognize the files in this album.

GStreamer to the rescue; for file in *.flac; do gst-launch-0.10 filesrc location="$file" ! decodebin ! audioconvert ! vorbisenc quality=0.4 ! oggmux ! filesink location="$file.ogg"; done;

Even though its much slower and more complicated, this should have done the trick. It didn't, and ogginfo showed that the Ogg muxer in GStreamer has some issues;

WARNING: granulepos in stream 1 decreases from 218558 to 205632
WARNING: granulepos in stream 1 decreases from 666174 to 655680
WARNING: granulepos in stream 1 decreases from 3150206 to 3142208
WARNING: granulepos in stream 1 decreases from 3605054 to 3597376
WARNING: granulepos in stream 1 decreases from 3828670 to 3822400
WARNING: granulepos in stream 1 decreases from 4741630 to 4733312
WARNING: granulepos in stream 1 decreases from 5636158 to 5630784
WARNING: granulepos in stream 1 decreases from 5858494 to 5854208
WARNING: granulepos in stream 1 decreases from 6096126 to 6090560
WARNING: granulepos in stream 1 decreases from 6317758 to 6314048
WARNING: granulepos in stream 1 decreases from 7668798 to 7665088
WARNING: granulepos in stream 1 decreases from 7902718 to 7898240

So with GStreamer not an option, I looked at the original FLAC files and found that whatever encoder I used added ID3 tags (which are not part of the FLAC spec). A quick ID3 removal command stripped these out so oggenc would recognize the files and work;

find . -name "*.flac" -exec id3v2 --delete-all {} \;
oggenc -q 4 *.flac

Done.

making an Android/iPhone/iPad home screen icon for your website

One of the cool features of new phones and tablets is the ability to drop a website bookmark as an icon on the home screen, as if it was a normal App for that device. Many websites provide icons which are automatically used when a user bookmarks them, however, the tools and resources for doing this are not immediately obvious. I just spent Memorial Day figuring this out and will share what I found here.

First, design your logo. I use Inkscape so I can later scale the icon for tshirts and printed material, but any tool will do as long as you end up with a square flattened 512x512 PNG. Open this in GIMP.

Next download this GIMP template and open it in GIMP. Copy/paste your icon into the already selected region of the template, it should "just work". Anchor your selection and merge visible layers for your 512x512 icon.

Now that you have your glossy icon with rounded edges, scale and save the image three times for 57x57, 72x72, and 114x114 as follows:


apple-touch-icon-57x57-precomposed.png
apple-touch-icon-72x72-precomposed.png
apple-touch-icon-114x114-precomposed.png


Also save your original (pre-template) logo at 57x57 as apple-touch-icon.png. These four files should cover iPhone, iPad, Android, and Blackberry. You may also want to save either the original logo or the glossy version at 16x16 for favicon.ico if you have not done so already.

Upload these files to your web server saved in the root directory of your website (like robots.txt) and add the following template to the head of your website:


<link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" />
<link rel="apple-touch-icon-precomposed" sizes="114x114" href="http://yourdomain.com/apple-touch-icon-114x114-precomposed.png" />
<link rel="apple-touch-icon-precomposed" sizes="72x72" href="http://yourdomain.com/apple-touch-icon-72x72-precomposed.png" />
<link rel="apple-touch-icon-precomposed" href="http://yourdomain.com/apple-touch-icon-57x57-precomposed.png" />
<link rel="apple-touch-icon" href="http://yourdomain.com/apple-touch-icon.png" />


That should take care of virtually every device that supports this feature. Test it out on your Android phone and, if you have any friends in the Cult of Mac, ask if you can check this on the half dozen iOS devices they own.

If you want to really encourage iPhone/iPad users to take advantage of your new, shiny icon to bookmark your webapp on their Home screen, check out this javascript library called Mobile Bookmark Bubble.

Of course you should also make sure your site works well on handheld devices, but that's a topic for another time.

Have fun!

Nodetree has Document and root Element

I think an example will explain the emerging API far better than I can.

You can replicate this with the code from http://hg.concordance-xmpp.org/nodetree, though its still very early the code is completely documented in its current state.

Python 3.2 (r32:88445, Feb 28 2011, 00:50:14)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import nodetree
>>> doc = nodetree.Document()
>>> foo = nodetree.Element('foo')
>>> doc.root = foo
>>> len(doc)
1
>>> print(doc)
<?xml version="1.0"?>
<foo/>

>>> bar = nodetree.Element('bar')
>>> doc.root = bar
>>> print(doc)
<?xml version="1.0"?>
<bar/>

>>> print(foo)
<foo/>
>>> print(bar)
<bar/>
>>> print(doc[0])
<bar/>
>>>


Nodetree design

I've spent the last two weeks charting out Nodetree, a XML data binding package for Python 3.

I'd rather not reiterate the multitude of reasons for this, but since they always get asked, I'll summarize this with needing a stream parser that supports xpath/xslt and passing chunks of an XML stream in Python, something that ElementTree and lxml do not support. SleekXMPP employs a fairly ingenious hack on lxml to achieve this, but I'd rather spend some time doing it right from the start for Concordance.

Since XPath and XSLT are best supported by libxml2 than any other free software library, and I'd rather not spend the next few months writing an implementation from scratch, the data must exist as a libxml2 DOM tree at the point of processing.

You can get a libxml2 DOM tree either by parsing a file (which gives you little control, it parses the entire file one-shot) or build the tree node by node. libxml2 also supports processing an XML stream through its SAX interface, node by node, which gives us the flexibility needed to pop and/or parse nodes from the stream as its being parsed while still having all the other tools we need by generating a DOM tree of the segments as we process them.

The problem here is that Nodetree isn't a DOM interface, its a Pythonic XML data binding interface, so such things as adding the same element to two documents (or being able to create a new document using a piece of another document while both are in memory) is a challenge. Even if we were not using libxml2/DOM, for XPath to work correctly every node must have a clear hierarchy which falls apart when your context is a node used in the same document three times and in two different documents.

One of the benefits of working on a wide variety of projects is reusing clever solutions used on one project for something completely different. For example, I recently implemented PySoy's atomic API whereas Python objects are created on the fly for underlying data structures and could be attached to multiple points of data (see ColorCanvas.py for example). This allows multiple data points to be updated with one step, and saves us from doing silly things like storing (ie) an object for every pixel in an image.

I'm implementing almost exactly the same mechanism for Nodetree, it'll limit XPath searches run from Python somewhat but should be fully XML compliant and Pythonic, something (IMHO) nobody has managed to do yet.

PyCon 2011 wrapup

This has been an awesome week at PyCon, though I'm completely exhausted.

The most notable outcome of several meetings with Python leadership about increasing diversity in the Python community. In my role with PSF's educational outreach programs I'm calling on 3rd party projects working with us to help work to increase gender balance in our community, and as PySoy's project maintainer I'm committing us to working seriously on this issue over the next year as an example.

Several college-aged women at PyCon knew about Google's Summer of Code but believed it was only for "geniuses who've been programming since they were 13" or that they were unqualified for other reasons, even though they're clearly more skilled (and moreso, have a better attitude) than some of the male students we get.

To address this, my first step is outreach to women for applying to us for Google's Summer of Code in hopes of resolving any miscommunication over required experience and what we're looking for. This is being sent out over the next week so potential applicants leverage of early involvement with mentors that many students have already started on.

Once Summer of Code is underway I plan to continue this an internship program in attempt to get more women involved in our community in ways not supported by GSoC such as through QA testing, documentation, video tutorials, and artwork. This program will be for women ages 18+ with no college requirement, though we're still working out many of the other details.

For the PyCon sprints I worked on the new "packaging" package for Python 3.3, formerly known as distutils2. My primary interest in this is a packaging.backport module based on 3to2 (which backports code from 3.2 to 2.7) extended with further fixers for 2.7 to 2.6, 2.5, and 2.4. This module will work for packages written for Python 3.3 and above to maintain Python 2 compatibility, and also to backport "packaging" to the standalone distutils2 package for previous Python versions.

We managed to get distutils2 largely upgraded for Python 3.3 syntax and the non-PyPI tests running cleanly, though much more work is needed. We have a few months given than Python 3.2 was just released and we're working on a roughly 18 month release cycle.

Sprinting toward Beta-3

We've made some massive headway in the last two weeks;

First, we have physics processing back. We're using ODE for now just to get us through the next release cycle or two, but we've modified its headers (ABI-compatible) to work better with libsoy. Done are the days when we stress over API compatibility, since we've decided to write our own fully integrated (and scaled down) physics processing with Orc we have a certain freedom to throw their conventions to the wind.

How this is going to play out is a slow migration of values to arrays. Body data, for example, will be stored an array inside a scene so SIMD processing can be used more efficiently. The Soy Body class will represent a pointer to that array and include properties and methods for working with its data. The actual storage of that data will be governed by the Scene, giving it the freedom to sort bodies per "island" (groups of bodies connected by group). This is similar to how ODE processes bodies already and we'll be likely using their code as a starting place.

Second, we have the basic types for rendering. At the moment this is just a rotating cube, but that involved roughly 6 classes operating in both GObject and Python domains. The next step is soy.models.Mesh for rendering more complex shapes.

Third and most exciting, our humble game engine can now run as a Firefox plugin! There's some bugs to be worked out still, but "embedded" support has been added to our soy.widgets.Window class that allows it to run inside virtually any application.

With luck and determination we'll have the next release out by the end of the month. I've dialed back our goals considerably given that its a rewrite from our last beta release but our release cycle will be much shorter now that the rewrite is finishing up.