making an Android/iPhone/iPad home screen icon for your website

One of the cool features of new phones and tablets is the ability to drop a website bookmark as an icon on the home screen, as if it was a normal App for that device. Many websites provide icons which are automatically used when a user bookmarks them, however, the tools and resources for doing this are not immediately obvious. I just spent Memorial Day figuring this out and will share what I found here.

First, design your logo. I use Inkscape so I can later scale the icon for tshirts and printed material, but any tool will do as long as you end up with a square flattened 512x512 PNG. Open this in GIMP.

Next download this GIMP template and open it in GIMP. Copy/paste your icon into the already selected region of the template, it should "just work". Anchor your selection and merge visible layers for your 512x512 icon.

Now that you have your glossy icon with rounded edges, scale and save the image three times for 57x57, 72x72, and 114x114 as follows:


apple-touch-icon-57x57-precomposed.png
apple-touch-icon-72x72-precomposed.png
apple-touch-icon-114x114-precomposed.png


Also save your original (pre-template) logo at 57x57 as apple-touch-icon.png. These four files should cover iPhone, iPad, Android, and Blackberry. You may also want to save either the original logo or the glossy version at 16x16 for favicon.ico if you have not done so already.

Upload these files to your web server saved in the root directory of your website (like robots.txt) and add the following template to the head of your website:


<link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" />
<link rel="apple-touch-icon-precomposed" sizes="114x114" href="http://yourdomain.com/apple-touch-icon-114x114-precomposed.png" />
<link rel="apple-touch-icon-precomposed" sizes="72x72" href="http://yourdomain.com/apple-touch-icon-72x72-precomposed.png" />
<link rel="apple-touch-icon-precomposed" href="http://yourdomain.com/apple-touch-icon-57x57-precomposed.png" />
<link rel="apple-touch-icon" href="http://yourdomain.com/apple-touch-icon.png" />


That should take care of virtually every device that supports this feature. Test it out on your Android phone and, if you have any friends in the Cult of Mac, ask if you can check this on the half dozen iOS devices they own.

If you want to really encourage iPhone/iPad users to take advantage of your new, shiny icon to bookmark your webapp on their Home screen, check out this javascript library called Mobile Bookmark Bubble.

Of course you should also make sure your site works well on handheld devices, but that's a topic for another time.

Have fun!

Nodetree has Document and root Element

I think an example will explain the emerging API far better than I can.

You can replicate this with the code from http://hg.concordance-xmpp.org/nodetree, though its still very early the code is completely documented in its current state.

Python 3.2 (r32:88445, Feb 28 2011, 00:50:14)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import nodetree
>>> doc = nodetree.Document()
>>> foo = nodetree.Element('foo')
>>> doc.root = foo
>>> len(doc)
1
>>> print(doc)
<?xml version="1.0"?>
<foo/>

>>> bar = nodetree.Element('bar')
>>> doc.root = bar
>>> print(doc)
<?xml version="1.0"?>
<bar/>

>>> print(foo)
<foo/>
>>> print(bar)
<bar/>
>>> print(doc[0])
<bar/>
>>>


Nodetree design

I've spent the last two weeks charting out Nodetree, a XML data binding package for Python 3.

I'd rather not reiterate the multitude of reasons for this, but since they always get asked, I'll summarize this with needing a stream parser that supports xpath/xslt and passing chunks of an XML stream in Python, something that ElementTree and lxml do not support. SleekXMPP employs a fairly ingenious hack on lxml to achieve this, but I'd rather spend some time doing it right from the start for Concordance.

Since XPath and XSLT are best supported by libxml2 than any other free software library, and I'd rather not spend the next few months writing an implementation from scratch, the data must exist as a libxml2 DOM tree at the point of processing.

You can get a libxml2 DOM tree either by parsing a file (which gives you little control, it parses the entire file one-shot) or build the tree node by node. libxml2 also supports processing an XML stream through its SAX interface, node by node, which gives us the flexibility needed to pop and/or parse nodes from the stream as its being parsed while still having all the other tools we need by generating a DOM tree of the segments as we process them.

The problem here is that Nodetree isn't a DOM interface, its a Pythonic XML data binding interface, so such things as adding the same element to two documents (or being able to create a new document using a piece of another document while both are in memory) is a challenge. Even if we were not using libxml2/DOM, for XPath to work correctly every node must have a clear hierarchy which falls apart when your context is a node used in the same document three times and in two different documents.

One of the benefits of working on a wide variety of projects is reusing clever solutions used on one project for something completely different. For example, I recently implemented PySoy's atomic API whereas Python objects are created on the fly for underlying data structures and could be attached to multiple points of data (see ColorCanvas.py for example). This allows multiple data points to be updated with one step, and saves us from doing silly things like storing (ie) an object for every pixel in an image.

I'm implementing almost exactly the same mechanism for Nodetree, it'll limit XPath searches run from Python somewhat but should be fully XML compliant and Pythonic, something (IMHO) nobody has managed to do yet.

PyCon 2011 wrapup

This has been an awesome week at PyCon, though I'm completely exhausted.

The most notable outcome of several meetings with Python leadership about increasing diversity in the Python community. In my role with PSF's educational outreach programs I'm calling on 3rd party projects working with us to help work to increase gender balance in our community, and as PySoy's project maintainer I'm committing us to working seriously on this issue over the next year as an example.

Several college-aged women at PyCon knew about Google's Summer of Code but believed it was only for "geniuses who've been programming since they were 13" or that they were unqualified for other reasons, even though they're clearly more skilled (and moreso, have a better attitude) than some of the male students we get.

To address this, my first step is outreach to women for applying to us for Google's Summer of Code in hopes of resolving any miscommunication over required experience and what we're looking for. This is being sent out over the next week so potential applicants leverage of early involvement with mentors that many students have already started on.

Once Summer of Code is underway I plan to continue this an internship program in attempt to get more women involved in our community in ways not supported by GSoC such as through QA testing, documentation, video tutorials, and artwork. This program will be for women ages 18+ with no college requirement, though we're still working out many of the other details.

For the PyCon sprints I worked on the new "packaging" package for Python 3.3, formerly known as distutils2. My primary interest in this is a packaging.backport module based on 3to2 (which backports code from 3.2 to 2.7) extended with further fixers for 2.7 to 2.6, 2.5, and 2.4. This module will work for packages written for Python 3.3 and above to maintain Python 2 compatibility, and also to backport "packaging" to the standalone distutils2 package for previous Python versions.

We managed to get distutils2 largely upgraded for Python 3.3 syntax and the non-PyPI tests running cleanly, though much more work is needed. We have a few months given than Python 3.2 was just released and we're working on a roughly 18 month release cycle.

Sprinting toward Beta-3

We've made some massive headway in the last two weeks;

First, we have physics processing back. We're using ODE for now just to get us through the next release cycle or two, but we've modified its headers (ABI-compatible) to work better with libsoy. Done are the days when we stress over API compatibility, since we've decided to write our own fully integrated (and scaled down) physics processing with Orc we have a certain freedom to throw their conventions to the wind.

How this is going to play out is a slow migration of values to arrays. Body data, for example, will be stored an array inside a scene so SIMD processing can be used more efficiently. The Soy Body class will represent a pointer to that array and include properties and methods for working with its data. The actual storage of that data will be governed by the Scene, giving it the freedom to sort bodies per "island" (groups of bodies connected by group). This is similar to how ODE processes bodies already and we'll be likely using their code as a starting place.

Second, we have the basic types for rendering. At the moment this is just a rotating cube, but that involved roughly 6 classes operating in both GObject and Python domains. The next step is soy.models.Mesh for rendering more complex shapes.

Third and most exciting, our humble game engine can now run as a Firefox plugin! There's some bugs to be worked out still, but "embedded" support has been added to our soy.widgets.Window class that allows it to run inside virtually any application.

With luck and determination we'll have the next release out by the end of the month. I've dialed back our goals considerably given that its a rewrite from our last beta release but our release cycle will be much shorter now that the rewrite is finishing up.

New import (and export) mechanism

I've started writing a new import mechanism for .soy files, replacing the old soy.transports module with a importer that allows .soy files to be loaded as modules with the standard Python import command.

With the old method, "transport" objects acted as dicts of objects in the .soy files and could be loaded and saved as such. The resulting code looked like this:
import soy
MyGame = soy.transports.File('data/MyGame.soy')
window = MyGame['window']
redblock = MyGame['redblock']
(etc)

Now, the new .soy importer is added to sys.path_hooks when PySoy is first initialized and applys to any directory added after. When finished, the above code could be rewritten as:
import soy
sys.path.append('data/')
import MyGame


No "etc" needed because in that one import command data/MyGame.soy is loaded and made available directly in a module called MyGame. Much more pythonic, IMHO.

A separate class will be written next called soy.Exporter which PySoy objects can be added and removed from, then saved to a file such as this example which modifies the previously imported MyGame module:
archive = soy.Exporter(MyGame)
archive.grasstex = grass_texture
del(archive.bricktex)
archive('data/MyGame2.soy')


Similarly, two archives could be combined like this:
archive = soy.Exporter(MyGame)
archive2 = soy.Exporter(CharacterFoo)
archive += archive2
archive('data/MyGame3.soy')


The reasoning behind this is in most cases .soy files will be written by GIMP or Blender, so the most common use case for the soy.Exporter class is for combining and pruning the .soy files created by these tools.

As always, constructive/critical feedback welcome and appreciated.

libsoy migration milestone

After a great deal of time (I'd rather not admit how long) we've finally managed to link the new libsoy code in with Python.

This afternoon (around 4:30) the following code opened a PySoy window on the screen:

arc@khonsu ~/work/pysoy $ python3
Python 3.1.3 (r313:86834, Dec  5 2010, 09:55:24) 
[GCC 4.5.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import soy.widgets
>>> w = soy.widgets.Window()
>>> 
(soy:16042): Gdk-CRITICAL **: IA__gdk_window_set_title: assertion `title != NULL' failed
IRQ's not enabled, falling back to busy waits: 2 0
OpenGL version 2.1 Mesa 7.10
GL_ARB_vertex_buffer_object: Yes

While incomplete and, well, its only a window, this is the first time we've gotten the two pieces to work together. David Czech and I have spent the rest of the evening fixing both build systems (libsoy and PySoy) and getting ready to copy the window binding code for the rest of the engine.

Our largest failure in this has been diverting so much energy into GObject Introspection. While its a really great idea and I'd love to see it evolve, its still so deep alpha that I can't imagine anyone but Gnome developers getting much practical use out of it. I would like to finish up GTypes at some point, but either the XML format for .gir or any level of documentation on typelib needs to be written first. I've wasted far too much time already trying to reverse engineer these from examples and their source that should have been put into my own projects.

common Orc opcodes

I've been going through liboil's 0.3 source to rewrite the oil_yuv2rgbx_sub2_u8 function we use for Theora decoding to Orc pseudo-assembly code.

Because the Orc opcode documentation splits opcode description and processor support between two tables, for reference I wrote a quick Python script to build a table of Orc opcodes common to SSE (x86), Altivec (PPC/Cell), and NEON (Arm Cortex) processors.

Here's that table for reference, at least until I put the time to format it for a wiki:

opcode	dst	src1	src2	description	pseudo code
absb	1	1		absolute value	(a < 0) ? -a : a
addb	1	1	1	add	a + b
addssb	1	1	1	add with signed saturate	clamp(a + b)
addusb	1	1	1	add with unsigned saturate	clamp(a + b)
andb	1	1	1	bitwise AND	a & b
andnb	1	1	1	bitwise AND NOT	a & (~b)
avgsb	1	1	1	signed average	(a + b + 1)>>1
avgub	1	1	1	unsigned average	(a + b + 1)>>1
cmpeqb	1	1	1	compare equal	(a == b) ? (~0) : 0
cmpgtsb	1	1	1	compare greater than	(a > b) ? (~0) : 0
copyb	1	1		copy	a
loadb	1	1		load from memory	array[i]
loadpb	1	1		load parameter or constant	scalar
maxsb	1	1	1	signed maximum	(a > b) ? a : b
maxub	1	1	1	unsigned maximum	(a > b) ? a : b
minsb	1	1	1	signed minimum	(a < b) ? a : b
minub	1	1	1	unsigned minimum	(a < b) ? a : b
mullb	1	1	1	low bits of multiply	a * b
mulhsb	1	1	1	high bits of signed multiply	(a * b) >> 8
mulhub	1	1	1	high bits of unsigned multiply	(a * b) >> 8
orb	1	1	1	bitwise or	a | b
shlb	1	1	1S	shift left	a << b
shrsb	1	1	1S	signed shift right	a >> b
shrub	1	1	1S	unsigned shift right	a >> b
signb	1	1		sign	sign(a)
storeb	1	1		store to memory	special
subb	1	1	1	subtract	a - b
subssb	1	1	1	subtract with signed saturate	clamp(a - b)
subusb	1	1	1	subtract with unsigned saturate	clamp(a - b)
xorb	1	1	1	bitwise XOR	a ^ b
absw	2	2		absolute value	(a < 0) ? -a : a
addw	2	2	2	add	a + b
addssw	2	2	2	add with signed saturate	clamp(a + b)
addusw	2	2	2	add with unsigned saturate	clamp(a + b)
andw	2	2	2	bitwise AND	a & b
andnw	2	2	2	bitwise AND NOT	a & (~b)
avgsw	2	2	2	signed average	(a + b + 1)>>1
avguw	2	2	2	unsigned average	(a + b + 1)>>1
cmpeqw	2	2	2	compare equal	(a == b) ? (~0) : 0
cmpgtsw	2	2	2	compare greater than	(a > b) ? (~0) : 0
copyw	2	2		copy	a
div255w	2	2		divide by 255	a/255
loadw	2	2		load from memory	array[i]
loadpw	2	2		load parameter or constant	scalar
maxsw	2	2	2	signed maximum	(a > b) ? a : b
maxuw	2	2	2	unsigned maximum	(a > b) ? a : b
minsw	2	2	2	signed minimum	(a < b) ? a : b
minuw	2	2	2	unsigned minimum	(a < b) ? a : b
mullw	2	2	2	low bits of multiply	a * b
mulhsw	2	2	2	high bits of signed multiply	(a * b) >> 8
mulhuw	2	2	2	high bits of unsigned multiply	(a * b) >> 8
orw	2	2	2	bitwise or	a | b
shlw	2	2	2S	shift left	a << b
shrsw	2	2	2S	signed shift right	a >> b
shruw	2	2	2S	unsigned shift right	a >> b
signw	2	2		sign	sign(a)
storew	2	2		store to memory	special
subw	2	2	2	subtract	a - b
subssw	2	2	2	subtract with signed saturate	clamp(a - b)
subusw	2	2	2	subtract with unsigned saturate	clamp(a - b)
xorw	2	2	2	bitwise XOR	a ^ b
absl	4	4		absolute value	(a < 0) ? -a : a
addl	4	4	4	add	a + b
addssl	4	4	4	add with signed saturate	clamp(a + b)
addusl	4	4	4	add with unsigned saturate	clamp(a + b)
andl	4	4	4	bitwise AND	a & b
andnl	4	4	4	bitwise AND NOT	a & (~b)
avgsl	4	4	4	signed average	(a + b + 1)>>1
avgul	4	4	4	unsigned average	(a + b + 1)>>1
cmpeql	4	4	4	compare equal	(a == b) ? (~0) : 0
cmpgtsl	4	4	4	compare greater than	(a > b) ? (~0) : 0
copyl	4	4		copy	a
loadl	4	4		load from memory	array[i]
loadpl	4	4		load parameter or constant	scalar
maxsl	4	4	4	signed maximum	(a > b) ? a : b
maxul	4	4	4	unsigned maximum	(a > b) ? a : b
minsl	4	4	4	signed minimum	(a < b) ? a : b
minul	4	4	4	unsigned minimum	(a < b) ? a : b
orl	4	4	4	bitwise or	a | b
shll	4	4	4S	shift left	a << b
shrsl	4	4	4S	signed shift right	a >> b
shrul	4	4	4S	unsigned shift right	a >> b
signl	4	4		sign	sign(a)
storel	4	4		store to memory	special
subl	4	4	4	subtract	a - b
subssl	4	4	4	subtract with signed saturate	clamp(a - b)
subusl	4	4	4	subtract with unsigned saturate	clamp(a - b)
xorl	4	4	4	bitwise XOR	a ^ b
loadq	8	8		load from memory	array[i]
storeq	8	8		store to memory	special
splatw3q	8	8		duplicates high 16-bits to lower 48 bits	special
convsbw	2	1		convert signed	a
convubw	2	1		convert unsigned	a
splatbw	2	1		duplicates 8 bits to both halfs of 16 bits	special
splatbl	4	1		duplicates 8 bits to all parts of 32 bits	special
convswl	4	2		convert signed	a
convuwl	4	2		convert unsigned	a
convslq	8	4		signed convert	a
convulq	8	4		unsigned convert	a
convwb	1	2		convert	a
convhwb	1	2		shift and convert	a>>8
convssswb	1	2		convert signed to signed with saturation	clamp(a)
convsuswb	1	2		convert signed to unsigned with saturation	clamp(a)
convuuswb	1	2		convert unsigned to unsigned with saturation	clamp(a)
convlw	2	4		convert	a
convhlw	2	4		shift and convert	a>>16
convssslw	2	4		convert signed to signed with saturation	clamp(a)
convql	4	8		convert	a
mulsbw	2	1	1	multiply signed	a * b
mulubw	2	1	1	multiply unsigned	a * b
mulswl	4	2	2	multiply signed	a * b
muluwl	4	2	2	multiply unsigned	a * b
accl	4	4		accumulate	+= a
swapw	2	2		endianness swap	special
swapl	4	4		endianness swap	special
select0wb	1	2		select first half	special
select1wb	1	2		select second half	special
select0lw	2	4		select first half	special
select1lw	2	4		select second half	special
mergewl	4	2	2	merge halves	special
mergebw	2	1	1	merge halves	special
splitlw	2	4		split first/second words	special
splitwb	1	2		split first/second bytes	special
addf	4	4	4	add	a + b
subf	4	4	4	subtract	a - b
mulf	4	4	4	multiply	a * b
maxf	4	4	4	maximum	max(a,b)
minf	4	4	4	minimum	min(a,b)
cmpeqf	4	4	4	compare equal	(a == b) ? (~0) : 0
convfl	4	4		convert float point to integer	a
convlf	4	4		convert integer to floating point	a

stuck, back to PySoy

I've been working on a streaming XML parser for Python, but need a break. At this point there's no way Concordance is getting out Jan 1st, but certainly by the end of Winter.

Our libsoy migration process PySoy got pretty far. We were migrating from Pyrex to Genie, essentially moving the core engine from PyObject to GObject to remove Python dependency in game clients and enable further multicore processing on both client and servers. Much of the rendering area of the engine has been migrated, but the process has been held up in two areas;

First, while libsoy is in pretty good shape, we still lack Python bindings - aka PySoy itself, which is what we intend games to be written and run with. Our original plan to use GObject Introspection failed in a horrible mess that I've documented in previous postings, we've looked at using SWIG and even building our own bindings generation with little measurable success. In order to get us moving forward again I'm going to just drop out some .c templates and write the custom wrapper classes by hand. The time it'd take to write and maintain these cannot possibly be greater than the time we've wasted talking about a more elegant solution that only exists conceptually.

When GObject Introspection reaches a state of even remote maturity, where it can offer a Pythonic API, we'll look at it again. We'd even help get it there if the current GIR developers would just document the .gir XML schema or typelib format so we wouldn't have to refer to their source code as the sole definition of these.

Second is our physics code. As I've posted, ODE worked for us in the past but has numerous issues with packaging for various Linux distros (and poor features, slow, and extremely difficult to port to mobile devices). We attempted to migrate to Bullet but this burned us out - virtually no work has gone into that in the past 6 months. We're all pretty frustrated with Bullet's haphazard API (whereas ODE is fairly clean) and the C++ only API doesn't play well with GObject (or anything other than c++ for that matter). Bullet's C API is minimal at best.

When it comes right down to it, the biggest barrier we face with physics is processing power on mobile devices, an issue that using Bullet would not solve. Most of the devices we're interested in include ARM6/7 processors from Qualcomm or TI. Many do not include a FPU (floating point unit), but they all seem to offer a fairly powerful DSP used extensively for processing multimedia. We do not, however, want to rewrite and maintain our physics processing for each platform.

A solution I've come up with is to write our physics, greatly simplified from even what ODE offers, using Orc. It's yet another metalanguage (first Pyrex, then Vala/Genie, now this..), but the successor to liboil (which we and much of the Gnome community use) and already supports many interesting platforms.

My plan is to first migrate our liboil-based YUV-RGB conversion code to Orc to get my feet wet, then implement a greatly simplified collision system using it, and expect the next release (or two) to still use ODE for at least rigid body physics with the plan to eventually replace even that with our own physics solver. It should be much faster, and the same Orc code we write now should be able to compile to DSP code for Android handsets and other mobile devices in the future.

Orc already supports ARM Cortex (NEON), so if we were to finish this work today we'd be able to run PySoy clients on more modern Android handsets without touching DSP code. DSP support in Orc would also be very useful for future hardware for PySoy game servers.

While we'd all really like to get the next PySoy release out ASAP, we'd also like to avoid rewriting the engine again down the road.

XML parsing in Python

Its been a couple months, so I'm going to give a brief update on what I've been working on.

Concordance is getting close to release, I plan to have the first release (0.1) out January 1st. More on this toward the end of December.

One of the roadblocks I've hit (again and again) is the lack of a decent XML parsing package for Python. The standard library is a shame when it comes to XML; at least four different modules (expat, sax, dom, etree) to choose from and none of them support even XPath. The most popular option, etree (or ElementTree), cannot even process an XML file with the namespace prefix intact.

There's lxml, which offers an etree-compatible API and fixes many of ElementTree's major faults (namespace prefix preservation, xpath/xslt support) but still cannot handle stream processing and, due to ElementTree's API, does not expose multiple text nodes broken up by a child element such as "<div>first string <br/> second string</div>".

To support XMPP streams we need to use expat or sax to handle the stream event-by-event, since the full XML document is only available once the root element closes at the end of the stream, but the direct children of the root element (what we call "stanzas" in XMPP) need to be processed as complete objects. While we may be able to hack something together using lxml, it would likely be less work than to implement a new XML parsing package. As long as the resulting API doesn't diverge very greatly etree the work necessary to switch should be minimal.

Beside this I've been working on a host of different packages around Concordance, from getting a javascript BOSH/XMPP library together to getting distutils2 ready for Python 3. I've even managed to ship a pitiful little serial library for Python 3, PyTTY that we're using to interface with some Arduinos.