tag:blogger.com,1999:blog-283587872024-03-07T22:52:28.453-05:00Arc's Soy MachineAnonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.comBlogger156125tag:blogger.com,1999:blog-28358787.post-5681548940246408272013-01-18T00:51:00.002-05:002013-01-18T00:51:14.170-05:00NodeTree 0.3 Released<p>I just wrapped up <a href="http://www.nodetree.org/">NodeTree</a> version 0.3, a little over a year since <a href="http://arcriley.blogspot.com/2011/12/nodetree-02-released.html">the last release</a>.</p>
<p>As promised, this release includes document and stream parsing with some very early use of XMPP. However, there's some basic features that haven't been implemented yet:</p>
<ul>
<li>Namespace access from Python</li>
<li>XPath matching</li>
<li>XSLT support</li>
<li>CDATA blocks</li>
</ul>
<p>For <a href="http://www.concordance-xmpp.org/">Concordance</a> we're going to need at least <a href="http://en.wikipedia.org/wiki/XSLT">XSLT</a> support so I'm adding this next. While we're already storing XML data in <a href="http://www.xmlsoft.org/">libxml2</a> <a href="http://en.wikipedia.org/wiki/Document_Object_Model">DOM</a> on the backend, XSLT will take a bit more than simply binding <a href="http://xmlsoft.org/XSLT/">libxslt</a> due to the manner NodeTree objects bind to their respective libxml nodes.</p>
<p><a href="http://en.wikipedia.org/wiki/XML_namespace">XML namespace</a> access also include some interesting challenges with the API, notably, how to expose the namespace prefix and/or uri with Element.name and Element.attributes.</p>
<p>There are also a few rough edges I haven't cleaned up yet such as being able to pass a generator (eg, File) to Document or Stream, exposing which documents an Element belongs to, or ensuring a node isn't added to the same document twice.</p>
<p>You can download <a href="http://pypi.python.org/pypi/NodeTree/0.3">NodeTree 0.3</a> from the <a href="http://pypi.python.org/pypi">Python Package Index</a> and let me know what you think.</p>Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com1tag:blogger.com,1999:blog-28358787.post-82413635086429924072011-12-16T12:06:00.002-05:002011-12-16T12:06:28.359-05:00GCI Low Hanging Fruit<p>I have to admit I'm pretty surprised that at halfway through Google Code-In 2011 only about a quarter of the tasks we posted for the first half have been completed or in-progress. Further, most of the tasks thus far haven't been coding tasks.</p>
<p>I'm going to publish here a list of what we consider "low hanging fruit"; coding tasks which are fairly easy to get started with. If you're a student age 13-17 these would be an easy way to earn a Google tshirt and some cash. All of these tasks deal with OpenGL rendering, usually just arrays for points and lines/triangles which connect the points. There's a lot of examples in the code already for this and numerous tutorials on the web (such as <a href="http://nehe.gamedev.net/">NeHe</a>).</p>
<dl>
<dt>Simple Rendering</dt>
<dd>We have 2 new tasks listed for rendering simple models; <a href="http://www.google-melange.com/gci/task/view/google/gci2011/7237313">Camera</a> and <a href="http://www.google-melange.com/gci/task/view/google/gci2011/7234264">Light</a>. Camera is very simple, while Light can be as simple or complex as you want to make it.</dd>
<dt>Shapes Rendering</dt>
<dd>We have 3 tasks for rendering shapes, <a href="http://www.google-melange.com/gci/task/view/google/gci2011/7127345">Box</a>, <a href="http://www.google-melange.com/gci/task/view/google/gci2011/7229261">Room</a>, and <a href="http://www.google-melange.com/gci/task/view/google/gci2011/7116340">Sphere</a>. Any of these could be knocked out in a few hours even without prior OpenGL knowledge.</dd>
<dt>Joints Rendering</dt>
<dd>We have 6 new tasks up for rendering joints; <a href="http://www.google-melange.com/gci/task/view/google/gci2011/7228305">Ball</a>, <a href="http://www.google-melange.com/gci/task/view/google/gci2011/7245257">Fixed</a>, <a href="http://www.google-melange.com/gci/task/view/google/gci2011/7228304">Hinge</a>, <a href="http://www.google-melange.com/gci/task/view/google/gci2011/7238256">Piston</a>, <a href="http://www.google-melange.com/gci/task/view/google/gci2011/7230270">Slider</a>, and <a href="http://www.google-melange.com/gci/task/view/google/gci2011/7233301">Universal</a>. Joints (soy.joints) connect two bodies such that they can only move in respect to each other in a certain way, such as a door hinge or piston. These are all <a href="http://opende.sourceforge.net/wiki/index.php/Manual_%28Joint_Types_and_Functions%29#Joint_parameter_setting_functions">documented</a> with graphic depictions. This is slightly more complex than the simple rendering tasks (above) in that there's two pieces to each joint and they can be rendered as either wireframe or solid (with provided materials).</dd>
</dl>
As always, we're on IRC if a student wants to discuss these or other tasks.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com2tag:blogger.com,1999:blog-28358787.post-12094383698492067932011-12-14T00:27:00.000-05:002011-12-14T00:27:30.382-05:00PyTTY 0.3<p>Continuing my annual end-of-year coding sprint, I just released PyTTY 0.3.</p>
<p><a href="http://www.pytty.org/">PyTTY</a> is a Python serial communication package I started last year after a friend said he couldn't use <a href="http://python.org/">Python</a> 3 yet because pyserial wasn't ported. The point of writing this was to show him that he didn't need an ancient, bloated, poorly-maintained package to do something as simple as serial communication.</p>
<p>What I wrote over an afternoon turned out to be a little over 100 lines of fairly useful code which I've since used in quite a few microcontroller projects (eg, <a href="http://www.arduino.cc/">Arduino</a>). Its by no means complete, the only setting is baud rate and there's no Windows support, but its done everything I've needed it to over the last year. The only problem that's been reported is poor documentation which this release aims to fix. It includes a short code example ("pydoc pytty.TTY") which runs on both legacy Python and Python 3.</p>
<p>PyTTY 0.3 is under 135 lines of pure Python and relies only on the Python standard library. If there's a feature you need which this doesn't have either email me a patch or a feature request so I can add it. The <a href="http://mercurial.selenic.com/">Mercurial</a> repository is <a href="http://hg.pytty.org/pytty">http://hg.pytty.org/pytty</a>.</p>Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-33693514927207511372011-12-08T00:50:00.001-05:002011-12-08T12:05:03.085-05:00NodeTree 0.2 Released<p>I just shipped <a href="http://nodetree.org/">NodeTree</a> 0.2.</p>
<p>This version will not parse an XML stream. All it contains are some basic types representing XML nodes such as <code>Comment</code>, <code>Document</code>, and <code>Element</code>. As promised, text is also handled as a node but uses standard Python strings (UTF-8 strings/bytes and unicode). These should all be fairly intuitive to use.</p>
<p>The magic is the XML data is being managed in C using a libxml2 DOM tree but accessed through a Pythonic object-oriented API. For example, in DOM each node may have exactly one parent - in NodeTree a node may be added to any number of parents with a separate DOM node and context for each.</p>
<p>I started this project because the existing XML packages for <a href="http://www.python.org/">Python</a> proved too difficult to use with <a href="http://xmpp.org/">XMPP</a>. Fritzy's <a href="https://github.com/fritzy/SleekXMPP/wiki">SleekXMPP</a> uses lxml but had to jump through several hoops to get stream parsing to work, looking over his work I certainly didn't want to repeat it with <a href="http://concordance-xmpp.org/">Concordance-XMPP</a>.</p>
<p>Beyond this the leading XML API for Python, ElementTree, includes several unfortunate design decisions that make it frustrating to use in the best cases and unusable in others. A full list of why can be left for another time, but the difference to NodeTree can be described in their names - ElementTree is a tree of XML Element nodes with other kinds of nodes either silently dropped, mangled, or made available in bizarre ways (eg, .text and .tail). In contrast, NodeTree provides XML data as a tree of nodes <i>starting with the Document node</i> and includes comment and text nodes in its tree. I plan to provide 100% XML 1.0 support in a future release while maintaining a clean, simple, and intuitive API.</p>
<p>Storing XML data in libxml2 DOM format gives us a few advantages over other XML libraries. First, we'll have XPath, XInclude, and XSLT available without having to convert the data between formats. Second, Python objects only need to be created for nodes Python wants a reference to so when we get to parsing data this will happen much faster and with less memory.</p>
<p>At version 0.2 NodeTree is still in its infancy but some of its API can be demonstrated. Here's a short example:</p>
<pre>
Python 3.2.2 (default, Oct 3 2011, 00:20:58)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import nodetree
>>> doc = nodetree.Document()
>>> doc.append(nodetree.Comment(' Start '))
>>> doc.append(nodetree.Element('data'))
>>> doc.append(nodetree.Comment(' Fini '))
>>> doc[1].attributes['thing'] = 'normal'
>>> doc[1].append(nodetree.Element('record'))
>>> doc[1][0].append('First')
>>> doc[1].append(nodetree.Element('record'))
>>> doc[1][1].append('Second')
>>> doc
<?xml version="1.0"?>
<!-- Start -->
<data thing="normal">
<record>First</record>
<record>Second</record>
</data>
<!-- Fini -->
</pre>
<p>NodeTree 0.2 is tested to work with Python 2.6, 2.7, 3.1, 3.2, and 3.3-pre. The next release is intended to support basic file and stream parsing.</p>Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-6356847392938479582011-12-03T20:55:00.001-05:002011-12-03T20:55:59.720-05:00XMPP on the web<p>A <a href="https://plus.google.com/109741359399131092509/posts/L9zcgcjzVsL">short thread on G+</a> has prompted this longer sharing of my vision for <a href="http://xmpp.org/">XMPP</a> on the web.</p>
<p>For XMPP use on a website we currently have <a href="http://xmpp.org/extensions/xep-0206.html">BOSH</a> and, in an extreme-alpha state, <a href="http://tools.ietf.org/html/draft-moffitt-xmpp-over-websocket-00">XMPP over websockets</a>. The advantage of websockets is obvious, BOSH is a high overhead protocol that we'd all rather not have to use, however both have the same problem: you either must share your login credentials (and thus access to your account) with every website you use a single account with, or must create a new account (JID) for every XMPP-based website you use.</p>
<p><a href="http://oauth.net/2/">Oauth2</a> for XMPP might be a partial solution to this by requiring your authorization through a central identity site and using the resulting token for logging in, however, you're still opening yourself up to the 3rd party website accessing your roster, sending spam messages on your behalf, and potentially worse. All this really gives you is the ability to later disable access to websites who misuse your account.</p>
<p>This is the crux of the issue: when using an Javascript library provided by a website and using a proxy provided by that website, whether BOSH, websockets, or otherwise, you're giving that website unlimited access to your account. I have not seen a workable proposal to solve this and until this is solved XMPP cannot see widespread use on the web.</p>
<p>I'm proposing that we solve this by putting XMPP in the browser, either directly or through a plugin. Expose a standard javascript API for allowing websites to use an XMPP connection along with a security model which gives users control as to what a website is allowed to use their connection for. Ie, if a script on a website wants access to their roster the user will be prompted for it, if they want to join a MUC room display a standard prompt for that. Browsers can have multiple XMPP sessions at once and allow the user to select which account they'd like to use with an XMPP-enabled site.</p>
<p>This is just some early ideas, I'm nowhere near implementing this though I think the conversation would be useful to get started.</p>Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com2tag:blogger.com,1999:blog-28358787.post-7326628069703781692011-11-28T01:56:00.001-05:002011-11-28T02:04:56.815-05:00OpenGL ES support complete<p>The experimental branch, where the OpenGL ES migration was being done, has just been closed and merged into the default branch of <a href="http://www.pysoy.org/">PySoy</a>.</p>
<p>Thanks to Steve Anton, one of our <a href="http://code.google.com/gci">Google Code-In 2011</a> students, for some of the last bits of work to complete the merge. We are now one step closer to mobile support! If you're a student ages 13-17 and would like to earn a Google tshirt and some cash by helping us with Android support, sign up for Google Code-In and claim <a href="http://www.google-melange.com/gci/task/view/google/gci2011/7129341">this task</a>.</p>Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-31033351430490117592011-11-21T04:45:00.001-05:002011-11-21T04:49:33.564-05:00Google Code-In 2011 is Open<div class="separator" style="clear: both; text-align: center;">
<a href="http://media.pysoy.org/img/gci_logo_2011.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://media.pysoy.org/img/gci_logo_2011.jpg" /></a></div>
Google Code-In has officially begun!<br />
<br />
From today through January 16th students age 13-17 can earn up to <b>$500</b> working on small tasks for software projects such as <a href="http://www.moinmoin.org/">MoinMoin</a>, <span class="proflinkWrapper"><span class="proflinkPrefix"></span><a class="proflink" href="http://www.sympy.org/">SymPy</a></span>, and <a href="http://www.pysoy.org/">PySoy</a>!<br />
<br />
Tasks
include coding, documentation, graphic design, video production,
testing, translation, research, public speaking, and many other kinds of
challenges of varying difficulty.<br />
<br />
Completing just one task earns
a student a Google tshirt. Every 3 tasks they complete earns them
$100, and the 10 top students worldwide will earn an all-expense paid
trip to Mountain View, CA to receive an award at Google.<br />
<br />
PySoy
has over 75 tasks offered for the first half of the program and another
75-100 will be made available December 16th. Our mentors are on
<a href="http://www.freenode.net/">Freenode</a> channel <a class="ot-hashtag" href="http://www.pysoy.org/">#PySoy</a> ready to help students start earning their tshirt and cash today.<br />
<br />
<a href="http://www.google-melange.com/">Sign up</a> today and get started!Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-7509594516158792132011-11-09T23:09:00.001-05:002011-11-10T01:15:57.343-05:00Rugby season nearly over, getting back to workWow its been a long time.
<p>We wrapped up <a href="http://socghop.appspot.com/gsoc/homepage/google/gsoc2011">Google's Summer of Code 2011</a> in August. The <a href="http://python.org/">Python Software Foundation</a> did wonderfully overall, for <a href="http://www.pysoy.org/">PySoy</a> 6 of our 7 students passed. A great year overall - thanks to all the mentors and students!</p>
<p><img src="http://blog.pysoy.org/images/ArcGettingSchooled-small.jpg" align="right" width="200" height="150" alt="Five man scrum vs Warrington"/>Rugby has been a life changer for me. My first game was in September, after floating in and out of practice for years and training pretty heavily since April. No serious injuries, but no shortage of pain; I've frequently needed to sleep in a reclining chair to keep blood from pooling in my shoulders and nurse bruised ribs, dislocated fingers and toes, shin splints, and pulled muscles everywhere. All so worth it.</p>
<p>These guys are like family to me. I know it sounds sappy, but I've come to trust the men in my pack with my life - in a way we all do every time we bind onto each other a scrum. Its not that big of an adjustment culturally though due to the large number of programmers, lawyers, and IT professionals on the team. When you work behind a desk all day its nice to balance it out with a physically intensive training in the evening and games on Saturday.</p>
<p><br clear="both"/><img src="http://blog.pysoy.org/images/renegades-hellfest2011.jpg" align="center" width="400" height="250" alt="Renegades Reds at Hellfest 2011"/></p>
<p>The climax of the season was Hellfest October 29th in Dallas, TX. <a href="http://dcrugby.com/">Washington Renegades</a> brought our B-side to compete and returned with the 1st place trophy. My teammate <a href="http://www.jimbo.info/weblog/?p=1996">Jimbo</a> has more pics on his blog of the tournament, I was wearing #23 as tighthead prop.</p>
<p>We have two more games this season before we settle in for the Winter and indoor off-season training at the gym. A group of us plan to do a 8-week program run by <a href="http://web.usarugby.org/cgi-bin/viadesto/natteams/mnt/15ProfileDetail.pl?playerId=368">a professional rugby player</a> this Winter to get ready for the Spring season and the <a href="http://www.binghamcup.com/">Bingham Cup 2012</a> in Manchester UK next June.</p>
<p>Today the PySoy project was accepted to <a href="http://code.google.com/gci">Google Code-In</a>. We've got a number of <a href="http://www.pysoy.org/wiki/CodeIn/2011">student tasks</a> lined up, with many more being worked on for the first batch set to release in less than two weeks. Interested students should hop on <a href="http://www.freenode.net/">Freenode</a> (#PySoy) and get oriented before the program starts so they're ready to jump right into their first task!</p>Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-83935963585347026622011-07-21T11:31:00.004-04:002011-07-21T11:46:50.534-04:00Transcoding FLAC to Ogg VorbisLast night I hit a dilemma; a very old CD I ripped to FLAC and now can't find was playable only on certain players, not my Android phone (despite FLAC support) and behaving strange on many desktop players.<br /><br />Usually I just use something like <code>oggenc -q 4 *.flac</code> since vorbis-tools supports FLAC as a source format (and preserves metadata like artist, title, etc). Strangely, oggenc didn't recognize the files in this album.<br /><br /><a href="http://gstreamer.freedesktop.org/">GStreamer</a> to the rescue; <code>for file in *.flac; do gst-launch-0.10 filesrc location="$file" ! decodebin ! audioconvert ! vorbisenc quality=0.4 ! oggmux ! filesink location="$file.ogg"; done;</code><br /><br />Even though its much slower and more complicated, this should have done the trick. It didn't, and ogginfo showed that the Ogg muxer in GStreamer has some issues;<br /><pre><br />WARNING: granulepos in stream 1 decreases from 218558 to 205632<br />WARNING: granulepos in stream 1 decreases from 666174 to 655680<br />WARNING: granulepos in stream 1 decreases from 3150206 to 3142208<br />WARNING: granulepos in stream 1 decreases from 3605054 to 3597376<br />WARNING: granulepos in stream 1 decreases from 3828670 to 3822400<br />WARNING: granulepos in stream 1 decreases from 4741630 to 4733312<br />WARNING: granulepos in stream 1 decreases from 5636158 to 5630784<br />WARNING: granulepos in stream 1 decreases from 5858494 to 5854208<br />WARNING: granulepos in stream 1 decreases from 6096126 to 6090560<br />WARNING: granulepos in stream 1 decreases from 6317758 to 6314048<br />WARNING: granulepos in stream 1 decreases from 7668798 to 7665088<br />WARNING: granulepos in stream 1 decreases from 7902718 to 7898240</pre><br /><br />So with GStreamer not an option, I looked at the original FLAC files and found that whatever encoder I used added ID3 tags (which are not part of the FLAC spec). A quick ID3 removal command stripped these out so oggenc would recognize the files and work;<br /><pre><br />find . -name "*.flac" -exec id3v2 --delete-all {} \;<br />oggenc -q 4 *.flac</pre><br /><br />Done.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-55207039019458855152011-05-31T16:21:00.007-04:002011-06-02T01:09:39.707-04:00making an Android/iPhone/iPad home screen icon for your website<a href="http://concordance-xmpp.org/"><img src="http://concordance-xmpp.org/apple-touch-icon-114x114-precomposed.png" alt="Glossy Concordance Icon" align="right" style="border: 0;" /></a>One of the cool features of new phones and tablets is the ability to drop a website bookmark as an icon on the home screen, as if it was a normal App for that device. Many websites provide icons which are automatically used when a user bookmarks them, however, the tools and resources for doing this are not immediately obvious. I just spent Memorial Day figuring this out and will share what I found here.<br /><br />First, design your logo. I use <a href="http://inkscape.org/">Inkscape</a> so I can later scale the icon for tshirts and printed material, but any tool will do as long as you end up with a square flattened 512x512 PNG. Open this in <a href="http://gimp.org/">GIMP</a>.<br /><br />Next download this <a href="http://www.iquatsch.de/achsogehtdas/?p=4">GIMP template</a> and open it in GIMP. Copy/paste your icon into the already selected region of the template, it should "just work". Anchor your selection and merge visible layers for your 512x512 icon.<br /><br />Now that you have your glossy icon with rounded edges, scale and save the image three times for 57x57, 72x72, and 114x114 as follows:<br /><br /><code><br /> apple-touch-icon-57x57-precomposed.png<br /> apple-touch-icon-72x72-precomposed.png<br /> apple-touch-icon-114x114-precomposed.png<br /></code><br /><br />Also save your original (pre-template) logo at 57x57 as <code>apple-touch-icon.png</code>. These four files should cover iPhone, iPad, Android, and Blackberry. You may also want to save either the original logo or the glossy version at 16x16 for <code>favicon.ico</code> if you have not done so already.<br /><br />Upload these files to your web server saved in the root directory of your website (like <code>robots.txt</code>) and add the following template to the head of your website:<br /><br /><code><br /> <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" /><br /> <link rel="apple-touch-icon-precomposed" sizes="114x114" href="http://yourdomain.com/apple-touch-icon-114x114-precomposed.png" /><br /> <link rel="apple-touch-icon-precomposed" sizes="72x72" href="http://yourdomain.com/apple-touch-icon-72x72-precomposed.png" /><br /> <link rel="apple-touch-icon-precomposed" href="http://yourdomain.com/apple-touch-icon-57x57-precomposed.png" /><br /> <link rel="apple-touch-icon" href="http://yourdomain.com/apple-touch-icon.png" /><br /></code><br /><br /><img src="http://blog.pysoy.org/screenshots/mobile_bookmark_bubble.png" alt="Mobile Bookmark Bubble Screenshot" align="right" width="207" height="385" style="border: 0;" />That should take care of virtually every device that supports this feature. Test it out on your Android phone and, if you have any friends in the <a href="http://crave.cnet.co.uk/gadgets/apple-stimulates-brains-religious-responses-claims-bbc-50003807/">Cult of Mac</a>, ask if you can check this on the half dozen iOS devices they own.<br /><br />If you want to really encourage iPhone/iPad users to take advantage of your new, shiny icon to bookmark your webapp on their Home screen, check out this javascript library called <a href="http://code.google.com/p/mobile-bookmark-bubble/">Mobile Bookmark Bubble</a>.<br /><br />Of course you should also make sure your site works well on handheld devices, but that's a topic for another time.<br /><br />Have fun!Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com3tag:blogger.com,1999:blog-28358787.post-82324730589144598352011-05-24T02:02:00.002-04:002011-05-24T02:13:28.589-04:00Nodetree has Document and root ElementI think an example will explain the emerging API far better than I can.<br /><br />You can replicate this with the code from <a href="http://hg.concordance-xmpp.org/nodetree">http://hg.concordance-xmpp.org/nodetree</a>, though its still very early the code is completely documented in its current state.<br /><br /><code>Python 3.2 (r32:88445, Feb 28 2011, 00:50:14) <br />[GCC 4.5.2] on linux2<br />Type "help", "copyright", "credits" or "license" for more information.<br />>>> import nodetree<br />>>> doc = nodetree.Document()<br />>>> foo = nodetree.Element('foo')<br />>>> doc.root = foo<br />>>> len(doc)<br />1<br />>>> print(doc)<br /><?xml version="1.0"?><br /><foo/><br /><br />>>> bar = nodetree.Element('bar')<br />>>> doc.root = bar<br />>>> print(doc)<br /><?xml version="1.0"?><br /><bar/><br /><br />>>> print(foo)<br /><foo/><br />>>> print(bar)<br /><bar/><br />>>> print(doc[0])<br /><bar/><br />>>> <br /></code>Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-64321839616902598462011-05-21T15:48:00.003-04:002011-05-21T16:31:16.604-04:00Nodetree designI've spent the last two weeks charting out <a href="http://ohloh.net/p/nodetree">Nodetree</a>, a <a href="http://en.wikipedia.org/wiki/XML_data_binding">XML data binding</a> package for Python 3.<br /><br />I'd rather not reiterate the multitude of reasons for this, but since they always get asked, I'll summarize this with needing a stream parser that supports xpath/xslt and passing chunks of an XML stream in Python, something that ElementTree and lxml do not support. <a href="https://github.com/fritzy/SleekXMPP/wiki">SleekXMPP</a> employs a fairly ingenious hack on lxml to achieve this, but I'd rather spend some time doing it right from the start for <a href="http://concordance-xmpp.org/">Concordance</a>.<br /><br />Since <a href="http://en.wikipedia.org/wiki/XPath_1.0">XPath</a> and <a href=" http://en.wikipedia.org/wiki/XSLT">XSLT</a> are best supported by <a href="http://xmlsoft.org/">libxml2</a> than any other free software library, and I'd rather not spend the next few months writing an implementation from scratch, the data must exist as a libxml2 DOM tree at the point of processing.<br /><br />You can get a libxml2 DOM tree either by parsing a file (which gives you little control, it parses the entire file one-shot) or build the tree node by node. libxml2 also supports processing an XML stream through its <a href="http://en.wikipedia.org/wiki/Simple_API_for_XML">SAX</a> interface, node by node, which gives us the flexibility needed to pop and/or parse nodes from the stream as its being parsed while still having all the other tools we need by generating a DOM tree of the segments as we process them.<br /><br />The problem here is that Nodetree isn't a <a href="http://en.wikipedia.org/wiki/Document_Object_Model">DOM interface</a>, its a Pythonic XML data binding interface, so such things as adding the same element to two documents (or being able to create a new document using a piece of another document while both are in memory) is a challenge. Even if we were not using libxml2/DOM, for XPath to work correctly every node must have a clear hierarchy which falls apart when your context is a node used in the same document three times and in two different documents.<br /><br />One of the benefits of working on a wide variety of projects is reusing clever solutions used on one project for something completely different. For example, I recently implemented <a href="http://www.pysoy.org/">PySoy</a>'s atomic API whereas Python objects are created on the fly for underlying data structures and could be attached to multiple points of data (see <a href="http://hg.pysoy.org/pysoy/file/0b9bcf356f2b/examples/ColorCanvas.py">ColorCanvas.py</a> for example). This allows multiple data points to be updated with one step, and saves us from doing silly things like storing (ie) an object for every pixel in an image.<br /><br />I'm implementing almost exactly the same mechanism for Nodetree, it'll limit XPath searches run from Python somewhat but should be fully XML compliant and Pythonic, something (IMHO) nobody has managed to do yet.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-50588780780636931662011-03-17T14:31:00.002-04:002011-03-17T15:04:44.976-04:00PyCon 2011 wrapupThis has been an awesome week at PyCon, though I'm completely exhausted.<br /><br />The most notable outcome of several meetings with <a href="http://www.python.org/psf/members/">Python leadership</a> about increasing diversity in the Python community. In my role with PSF's educational outreach programs I'm calling on 3rd party projects working with us to help work to increase gender balance in our community, and as <a href="http://www.pysoy.org/">PySoy</a>'s project maintainer I'm committing us to working seriously on this issue over the next year as an example.<br /><br />Several college-aged women at PyCon knew about <a href="http://socghop.appspot.com/">Google's Summer of Code</a> but believed it was only for "geniuses who've been programming since they were 13" or that they were unqualified for other reasons, even though they're clearly more skilled (and moreso, have a better attitude) than some of the male students we get.<br /><br />To address this, my first step is <a href="http://www.pysoy.org/wiki/SummerOfCode/Diversity">outreach to women</a> for applying to us for Google's Summer of Code in hopes of resolving any miscommunication over required experience and what we're looking for. This is being sent out over the next week so potential applicants leverage of early involvement with mentors that many students have already started on.<br /><br />Once Summer of Code is underway I plan to continue this an internship program in attempt to get more women involved in our community in ways not supported by GSoC such as through QA testing, documentation, video tutorials, and artwork. This program will be for women ages 18+ with no college requirement, though we're still working out many of the other details.<br /><br />For the PyCon sprints I worked on the new "packaging" package for Python 3.3, formerly known as distutils2. My primary interest in this is a packaging.backport module based on <a href="http://pypi.python.org/pypi/3to2/1.0">3to2</a> (which backports code from 3.2 to 2.7) extended with further fixers for 2.7 to 2.6, 2.5, and 2.4. This module will work for packages written for Python 3.3 and above to maintain Python 2 compatibility, and also to backport "packaging" to the standalone distutils2 package for previous Python versions.<br /><br />We managed to get distutils2 largely upgraded for Python 3.3 syntax and the non-PyPI tests running cleanly, though much more work is needed. We have a few months given than Python 3.2 was just released and we're working on a roughly 18 month release cycle.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-85929517004529382892011-02-28T10:48:00.002-05:002011-02-28T11:17:10.078-05:00Sprinting toward Beta-3We've made some massive headway in the last two weeks;<br /><br />First, we have physics processing back. We're using <a href="http://www.ode.org/">ODE</a> for now just to get us through the next release cycle or two, but we've modified its headers (ABI-compatible) to work better with libsoy. Done are the days when we stress over API compatibility, since we've decided to write our own fully integrated (and scaled down) physics processing with <a href="http://code.entropywave.com/projects/orc/">Orc</a> we have a certain freedom to throw their conventions to the wind.<br /><br />How this is going to play out is a slow migration of values to arrays. Body data, for example, will be stored an array inside a scene so SIMD processing can be used more efficiently. The Soy Body class will represent a pointer to that array and include properties and methods for working with its data. The actual storage of that data will be governed by the Scene, giving it the freedom to sort bodies per "island" (groups of bodies connected by group). This is similar to how ODE processes bodies already and we'll be likely using their code as a starting place.<br /><br />Second, we have the basic types for rendering. At the moment this is just a rotating cube, but that involved roughly 6 classes operating in both GObject and Python domains. The next step is soy.models.Mesh for rendering more complex shapes.<br /><br />Third and most exciting, our humble <a href="http://www.pysoy.org/">game engine</a> can now run as a <a href="http://getfirefox.org/">Firefox</a> plugin! There's some bugs to be worked out still, but "embedded" support has been added to our soy.widgets.Window class that allows it to run inside virtually any application.<br /><br />With luck and determination we'll have the next release out by the end of the month. I've dialed back our goals considerably given that its a rewrite from our last beta release but our release cycle will be much shorter now that the rewrite is finishing up.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-17332506959561010652011-02-19T15:54:00.003-05:002011-02-19T16:23:04.165-05:00New import (and export) mechanismI've started writing a new import mechanism for <a href="http://www.pysoy.org/wiki/Formats/soy">.soy</a> files, replacing the old <code>soy.transports</code> module with a <a href="http://docs.python.org/dev/library/sys.html#sys.path_hooks">importer</a> that allows .soy files to be loaded as modules with the standard Python <code>import</code> command.<br /><br />With the old method, "transport" objects acted as dicts of objects in the .soy files and could be loaded and saved as such. The resulting code looked like this:<code><br />import soy<br />MyGame = soy.transports.File('data/MyGame.soy')<br />window = MyGame['window']<br />redblock = MyGame['redblock']<br /></code>(etc)<br /><br />Now, the new .soy importer is added to sys.path_hooks when PySoy is first initialized and applys to any directory added after. When finished, the above code could be rewritten as:<code><br />import soy<br />sys.path.append('data/')<br />import MyGame<br /></code><br /><br />No "etc" needed because in that one import command data/MyGame.soy is loaded and made available directly in a module called MyGame. Much more pythonic, IMHO.<br /><br />A separate class will be written next called <code>soy.Exporter</code> which <a href="http://www.pysoy.org/">PySoy</a> objects can be added and removed from, then saved to a file such as this example which modifies the previously imported MyGame module:<code><br />archive = soy.Exporter(MyGame)<br />archive.grasstex = grass_texture<br />del(archive.bricktex)<br />archive('data/MyGame2.soy')<br /></code><br /><br />Similarly, two archives could be combined like this:<code><br />archive = soy.Exporter(MyGame)<br />archive2 = soy.Exporter(CharacterFoo)<br />archive += archive2<br />archive('data/MyGame3.soy')<br /></code><br /><br />The reasoning behind this is in most cases .soy files will be written by <a href="http://www.gimp.org/">GIMP</a> or <a href="http://www.blender.org/">Blender</a>, so the most common use case for the <code>soy.Exporter</code> class is for combining and pruning the .soy files created by these tools.<br /><br />As always, constructive/critical feedback welcome and appreciated.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-13719964262916818412011-02-05T19:39:00.002-05:002011-02-05T19:52:44.500-05:00libsoy migration milestoneAfter a great deal of time (I'd rather not admit how long) we've finally managed to link the new <a href="http://hg.pysoy.org/libsoy">libsoy code</a> in with <a href="http://www.python.org/">Python</a>.<br /><br />This afternoon (around 4:30) the following code opened a PySoy window on the screen:<br /><pre><br />arc@khonsu ~/work/pysoy $ python3<br />Python 3.1.3 (r313:86834, Dec 5 2010, 09:55:24) <br />[GCC 4.5.1] on linux2<br />Type "help", "copyright", "credits" or "license" for more information.<br />>>> import soy.widgets<br />>>> w = soy.widgets.Window()<br />>>> <br />(soy:16042): Gdk-CRITICAL **: IA__gdk_window_set_title: assertion `title != NULL' failed<br />IRQ's not enabled, falling back to busy waits: 2 0<br />OpenGL version 2.1 Mesa 7.10<br />GL_ARB_vertex_buffer_object: Yes<br /></pre><br />While incomplete and, well, its only a window, this is the first time we've gotten the two pieces to work together. <a href="https://www.ohloh.net/accounts/kapace">David Czech</a> and I have spent the rest of the evening fixing both build systems (libsoy and <a href="http://www.pysoy.org/">PySoy</a>) and getting ready to copy the window binding code for the rest of the engine.<br /><br />Our largest failure in this has been diverting so much energy into <a href="http://live.gnome.org/GObjectIntrospection">GObject Introspection</a>. While its a really great idea and I'd love to see it evolve, its still so deep alpha that I can't imagine anyone but Gnome developers getting much practical use out of it. I would like to finish up <a href="http://gtypes.org/">GTypes</a> at some point, but either the XML format for .gir or any level of documentation on typelib needs to be written first. I've wasted far too much time already trying to reverse engineer these from examples and their source that should have been put into my own projects.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-52078767129615539792010-12-30T05:02:00.003-05:002010-12-30T05:19:32.957-05:00common Orc opcodesI've been going through <a href="http://liboil.freedesktop.org/wiki/">liboil</a>'s 0.3 source to rewrite the <a href="http://liboil.freedesktop.org/documentation/liboil-liboilfuncs-pixel.html#oil-yuv2rgbx-sub2-u8">oil_yuv2rgbx_sub2_u8 function</a> we use for <a href="http://theora.org/">Theora</a> decoding to Orc pseudo-assembly code.<br /><br />Because the <a href="http://code.entropywave.com/projects/orc/">Orc</a> <a href="http://code.entropywave.com/documentation/orc/orc-opcodes.html">opcode documentation</a> splits opcode description and processor support between two tables, for reference I wrote a quick Python script to build a table of Orc opcodes common to <a href="http://en.wikipedia.org/wiki/Streaming_SIMD_Extensions">SSE</a> (x86), <a href="http://en.wikipedia.org/wiki/AltiVec">Altivec</a> (PPC/Cell), and <a href="http://en.wikipedia.org/wiki/ARM_architecture#Advanced_SIMD_.28NEON.29">NEON</a> (Arm Cortex) processors.<br /><br />Here's that table for reference, at least until I put the time to format it for a wiki:<br /><br /><table><tr><th>opcode</th><th>dst</th><th>src1</th><th>src2</th><th>description</th><th>pseudo code</th></tr><tr><td>absb</td><td>1 </td><td>1 </td><td> </td><td>absolute value </td><td>(a < 0) ? -a : a<br /></td></tr><tr><td>addb</td><td>1 </td><td>1 </td><td>1 </td><td>add </td><td>a + b<br /></td></tr><tr><td>addssb</td><td>1 </td><td>1 </td><td>1 </td><td>add with signed saturate </td><td>clamp(a + b)<br /></td></tr><tr><td>addusb</td><td>1 </td><td>1 </td><td>1 </td><td>add with unsigned saturate </td><td>clamp(a + b)<br /></td></tr><tr><td>andb</td><td>1 </td><td>1 </td><td>1 </td><td>bitwise AND </td><td>a & b<br /></td></tr><tr><td>andnb</td><td>1 </td><td>1 </td><td>1 </td><td>bitwise AND NOT </td><td>a & (~b)<br /></td></tr><tr><td>avgsb</td><td>1 </td><td>1 </td><td>1 </td><td>signed average </td><td>(a + b + 1)>>1<br /></td></tr><tr><td>avgub</td><td>1 </td><td>1 </td><td>1 </td><td>unsigned average </td><td>(a + b + 1)>>1<br /></td></tr><tr><td>cmpeqb</td><td>1 </td><td>1 </td><td>1 </td><td>compare equal </td><td>(a == b) ? (~0) : 0<br /></td></tr><tr><td>cmpgtsb</td><td>1 </td><td>1 </td><td>1 </td><td>compare greater than </td><td>(a > b) ? (~0) : 0<br /></td></tr><tr><td>copyb</td><td>1 </td><td>1 </td><td> </td><td>copy </td><td>a<br /></td></tr><tr><td>loadb</td><td>1 </td><td>1 </td><td> </td><td>load from memory </td><td>array[i]<br /></td></tr><tr><td>loadpb</td><td>1 </td><td>1 </td><td> </td><td>load parameter or constant </td><td>scalar<br /></td></tr><tr><td>maxsb</td><td>1 </td><td>1 </td><td>1 </td><td>signed maximum </td><td>(a > b) ? a : b<br /></td></tr><tr><td>maxub</td><td>1 </td><td>1 </td><td>1 </td><td>unsigned maximum </td><td>(a > b) ? a : b<br /></td></tr><tr><td>minsb</td><td>1 </td><td>1 </td><td>1 </td><td>signed minimum </td><td>(a < b) ? a : b<br /></td></tr><tr><td>minub</td><td>1 </td><td>1 </td><td>1 </td><td>unsigned minimum </td><td>(a < b) ? a : b<br /></td></tr><tr><td>mullb</td><td>1 </td><td>1 </td><td>1 </td><td>low bits of multiply </td><td>a * b<br /></td></tr><tr><td>mulhsb</td><td>1 </td><td>1 </td><td>1 </td><td>high bits of signed multiply </td><td>(a * b) >> 8<br /></td></tr><tr><td>mulhub</td><td>1 </td><td>1 </td><td>1 </td><td>high bits of unsigned multiply </td><td>(a * b) >> 8<br /></td></tr><tr><td>orb</td><td>1 </td><td>1 </td><td>1 </td><td>bitwise or </td><td>a | b<br /></td></tr><tr><td>shlb</td><td>1 </td><td>1 </td><td>1S </td><td>shift left </td><td>a << b<br /></td></tr><tr><td>shrsb</td><td>1 </td><td>1 </td><td>1S </td><td>signed shift right </td><td>a >> b<br /></td></tr><tr><td>shrub</td><td>1 </td><td>1 </td><td>1S </td><td>unsigned shift right </td><td>a >> b<br /></td></tr><tr><td>signb</td><td>1 </td><td>1 </td><td> </td><td>sign </td><td>sign(a)<br /></td></tr><tr><td>storeb</td><td>1 </td><td>1 </td><td> </td><td>store to memory </td><td>special<br /></td></tr><tr><td>subb</td><td>1 </td><td>1 </td><td>1 </td><td>subtract </td><td>a - b<br /></td></tr><tr><td>subssb</td><td>1 </td><td>1 </td><td>1 </td><td>subtract with signed saturate </td><td>clamp(a - b)<br /></td></tr><tr><td>subusb</td><td>1 </td><td>1 </td><td>1 </td><td>subtract with unsigned saturate </td><td>clamp(a - b)<br /></td></tr><tr><td>xorb</td><td>1 </td><td>1 </td><td>1 </td><td>bitwise XOR </td><td>a ^ b<br /></td></tr><tr><td>absw</td><td>2 </td><td>2 </td><td> </td><td>absolute value </td><td>(a < 0) ? -a : a<br /></td></tr><tr><td>addw</td><td>2 </td><td>2 </td><td>2 </td><td>add </td><td>a + b<br /></td></tr><tr><td>addssw</td><td>2 </td><td>2 </td><td>2 </td><td>add with signed saturate </td><td>clamp(a + b)<br /></td></tr><tr><td>addusw</td><td>2 </td><td>2 </td><td>2 </td><td>add with unsigned saturate </td><td>clamp(a + b)<br /></td></tr><tr><td>andw</td><td>2 </td><td>2 </td><td>2 </td><td>bitwise AND </td><td>a & b<br /></td></tr><tr><td>andnw</td><td>2 </td><td>2 </td><td>2 </td><td>bitwise AND NOT </td><td>a & (~b)<br /></td></tr><tr><td>avgsw</td><td>2 </td><td>2 </td><td>2 </td><td>signed average </td><td>(a + b + 1)>>1<br /></td></tr><tr><td>avguw</td><td>2 </td><td>2 </td><td>2 </td><td>unsigned average </td><td>(a + b + 1)>>1<br /></td></tr><tr><td>cmpeqw</td><td>2 </td><td>2 </td><td>2 </td><td>compare equal </td><td>(a == b) ? (~0) : 0<br /></td></tr><tr><td>cmpgtsw</td><td>2 </td><td>2 </td><td>2 </td><td>compare greater than </td><td>(a > b) ? (~0) : 0<br /></td></tr><tr><td>copyw</td><td>2 </td><td>2 </td><td> </td><td>copy </td><td>a<br /></td></tr><tr><td>div255w</td><td>2 </td><td>2 </td><td> </td><td>divide by 255 </td><td>a/255<br /></td></tr><tr><td>loadw</td><td>2 </td><td>2 </td><td> </td><td>load from memory </td><td>array[i]<br /></td></tr><tr><td>loadpw</td><td>2 </td><td>2 </td><td> </td><td>load parameter or constant </td><td>scalar<br /></td></tr><tr><td>maxsw</td><td>2 </td><td>2 </td><td>2 </td><td>signed maximum </td><td>(a > b) ? a : b<br /></td></tr><tr><td>maxuw</td><td>2 </td><td>2 </td><td>2 </td><td>unsigned maximum </td><td>(a > b) ? a : b<br /></td></tr><tr><td>minsw</td><td>2 </td><td>2 </td><td>2 </td><td>signed minimum </td><td>(a < b) ? a : b<br /></td></tr><tr><td>minuw</td><td>2 </td><td>2 </td><td>2 </td><td>unsigned minimum </td><td>(a < b) ? a : b<br /></td></tr><tr><td>mullw</td><td>2 </td><td>2 </td><td>2 </td><td>low bits of multiply </td><td>a * b<br /></td></tr><tr><td>mulhsw</td><td>2 </td><td>2 </td><td>2 </td><td>high bits of signed multiply </td><td>(a * b) >> 8<br /></td></tr><tr><td>mulhuw</td><td>2 </td><td>2 </td><td>2 </td><td>high bits of unsigned multiply </td><td>(a * b) >> 8<br /></td></tr><tr><td>orw</td><td>2 </td><td>2 </td><td>2 </td><td>bitwise or </td><td>a | b<br /></td></tr><tr><td>shlw</td><td>2 </td><td>2 </td><td>2S </td><td>shift left </td><td>a << b<br /></td></tr><tr><td>shrsw</td><td>2 </td><td>2 </td><td>2S </td><td>signed shift right </td><td>a >> b<br /></td></tr><tr><td>shruw</td><td>2 </td><td>2 </td><td>2S </td><td>unsigned shift right </td><td>a >> b<br /></td></tr><tr><td>signw</td><td>2 </td><td>2 </td><td> </td><td>sign </td><td>sign(a)<br /></td></tr><tr><td>storew</td><td>2 </td><td>2 </td><td> </td><td>store to memory </td><td>special<br /></td></tr><tr><td>subw</td><td>2 </td><td>2 </td><td>2 </td><td>subtract </td><td>a - b<br /></td></tr><tr><td>subssw</td><td>2 </td><td>2 </td><td>2 </td><td>subtract with signed saturate </td><td>clamp(a - b)<br /></td></tr><tr><td>subusw</td><td>2 </td><td>2 </td><td>2 </td><td>subtract with unsigned saturate </td><td>clamp(a - b)<br /></td></tr><tr><td>xorw</td><td>2 </td><td>2 </td><td>2 </td><td>bitwise XOR </td><td>a ^ b<br /></td></tr><tr><td>absl</td><td>4 </td><td>4 </td><td> </td><td>absolute value </td><td>(a < 0) ? -a : a<br /></td></tr><tr><td>addl</td><td>4 </td><td>4 </td><td>4 </td><td>add </td><td>a + b<br /></td></tr><tr><td>addssl</td><td>4 </td><td>4 </td><td>4 </td><td>add with signed saturate </td><td>clamp(a + b)<br /></td></tr><tr><td>addusl</td><td>4 </td><td>4 </td><td>4 </td><td>add with unsigned saturate </td><td>clamp(a + b)<br /></td></tr><tr><td>andl</td><td>4 </td><td>4 </td><td>4 </td><td>bitwise AND </td><td>a & b<br /></td></tr><tr><td>andnl</td><td>4 </td><td>4 </td><td>4 </td><td>bitwise AND NOT </td><td>a & (~b)<br /></td></tr><tr><td>avgsl</td><td>4 </td><td>4 </td><td>4 </td><td>signed average </td><td>(a + b + 1)>>1<br /></td></tr><tr><td>avgul</td><td>4 </td><td>4 </td><td>4 </td><td>unsigned average </td><td>(a + b + 1)>>1<br /></td></tr><tr><td>cmpeql</td><td>4 </td><td>4 </td><td>4 </td><td>compare equal </td><td>(a == b) ? (~0) : 0<br /></td></tr><tr><td>cmpgtsl</td><td>4 </td><td>4 </td><td>4 </td><td>compare greater than </td><td>(a > b) ? (~0) : 0<br /></td></tr><tr><td>copyl</td><td>4 </td><td>4 </td><td> </td><td>copy </td><td>a<br /></td></tr><tr><td>loadl</td><td>4 </td><td>4 </td><td> </td><td>load from memory </td><td>array[i]<br /></td></tr><tr><td>loadpl</td><td>4 </td><td>4 </td><td> </td><td>load parameter or constant </td><td>scalar<br /></td></tr><tr><td>maxsl</td><td>4 </td><td>4 </td><td>4 </td><td>signed maximum </td><td>(a > b) ? a : b<br /></td></tr><tr><td>maxul</td><td>4 </td><td>4 </td><td>4 </td><td>unsigned maximum </td><td>(a > b) ? a : b<br /></td></tr><tr><td>minsl</td><td>4 </td><td>4 </td><td>4 </td><td>signed minimum </td><td>(a < b) ? a : b<br /></td></tr><tr><td>minul</td><td>4 </td><td>4 </td><td>4 </td><td>unsigned minimum </td><td>(a < b) ? a : b<br /></td></tr><tr><td>orl</td><td>4 </td><td>4 </td><td>4 </td><td>bitwise or </td><td>a | b<br /></td></tr><tr><td>shll</td><td>4 </td><td>4 </td><td>4S </td><td>shift left </td><td>a << b<br /></td></tr><tr><td>shrsl</td><td>4 </td><td>4 </td><td>4S </td><td>signed shift right </td><td>a >> b<br /></td></tr><tr><td>shrul</td><td>4 </td><td>4 </td><td>4S </td><td>unsigned shift right </td><td>a >> b<br /></td></tr><tr><td>signl</td><td>4 </td><td>4 </td><td> </td><td>sign </td><td>sign(a)<br /></td></tr><tr><td>storel</td><td>4 </td><td>4 </td><td> </td><td>store to memory </td><td>special<br /></td></tr><tr><td>subl</td><td>4 </td><td>4 </td><td>4 </td><td>subtract </td><td>a - b<br /></td></tr><tr><td>subssl</td><td>4 </td><td>4 </td><td>4 </td><td>subtract with signed saturate </td><td>clamp(a - b)<br /></td></tr><tr><td>subusl</td><td>4 </td><td>4 </td><td>4 </td><td>subtract with unsigned saturate </td><td>clamp(a - b)<br /></td></tr><tr><td>xorl</td><td>4 </td><td>4 </td><td>4 </td><td>bitwise XOR </td><td>a ^ b<br /></td></tr><tr><td>loadq</td><td>8 </td><td>8 </td><td> </td><td>load from memory </td><td>array[i]<br /></td></tr><tr><td>storeq</td><td>8 </td><td>8 </td><td> </td><td>store to memory </td><td>special<br /></td></tr><tr><td>splatw3q</td><td>8 </td><td>8 </td><td> </td><td>duplicates high 16-bits to lower 48 bits </td><td>special<br /></td></tr><tr><td>convsbw</td><td>2 </td><td>1 </td><td> </td><td>convert signed </td><td>a<br /></td></tr><tr><td>convubw</td><td>2 </td><td>1 </td><td> </td><td>convert unsigned </td><td>a<br /></td></tr><tr><td>splatbw</td><td>2 </td><td>1 </td><td> </td><td>duplicates 8 bits to both halfs of 16 bits </td><td>special<br /></td></tr><tr><td>splatbl</td><td>4 </td><td>1 </td><td> </td><td>duplicates 8 bits to all parts of 32 bits </td><td>special<br /></td></tr><tr><td>convswl</td><td>4 </td><td>2 </td><td> </td><td>convert signed </td><td>a<br /></td></tr><tr><td>convuwl</td><td>4 </td><td>2 </td><td> </td><td>convert unsigned </td><td>a<br /></td></tr><tr><td>convslq</td><td>8 </td><td>4 </td><td> </td><td>signed convert </td><td>a<br /></td></tr><tr><td>convulq</td><td>8 </td><td>4 </td><td> </td><td>unsigned convert </td><td>a<br /></td></tr><tr><td>convwb</td><td>1 </td><td>2 </td><td> </td><td>convert </td><td>a<br /></td></tr><tr><td>convhwb</td><td>1 </td><td>2 </td><td> </td><td>shift and convert </td><td>a>>8<br /></td></tr><tr><td>convssswb</td><td>1 </td><td>2 </td><td> </td><td>convert signed to signed with saturation </td><td>clamp(a)<br /></td></tr><tr><td>convsuswb</td><td>1 </td><td>2 </td><td> </td><td>convert signed to unsigned with saturation </td><td>clamp(a)<br /></td></tr><tr><td>convuuswb</td><td>1 </td><td>2 </td><td> </td><td>convert unsigned to unsigned with saturation </td><td>clamp(a)<br /></td></tr><tr><td>convlw</td><td>2 </td><td>4 </td><td> </td><td>convert </td><td>a<br /></td></tr><tr><td>convhlw</td><td>2 </td><td>4 </td><td> </td><td>shift and convert </td><td>a>>16<br /></td></tr><tr><td>convssslw</td><td>2 </td><td>4 </td><td> </td><td>convert signed to signed with saturation </td><td>clamp(a)<br /></td></tr><tr><td>convql</td><td>4 </td><td>8 </td><td> </td><td>convert </td><td>a<br /></td></tr><tr><td>mulsbw</td><td>2 </td><td>1 </td><td>1 </td><td>multiply signed </td><td>a * b<br /></td></tr><tr><td>mulubw</td><td>2 </td><td>1 </td><td>1 </td><td>multiply unsigned </td><td>a * b<br /></td></tr><tr><td>mulswl</td><td>4 </td><td>2 </td><td>2 </td><td>multiply signed </td><td>a * b<br /></td></tr><tr><td>muluwl</td><td>4 </td><td>2 </td><td>2 </td><td>multiply unsigned </td><td>a * b<br /></td></tr><tr><td>accl</td><td>4 </td><td>4 </td><td> </td><td>accumulate </td><td>+= a<br /></td></tr><tr><td>swapw</td><td>2 </td><td>2 </td><td> </td><td>endianness swap </td><td>special<br /></td></tr><tr><td>swapl</td><td>4 </td><td>4 </td><td> </td><td>endianness swap </td><td>special<br /></td></tr><tr><td>select0wb</td><td>1 </td><td>2 </td><td> </td><td>select first half </td><td>special<br /></td></tr><tr><td>select1wb</td><td>1 </td><td>2 </td><td> </td><td>select second half </td><td>special<br /></td></tr><tr><td>select0lw</td><td>2 </td><td>4 </td><td> </td><td>select first half </td><td>special<br /></td></tr><tr><td>select1lw</td><td>2 </td><td>4 </td><td> </td><td>select second half </td><td>special<br /></td></tr><tr><td>mergewl</td><td>4 </td><td>2 </td><td>2 </td><td>merge halves </td><td>special<br /></td></tr><tr><td>mergebw</td><td>2 </td><td>1 </td><td>1 </td><td>merge halves </td><td>special<br /></td></tr><tr><td>splitlw</td><td>2 </td><td>4 </td><td> </td><td>split first/second words </td><td>special<br /></td></tr><tr><td>splitwb</td><td>1 </td><td>2 </td><td> </td><td>split first/second bytes </td><td>special<br /></td></tr><tr><td>addf</td><td>4 </td><td>4 </td><td>4 </td><td>add </td><td>a + b<br /></td></tr><tr><td>subf</td><td>4 </td><td>4 </td><td>4 </td><td>subtract </td><td>a - b<br /></td></tr><tr><td>mulf</td><td>4 </td><td>4 </td><td>4 </td><td>multiply </td><td>a * b<br /></td></tr><tr><td>maxf</td><td>4 </td><td>4 </td><td>4 </td><td>maximum </td><td>max(a,b)<br /></td></tr><tr><td>minf</td><td>4 </td><td>4 </td><td>4 </td><td>minimum </td><td>min(a,b)<br /></td></tr><tr><td>cmpeqf</td><td>4 </td><td>4 </td><td>4 </td><td>compare equal </td><td>(a == b) ? (~0) : 0<br /></td></tr><tr><td>convfl</td><td>4 </td><td>4 </td><td> </td><td>convert float point to integer </td><td>a<br /></td></tr><tr><td>convlf</td><td>4 </td><td>4 </td><td> </td><td>convert integer to floating point </td><td>a<br /></td></tr></table>Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com3tag:blogger.com,1999:blog-28358787.post-33185882873522877822010-12-29T04:17:00.003-05:002010-12-29T22:57:43.597-05:00stuck, back to PySoyI've been working on a streaming XML parser for Python, but need a break. At this point there's no way Concordance is getting out Jan 1st, but certainly by the end of Winter.<br /><br />Our libsoy migration process <a href="http://www.pysoy.org/">PySoy</a> got pretty far. We were migrating from Pyrex to <a href="http://live.gnome.org/Genie">Genie</a>, essentially moving the core engine from PyObject to GObject to remove Python dependency in game clients and enable further multicore processing on both client and servers. Much of the rendering area of the engine has been migrated, but the process has been held up in two areas;<br /><br />First, while libsoy is in pretty good shape, we still lack <a href="http://www.python.org/">Python</a> bindings - aka PySoy itself, which is what we intend games to be written and run with. Our original plan to use GObject Introspection failed in a horrible mess that I've documented in previous postings, we've looked at using SWIG and even building our own bindings generation with little measurable success. In order to get us moving forward again I'm going to just drop out some .c templates and write the custom wrapper classes by hand. The time it'd take to write and maintain these cannot possibly be greater than the time we've wasted talking about a more elegant solution that only exists conceptually.<br /><br />When GObject Introspection reaches a state of even remote maturity, where it can offer a Pythonic API, we'll look at it again. We'd even help get it there if the current GIR developers would just document the .gir XML schema or typelib format so we wouldn't have to refer to their source code as the sole definition of these.<br /><br />Second is our physics code. As I've posted, <a href="http://www.ode.org/">ODE</a> worked for us in the past but has numerous issues with packaging for various Linux distros (and poor features, slow, and extremely difficult to port to mobile devices). We attempted to migrate to Bullet but this burned us out - virtually no work has gone into that in the past 6 months. We're all pretty frustrated with Bullet's haphazard API (whereas ODE is fairly clean) and the C++ only API doesn't play well with GObject (or anything other than c++ for that matter). Bullet's C API is minimal at best.<br /><br />When it comes right down to it, the biggest barrier we face with physics is processing power on mobile devices, an issue that using Bullet would not solve. Most of the devices we're interested in include ARM6/7 processors from Qualcomm or TI. Many do not include a FPU (floating point unit), but they all seem to offer a fairly powerful DSP used extensively for processing multimedia. We do not, however, want to rewrite and maintain our physics processing for each platform.<br /><br />A solution I've come up with is to write our physics, greatly simplified from even what ODE offers, using <a href="http://code.entropywave.com/projects/orc/">Orc</a>. It's yet another metalanguage (first Pyrex, then Vala/Genie, now this..), but the successor to liboil (which we and much of the Gnome community use) and already supports many interesting platforms.<br /><br />My plan is to first migrate our liboil-based YUV-RGB conversion code to Orc to get my feet wet, then implement a greatly simplified collision system using it, and expect the next release (or two) to still use ODE for at least rigid body physics with the plan to eventually replace even that with our own physics solver. It should be much faster, and the same Orc code we write now should be able to compile to DSP code for Android handsets and other mobile devices in the future.<br /><br />Orc already supports ARM Cortex (NEON), so if we were to finish this work today we'd be able to run PySoy clients on more modern Android handsets without touching DSP code. DSP support in Orc would also be very useful for future hardware for PySoy game servers.<br /><br />While we'd all really like to get the next PySoy release out ASAP, we'd also like to avoid rewriting the engine again down the road.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-38908224407524489922010-12-08T06:37:00.003-05:002010-12-08T07:14:30.141-05:00XML parsing in PythonIts been a couple months, so I'm going to give a brief update on what I've been working on.<br /><br /><a href="http://concordance-xmpp.org/">Concordance</a> is getting close to release, I plan to have the first release (0.1) out January 1st. More on this toward the end of December.<br /><br />One of the roadblocks I've hit (again and again) is the lack of a decent XML parsing package for <a href="http://python.org/">Python</a>. The standard library is a shame when it comes to XML; at least four different modules (expat, sax, dom, etree) to choose from and none of them support even <a href="http://en.wikipedia.org/wiki/XPath">XPath</a>. The most popular option, etree (or <a href="http://effbot.org/zone/element-index.htm">ElementTree</a>), cannot even process an XML file with the namespace prefix intact.<br /><br />There's <a href="http://codespeak.net/lxml/">lxml</a>, which offers an etree-compatible API and fixes many of ElementTree's major faults (namespace prefix preservation, xpath/xslt support) but still cannot handle stream processing and, due to ElementTree's API, does not expose multiple text nodes broken up by a child element such as "<div>first string <br/> second string</div>".<br /><br />To support <a href="http://xmpp.org/">XMPP</a> streams we need to use expat or sax to handle the stream event-by-event, since the full XML document is only available once the root element closes at the end of the stream, but the direct children of the root element (what we call "stanzas" in XMPP) need to be processed as complete objects. While we may be able to hack something together using lxml, it would likely be less work than to implement a new XML parsing package. As long as the resulting API doesn't diverge very greatly etree the work necessary to switch should be minimal.<br /><br />Beside this I've been working on a host of different packages around Concordance, from getting a javascript BOSH/XMPP library together to getting distutils2 ready for Python 3. I've even managed to ship a pitiful little serial library for Python 3, <a href="http://pypi.python.org/pypi/PyTTY">PyTTY</a> that we're using to interface with some <a href="http://arduino.cc/">Arduino</a>s.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-86549347438712773242010-07-02T15:13:00.003-04:002010-07-02T15:48:47.055-04:00Diaspora: the $200k scam on free softwareAfter a month of work, its bluntly clear the community <a href="http://www.kickstarter.com/projects/196017994/diaspora-the-personally-controlled-do-it-all-distr">has been scammed for $200k</a>.<br /><br />Four NYU students posted a video and solicited funding on the <a href="http://www.kickstarter.com/">kickstarter</a> website, promising to build an open source, pro-privacy alternative to Facebook. The community response was astonishing. Its too bad the people pledging didn't research these students, for example, checking to see if they have <a href="http://www.ohloh.net/">Ohloh</a> pages with their past contributions to other projects.<br /><br />The first and most severe, after a month of work, they have posted that they have running code but will not allow the community (including the people who paid for it) to view until its "complete". This is commonly referred to as the <a href="http://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar">Cathedral model</a>. People have invested a great deal of trust in them to develop the code they have promised to, the least they could offer is read only access to their VCS. Instead they posted a video demonstrating it, which poses the second problem.<br /><br />Their claim to use the "latest and greatest in web standards" fell flat on its face in their choice of method to post their video. Instead of using HTML/5 <video>, which they could have even used Ogg with the <a href="http://www.theora.org/cortado/">Cortado</a> java player as fallback for MSIE and Safari, they used a flash player. Perhaps they don't realize Flash is not a web standard, or that HTML/5 has built-in video support, in either case its a really bad sign.<br /><br />The most blunt demonstration of their lack of skill and commitment is in their use of <a href=" http://hacks.mozilla.org/2010/04/websockets-in-firefox/">websockets</a>. At this point, websockets is still in heavy draft state with the few browsers/servers implementing it all running a slightly different, incompatible variants, none of which will be the final draft which is (at least) months off. Moreso, they propose to implement chat and gaming over their own websockets-based protocol rather than using existing standards such as <a href="http://xmpp.org/">XMPP</a>.<br /><br />XMPP is being used by a wide number of social network services already including Google, Livejournal, and Facebook. Its an established, widely implemented standard recognized by the IETF and the community at large for federated, real-time exchange of chat and presence (status). Beyond chat, XMPP has support for <a href="http://xmpp.org/extensions/xep-0060.html">PubSub</a>, <a href="http://xmpp.org/extensions/xep-0004.html">Data Forms</a>, and <a href="http://xmpp.org/extensions/xep-0050.html">Ad-Hoc Commands</a>. There is virtually nothing Facebook, or Diaspora, aims to accomplish that cannot be implemented using these existing standards.<br /><br />What these four students are building is lock-in, if they finish at all. Instead of replacing the walled gardens of Facebook and other proprietary social network sites, they aim to build a network of little sites running the same software, only able to federate with each other, and only able to be extended with their custom APIs. These sites will not federate with other distributed social network efforts being built such as <a href="http://www.movim.eu/">movim</a> and <a href="http://concordance-xmpp.org/">concordance</a>.<br /><br />Hopefully the community will learn a lesson from this. Pledge money to experienced developers with demonstratable code, not charismatic students pledging to work on free software only if they're paid to do it.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com2tag:blogger.com,1999:blog-28358787.post-26472986999183505732010-06-28T00:01:00.002-04:002010-06-28T00:38:35.987-04:00Using Android without Google (part 1)As <a href="http://identi.ca/magicfab">magicfab</a> on <a href="http://identi.ca/">identi.ca</a> <a href="http://identi.ca/conversation/38114077#notice-38455526">requested</a>, I'm starting a journal of my experience using <a href="http://www.android.com/">Android</a> without Google's proprietary apps.<br /><br />I own an <a href="http://www.htc.com/us/products/droid-eris-verizon">HTC Eris</a> phone which originally came with Android 1.5. A few months ago a firmware was released by a 3rd party which allows users to gain root on their phone (about time) and I've been experimenting with this since. My primary interests were removing HTC's "Sense" UI so I could use a different <a href="http://code.google.com/p/softkeyboard/">soft keyboard</a> and removing the backdoors HTC, Verizon, and Google installed in the phone.<br /><br />After some <a href="http://arcriley.blogspot.com/2010/05/killer-android-app.html">disturbing problems</a> with the VanillaDroid firmware, I switched to CyanogenEris 3.0 last week. Not only did this new build not come with HTC's Sense, thanks to Google hitting them with a threatening DMCA letter, the firmware came without Google's apps either. This includes GMail, GTalk, Google Maps and Market.<br /><br />Everything works on the firmware except for USB tethering (which is a feature of the firmware). I started by using the browser to download the <a href="http://code.google.com/p/zxing/">Barcode Scanner</a> and then used that to install other apps via QR code:<ul><li><a href="http://code.google.com/p/android-wifi-tether/">wifi tether for root</a></li><li><a href="http://code.google.com/p/k9mail/">k9 mail</a></li><li><a href="http://code.google.com/p/connectbot/">connectbot (ssh client)</a></li><li><a href="http://mustard.macno.org/">mustard (identi.ca client)</a></li></ul><br />The <a href="http://code.google.com/p/foursquared/">Foursquare app</a> I was previously using refuses to install (likely because it depends on Google Maps app) and I haven't figured out how to install the dvorak keyboard add-on to <a href="http://code.google.com/p/softkeyboard/">soft keyboard</a>.<br /><br />I have the standard Android Contacts app, but without Google it does not sync to my Google contacts. I plan to remove my contacts from Google and write a small app that provides ContactsProvider2 via <a href="http://xmpp.org/">XMPP</a> in an effort to decentralize and federate contact syncing.<br /><br />I found the web version of GMail superior to the old app, though I do miss the Google Maps app (there are various OpenStreetMaps alternatives but they don't have the same features). The biggest thing I miss is the GTalk app, though its pitiful as far as XMPP clients go and we should be able to do a lot better.<br /><br />Most importantly, my phone is now 99% <a href="http://softwarefreedom.org/">free software</a> with only a few of HTC's drivers left to be reverse engineered. Even for an advertising company Google is really stretching the truth when they call Android "open source" - but it should be an attainable goal.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com6tag:blogger.com,1999:blog-28358787.post-63827797567003934532010-06-27T02:08:00.002-04:002010-06-27T02:17:34.475-04:00concordance progressWhile dredging through framework API design, its sometimes difficult to sense how close you are to finish. Tonight the <a href="http://concordance-xmpp.org/">Concordance</a> "pure Python" redesign reached a critical point - being able to add an extension and respond to an <a href="http://xmpp.org/">XMPP</a> <iq> stanza:<pre><br />>>> import concordance<br />>>> import concordance.extensions.ping<br />>>> s = concordance.Service('host')<br />>>> s.append(concordance.extensions.ping.Ping)<br />>>> import xml.etree.ElementTree as et<br />>>> st = et.fromstring('<iq from="arc@host" id="f00" to="host" type="get"><ping xmlns="urn:xmpp:ping" /></iq>')<br />>>> s.stream(st)<br /><iq from="host" id="f00" to="arc@host" type="result" /><br />>>> <br /></pre><br />That's a correct response to an <a href="http://xmpp.org/extensions/xep-0199.html">XMPP Ping</a> request, one of the simplest XMPP extensions. Many of these basic extensions will be combined into a single extension module and a ready-made Service class, this is just proof that it works.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-88262183118908113452010-06-16T21:50:00.002-04:002010-06-16T21:59:27.894-04:00Concordance stickers ordered<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://media.pysoy.org/img/concordance-xmpp-192.png"><img style="float:left; margin:0 10px 10px 0;cursor:pointer; cursor:hand;width: 192px; height: 192px;" src="http://media.pysoy.org/img/concordance-xmpp-192.png" border="0" alt="Concordance-XMPP Logo" /></a><br />I just ordered 1000 2" x 2" vinyl laptop stickers with the Concordance logo on white. They should arrive by the end of the month. Contributors can send me their mailing address to receive one along with one of the new Python 3 stickers from the PSF.<br /><br />New PySoy stickers will be made as soon as we have a new logo.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-47938598655717348322010-06-06T04:40:00.002-04:002010-06-06T04:53:57.031-04:00Happy 4th Birthday, PySoyOn this day, June 6th 2006, the <a href="http://www.pysoy.org/">PySoy Project</a> was born.<br /><br />We've come a long way from the small homebrew <a href="http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/">Pyrex</a>-based game engine we started from. We've built a solid development team, spun off numerous related projects, contributed hundreds of bug reports and patches for libraries and applications, and grown close to the <a href="http://www.python.org/">Python</a>, <a href="http://www.gnome.org/">Gnome</a>, and <a href="http://www.xmpp.org/">XMPP</a> communities.<br /><br />Currently halfway through our most ambitious endeavour yet, moving our codebase to the GObject model and preparing the engine and related projects for cloud gaming, there's a bright future ahead for this tiny project. While its sometimes frustrating that we're still not to 1.0 release, looking back to what we started from four years ago its clear we've made tremendous progress.<br /><br />Here's to another exciting year!Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com0tag:blogger.com,1999:blog-28358787.post-20909516055971662272010-05-27T03:13:00.002-04:002010-05-27T03:32:18.166-04:00A killer Android appI've been running the Vanilladroid firmware for my HTC Eris for a few weeks now. Beside the phone spontaneously rebooting itself every few hours, its run great, and with built in wifi tethering.<br /><br />Tonight it proved potentially fatal for one young man. While walking home it appeared like I interupted a mugging near <a href="http://hacdc.org/">HacDC</a>. The victim was being kicked by a man who ran away as I got closer. He was laying sprawled out on the concrete, face right on the sidewalk and shirt ripped off. I pulled out my phone, hit the icon for Dialer One (which comes with Vanilladroid), dialed 911, dial key, and... nothing. Dial key again, nothing. Checked, it says 911 in the number box, but the app isn't dialing it. I closed the app and tried again, notta. I can't believe it would refuse to dial 911 so I hard reset the phone and try again, still the dialer just sits there as I press the dial key.<br /><br />I finally had to go on the web, look up the number for Dc Police, and have them transfer me to 911.<br /><br />I take partial blame for not looking into this app and switching it out sooner. Hopefully Google removes it before someone dies while trying to call for help.Anonymoushttp://www.blogger.com/profile/17200801866043907681noreply@blogger.com11