Wednesday, March 28, 2012

HTML5 Audio on Desktop and Mobile

In addition to All the King's Men, we've been spending the past few weeks working on two new games that my boys will love playing. Powered by Ready To Learn, PBS Kids has given us the opportunity to create two math games for children that work across desktop and mobile devices.

This project brings a few new challenges that are unique to this style of game, one of them being a heavy reliance on audio. In our earlier games, audio is a great tool for creating a complimentary ambiance, but in these two games it plays a key role in providing instructions and prompts to maintain game flow.

Since we're targeting desktops and mobile devices, we decided to work our way through the different platforms until we had it functioning appropriately across the board. I'm not an expert on audio file formats, containers, compression, encoding and the like, so the following is simply the series of trials and bits of information I found elsewhere on the web as we worked through the evolving landscape of HTML5 Audio.
This picture has nothing to do with HTML5 Audio.
Except maybe metaphorically.

Ogg and MP3

With our earlier projects I used a two-pronged audio approach that worked well for desktop-based HTML5 games. Using Audacity, I exported all of our audio clips as both MP3 and Ogg files. Ogg covers Firefox and Chrome and the MP3 version is used by the other browsers.

Unfortunately this would not carry over to mobile devices. Testing our games on mobile devices, we found that neither file type would play.

Playing Audio on Mobile Devices

The first issue is that iOS Safari will not play a sound that is not initiated by the user (which also means we cannot preload audio clips :-/) and will only allow one audio clip to be played at a time. We decided to use a single audio clip and employ the "audio sprite" method first heralded by Remy Sharp. He also provides a good solution to getting the audio started by the user. We tied our initial audio play to a simple "Play Game" button at the beginning of the game.

The second issue was the audio format. Ogg is not supported by either browser on Android and iOS, and the MP3 I had created using Audacity fared no better. After searching for a bit, I found that AAC is the format of choice. First I tried exporting via Audacity. This version did not work on any of our test devices. Next I tried exporting from iTunes on my main development computer (a Windows box). This version also did not work. On a whim, I tried exporting from iTunes on a MacBook Pro. For some reason, the mobile devices (both iOS and Android) were able to play this version. I have no idea why exporting using iTunes on Windows produces a different m4a file format than using iTunes on OSX - if you do, please let me know.

I thought using MP4 would release us from MP3, but quickly found that Internet Explorer did not like the mobile-friendly version I created from iTunes, so now we have three audio formats and three different sets of audio files to cover the different browsers: OGG, MP3, and M4A

Playing Audio in Google Chrome

Most browsers handle bouncing around a single audio clip relatively well using the audio sprite method, but Chrome suffered from incredible and inconsistent lag while jumping from one time to another on a single audio clip. This made it essentially unusable when attempting to compose voice-over sentences on the fly: "Can you count" long pause "4" uncomfortably long pause "seahorses?"

Enter "Audia" by Lost Decade Games. I have not had a chance to look into Google Chrome's Web Audio API and wanted a simple drop-in solution to handle the audio analogous to HTML5's audio tag. We already have a lot of code built around the Audio() class, and using Audia allowed us to insert it into our current structure with very little change to our existing code-base. I did throw a try/catch around it so Internet Explorer wouldn't complain about its getters and setters, but the rest was merely checking to see if Audia was supported and replacing "new Audio()" with "new Audia()" if so.

Audio and App Cache

Lastly, loading three different audio streams and storing them locally via the application cache is inefficient and wasteful, considering that the any given browser only has need of a single version of the audio, not all three. Fortunately, handling this client side is not difficult, although admittedly my solution is a bit hacked.
Update (1-21-2013) by Derek Detweiler: The crossed-out method mentioned below worked at one time, but as mentioned above this was a hack. It no longer works in any of the modern browsers we have tested. Our current solution uses a server-side check of the user agent to pass along the correct manifest. An example of this method could look something like this for an .htaccess file: 
AddType text/cache-manifest .manifest

RewriteEngine on

RewriteCond %{HTTP_USER_AGENT} "firefox|opera|chrome" [NC]
RewriteRule ^cache\.manifest$ ogg.manifest [L]

RewriteCond %{HTTP_USER_AGENT} "android|silk|ipod|ipad|iphone" [NC]
RewriteRule ^cache\.manifest$ m4aCombined.manifest [L]

RewriteCond %{HTTP_USER_AGENT} "msie|safari" [NC]
RewriteRule ^cache\.manifest$ mp3.manifest [L]

Since the app cache manifest is not checked until the HTML document is loaded, I inserted the following JavaScript code inline to dynamically assign the appropriate manifest file to the document's html tag. First I assume the browser supports Ogg and the document uses <html manifest="ogg.manifest"> where ogg.manifest lists all the files used, including the appropriate Ogg audio files. The following code replaces "ogg.manifest" with either "mp3.manifest" or "m4a.manifest" depending on what the browser requires:

var myAudio = document.createElement('audio'), isMSIE = /*@cc_on!@*/false;
if ((myAudio.canPlayType) &amp;&amp; !(!!myAudio.canPlayType &amp;&amp; "" != myAudio.canPlayType('audio/ogg; codecs="vorbis"'))){
        if(document.documentElement.getAttribute("manifest")) document.documentElement.setAttribute("manifest", document.documentElement.getAttribute("manifest").replace('ogg','mp3'));
    } else {
        if(document.documentElement.getAttribute("manifest")) document.documentElement.setAttribute("manifest", document.documentElement.getAttribute("manifest").replace('ogg','m4a'));

We're continuing to develop these two titles that my three boys are sure to enjoy, but I hope the above experience proves helpful on your own exploration of HTML5 Audio.

Friday, March 09, 2012

Positioning and Mirroring Images in HTML5... ...and Dragons!

While working on our most recent game, Todd implored me to sharpen some of our sprite rendering. Fuzzy sprites are a result of antialiasing and in our case some images were undergoing this process twice. Rendering pixel to pixel makes a much sharper image, especially if the image has already been anti-aliased once. For All the King's Men, we check the screen size and automatically create all the art at the correct resolution by scaling our sprites from the original size to the appropriate size needed for the current resolution (determined in part by our canvas auto-resizing). Since we pre-render the art at the correct size, and anti-alias it in the process, the last thing we want to do is render it to the visible canvas with a pixel offset, effectively anti-aliasing it twice. Todd accuses me of implementing my awesome "Blurrification Technology" whenever he noticed me doing this. This is how I addressed it (and hopefully stopped him from using made-up words).

At right is a scaled dragon sprite that is still relatively clear, after it's initial anti-aliasing. Also note the sharp color bars I've inserted to highlight the blur. This image is rendered at (0,0), so each pixel on the destination canvas matches a pixel on the source canvas. If our source image was stored in img and the canvas context is ctx, the code might look like this:

x = 0;
y = 0;
ctx.drawImage(img, x, y);

However, in converting from game world coordinates to visual coordinates, rarely are exact integers produced. So, for example, if we offset x by 0.35, the image is anti-aliased for the final rendering, giving the blurred image at right.

x = 0.35;
y = 0;
ctx.drawImage(img, x, y);

Addressing this is simple, since we can simply round to the closest integer coordinate and we once again see our pristine dragon.

x = 0.35;
y = 0;
ctx.drawImage(img, Math.round(x), Math.round(y));

This doesn't hold true when we decide to mirror the image, however. Now we have two items to consider: the original coordinates and where we're flipping the image. Generally, we want to flip the image at it's midpoint, so we could store that in halfImageWidth. Using translate and scale to set things up, we might have:

x = 0.35;
y = 0;
flipAxis = x + halfImageWidth;
ctx.translate(flipAxis, 0);
ctx.scale(-1, 1);
ctx.translate(-flipAxis, 0);
ctx.drawImage(img, Math.round(x), Math.round(y));

Notably, we have once again offset our image to sub-pixels as seen at right. We could round the flipAxis to the closest integer, but images can be flipped between two pixels or in the exact center of a pixel to render a pixel perfect mirror image, so we do this instead:

x = 0.35;
y = 0;
flipAxis = Math.round((x + halfImageWidth) * 2) / 2;
ctx.translate(flipAxis, 0);
ctx.scale(-1, 1);
ctx.translate(-flipAxis, 0);
ctx.drawImage(img, Math.round(x), Math.round(y));

Multiplying the original flip axis (x + halfImageWidth) by two, rounding that number to the closest integer and then dividing by two gives us a flip axis that's between two pixels or directly centered on a pixel. Once again, we have a sharp, pixel-perfect rendering from the source image to the visible canvas.

Thursday, March 08, 2012

Entanglement iOS - What's Going On?

Nothing has changed. :-/
To all of our wonderful Entanglement iOS fans, you may have noticed the promise of a new level "Coming Soon! Free!" that's been on the intro screen for... ...over a year. Well, it's not coming. Not because we don't want to, but because fulfilling that promise is unfortunately out of our control. Entanglement iOS was developed last year by CrateSoft and shortly after its release, the company all but disbanded. It does not appear that we will be able to work out a well-deserved future for Entanglement iOS in spite of the success you have helped make it. It is currently in the state it will remain for the foreseeable future.

If you own the iOS version of Entanglement, we want to make it up to you. We are prepared to give you the online Sakura Grove Expansion Set at no cost to you: simply send a note to us at using the same email address you use to log into, and we will enable the expansion set for your account.

We've learned a few things from this unfortunate turn of events and we are sorry it has affected you this way. Thanks for your understanding and we hope you continue to enjoy Entanglement!