HTML5 Audio on Desktop and Mobile

In addition to All the King's Men, we've been spending the past few weeks working on two new games that my boys will love playing. Powered by Ready To Learn, PBS Kids has given us the opportunity to create two math games for children that work across desktop and mobile devices.

This project brings a few new challenges that are unique to this style of game, one of them being a heavy reliance on audio. In our earlier games, audio is a great tool for creating a complimentary ambiance, but in these two games it plays a key role in providing instructions and prompts to maintain game flow.

Since we're targeting desktops and mobile devices, we decided to work our way through the different platforms until we had it functioning appropriately across the board. I'm not an expert on audio file formats, containers, compression, encoding and the like, so the following is simply the series of trials and bits of information I found elsewhere on the web as we worked through the evolving landscape of HTML5 Audio.

This picture has nothing to do with HTML5 Audio.
Except maybe metaphorically.

Ogg and MP3

With our earlier projects I used a two-pronged audio approach that worked well for desktop-based HTML5 games. Using Audacity, I exported all of our audio clips as both MP3 and Ogg files. Ogg covers Firefox and Chrome and the MP3 version is used by the other browsers.

Unfortunately this would not carry over to mobile devices. Testing our games on mobile devices, we found that neither file type would play.

Playing Audio on Mobile Devices

The first issue is that iOS Safari will not play a sound that is not initiated by the user (which also means we cannot preload audio clips :-/) and will only allow one audio clip to be played at a time. We decided to use a single audio clip and employ the "audio sprite" method first heralded by Remy Sharp. He also provides a good solution to getting the audio started by the user. We tied our initial audio play to a simple "Play Game" button at the beginning of the game.

The second issue was the audio format. Ogg is not supported by either browser on Android and iOS, and the MP3 I had created using Audacity fared no better. After searching for a bit, I found that AAC is the format of choice. First I tried exporting via Audacity. This version did not work on any of our test devices. Next I tried exporting from iTunes on my main development computer (a Windows box). This version also did not work. On a whim, I tried exporting from iTunes on a MacBook Pro. For some reason, the mobile devices (both iOS and Android) were able to play this version. I have no idea why exporting using iTunes on Windows produces a different m4a file format than using iTunes on OSX - if you do, please let me know.

I thought using MP4 would release us from MP3, but quickly found that Internet Explorer did not like the mobile-friendly version I created from iTunes, so now we have three audio formats and three different sets of audio files to cover the different browsers: OGG, MP3, and M4A

Playing Audio in Google Chrome

Most browsers handle bouncing around a single audio clip relatively well using the audio sprite method, but Chrome suffered from incredible and inconsistent lag while jumping from one time to another on a single audio clip. This made it essentially unusable when attempting to compose voice-over sentences on the fly: "Can you count" long pause "4" uncomfortably long pause "seahorses?"

Enter "Audia" by Lost Decade Games. I have not had a chance to look into Google Chrome's Web Audio API and wanted a simple drop-in solution to handle the audio analogous to HTML5's audio tag. We already have a lot of code built around the Audio() class, and using Audia allowed us to insert it into our current structure with very little change to our existing code-base. I did throw a try/catch around it so Internet Explorer wouldn't complain about its getters and setters, but the rest was merely checking to see if Audia was supported and replacing "new Audio()" with "new Audia()" if so.

Audio and App Cache

Lastly, loading three different audio streams and storing them locally via the application cache is inefficient and wasteful, considering that the any given browser only has need of a single version of the audio, not all three. Fortunately, handling this client side is not difficult, although admittedly my solution is a bit hacked.

Update (1-21-2013) by Derek Detweiler: The crossed-out method mentioned below worked at one time, but as mentioned above this was a hack. It no longer works in any of the modern browsers we have tested. Our current solution uses a server-side check of the user agent to pass along the correct manifest. An example of this method could look something like this for an .htaccess file: 

AddType text/cache-manifest .manifest

RewriteEngine on

RewriteCond %{HTTP_USER_AGENT} "firefox|opera|chrome" [NC]
RewriteRule ^cache\.manifest$ ogg.manifest [L]

RewriteCond %{HTTP_USER_AGENT} "android|silk|ipod|ipad|iphone" [NC]
RewriteRule ^cache\.manifest$ m4aCombined.manifest [L]

RewriteCond %{HTTP_USER_AGENT} "msie|safari" [NC]
RewriteRule ^cache\.manifest$ mp3.manifest [L]

Since the app cache manifest is not checked until the HTML document is loaded, I inserted the following JavaScript code inline to dynamically assign the appropriate manifest file to the document's html tag. First I assume the browser supports Ogg and the document uses <html manifest="ogg.manifest"> where ogg.manifest lists all the files used, including the appropriate Ogg audio files. The following code replaces "ogg.manifest" with either "mp3.manifest" or "m4a.manifest" depending on what the browser requires:

var myAudio = document.createElement('audio'), isMSIE = /*@cc_on!@*/false;

if ((myAudio.canPlayType) &amp;&amp; !(!!myAudio.canPlayType &amp;&amp; "" != myAudio.canPlayType('audio/ogg; codecs="vorbis"'))){

    if(isMSIE){

        if(document.documentElement.getAttribute("manifest")) document.documentElement.setAttribute("manifest", document.documentElement.getAttribute("manifest").replace('ogg','mp3'));

    } else {

        if(document.documentElement.getAttribute("manifest")) document.documentElement.setAttribute("manifest", document.documentElement.getAttribute("manifest").replace('ogg','m4a'));

    }

}

We're continuing to develop these two titles that my three boys are sure to enjoy, but I hope the above experience proves helpful on your own exploration of HTML5 Audio.