Enabling HTML5 Audio Playback on iOS Safari

I hate wearing headphones. :-P

Building off of a few things we've learned about audio on mobile devices, here's a convenient function for enabling audio on iOS devices. Essentially, as described in our first foray into audio issues on mobile devices, iOS will not play audio without it being first invoked by the user.

The following function takes a DOM element and assigns it an HTML5 audio tag to play when the DOM element is tapped. This hides the first user-invoked "audio.play()" behind another button so that it can now be invoked by code for the rest of the game without explicit user interaction. For example, in our PBS Kids games, we have a "Play!" button on the opening screen, assign our voice-over soundtrack to this button, and once a child taps "Play!", the game is thereafter able to play voice over clips without the need for additional user interaction.

function enableAudio(element, audio, onEnd){
  var callback = false,
      click    = false;

  click = function(e){
    var forceStop = function () {
          audio.removeEventListener('play', forceStop, false);
          audio.pause();
          element.removeEventListener('touchstart', click, false);
          if(onEnd) onEnd();
        },
        progress  = function () {
          audio.removeEventListener('canplaythrough', progress, false);
          if (callback) callback();
        };

    audio.addEventListener('play', forceStop, false);

    audio.addEventListener('canplaythrough', progress, false);

    try { 
      audio.play();
    } catch (e) {
      callback = function () {

        callback = false;
        audio.play();
      };
    }
  };

  element.addEventListener('touchstart', click, false);
}

To use the above, call the function giving it a clickable DOM element and an audio element to invoke as shown below:

enableAudio(element,audio);

As a bonus, if you want to make sure the audio is indeed ready before performing a next action, you can send a function as the third argument; this function will be called once the audio is ready.

HTML5 Audio on Desktop and Mobile

In addition to All the King's Men, we've been spending the past few weeks working on two new games that my boys will love playing. Powered by Ready To Learn, PBS Kids has given us the opportunity to create two math games for children that work across desktop and mobile devices.

This project brings a few new challenges that are unique to this style of game, one of them being a heavy reliance on audio. In our earlier games, audio is a great tool for creating a complimentary ambiance, but in these two games it plays a key role in providing instructions and prompts to maintain game flow.

Since we're targeting desktops and mobile devices, we decided to work our way through the different platforms until we had it functioning appropriately across the board. I'm not an expert on audio file formats, containers, compression, encoding and the like, so the following is simply the series of trials and bits of information I found elsewhere on the web as we worked through the evolving landscape of HTML5 Audio.

This picture has nothing to do with HTML5 Audio.
Except maybe metaphorically.

Ogg and MP3

With our earlier projects I used a two-pronged audio approach that worked well for desktop-based HTML5 games. Using Audacity, I exported all of our audio clips as both MP3 and Ogg files. Ogg covers Firefox and Chrome and the MP3 version is used by the other browsers.

Unfortunately this would not carry over to mobile devices. Testing our games on mobile devices, we found that neither file type would play.

Playing Audio on Mobile Devices

The first issue is that iOS Safari will not play a sound that is not initiated by the user (which also means we cannot preload audio clips :-/) and will only allow one audio clip to be played at a time. We decided to use a single audio clip and employ the "audio sprite" method first heralded by Remy Sharp. He also provides a good solution to getting the audio started by the user. We tied our initial audio play to a simple "Play Game" button at the beginning of the game.

The second issue was the audio format. Ogg is not supported by either browser on Android and iOS, and the MP3 I had created using Audacity fared no better. After searching for a bit, I found that AAC is the format of choice. First I tried exporting via Audacity. This version did not work on any of our test devices. Next I tried exporting from iTunes on my main development computer (a Windows box). This version also did not work. On a whim, I tried exporting from iTunes on a MacBook Pro. For some reason, the mobile devices (both iOS and Android) were able to play this version. I have no idea why exporting using iTunes on Windows produces a different m4a file format than using iTunes on OSX - if you do, please let me know.

I thought using MP4 would release us from MP3, but quickly found that Internet Explorer did not like the mobile-friendly version I created from iTunes, so now we have three audio formats and three different sets of audio files to cover the different browsers: OGG, MP3, and M4A

Playing Audio in Google Chrome

Most browsers handle bouncing around a single audio clip relatively well using the audio sprite method, but Chrome suffered from incredible and inconsistent lag while jumping from one time to another on a single audio clip. This made it essentially unusable when attempting to compose voice-over sentences on the fly: "Can you count" long pause "4" uncomfortably long pause "seahorses?"

Enter "Audia" by Lost Decade Games. I have not had a chance to look into Google Chrome's Web Audio API and wanted a simple drop-in solution to handle the audio analogous to HTML5's audio tag. We already have a lot of code built around the Audio() class, and using Audia allowed us to insert it into our current structure with very little change to our existing code-base. I did throw a try/catch around it so Internet Explorer wouldn't complain about its getters and setters, but the rest was merely checking to see if Audia was supported and replacing "new Audio()" with "new Audia()" if so.

Audio and App Cache

Lastly, loading three different audio streams and storing them locally via the application cache is inefficient and wasteful, considering that the any given browser only has need of a single version of the audio, not all three. Fortunately, handling this client side is not difficult, although admittedly my solution is a bit hacked.

Update (1-21-2013) by Derek Detweiler: The crossed-out method mentioned below worked at one time, but as mentioned above this was a hack. It no longer works in any of the modern browsers we have tested. Our current solution uses a server-side check of the user agent to pass along the correct manifest. An example of this method could look something like this for an .htaccess file: 

AddType text/cache-manifest .manifest

RewriteEngine on

RewriteCond %{HTTP_USER_AGENT} "firefox|opera|chrome" [NC]
RewriteRule ^cache\.manifest$ ogg.manifest [L]

RewriteCond %{HTTP_USER_AGENT} "android|silk|ipod|ipad|iphone" [NC]
RewriteRule ^cache\.manifest$ m4aCombined.manifest [L]

RewriteCond %{HTTP_USER_AGENT} "msie|safari" [NC]
RewriteRule ^cache\.manifest$ mp3.manifest [L]

Since the app cache manifest is not checked until the HTML document is loaded, I inserted the following JavaScript code inline to dynamically assign the appropriate manifest file to the document's html tag. First I assume the browser supports Ogg and the document uses <html manifest="ogg.manifest"> where ogg.manifest lists all the files used, including the appropriate Ogg audio files. The following code replaces "ogg.manifest" with either "mp3.manifest" or "m4a.manifest" depending on what the browser requires:

var myAudio = document.createElement('audio'), isMSIE = /*@cc_on!@*/false;

if ((myAudio.canPlayType) &amp;&amp; !(!!myAudio.canPlayType &amp;&amp; "" != myAudio.canPlayType('audio/ogg; codecs="vorbis"'))){

    if(isMSIE){

        if(document.documentElement.getAttribute("manifest")) document.documentElement.setAttribute("manifest", document.documentElement.getAttribute("manifest").replace('ogg','mp3'));

    } else {

        if(document.documentElement.getAttribute("manifest")) document.documentElement.setAttribute("manifest", document.documentElement.getAttribute("manifest").replace('ogg','m4a'));

    }

}

We're continuing to develop these two titles that my three boys are sure to enjoy, but I hope the above experience proves helpful on your own exploration of HTML5 Audio.

Entanglement Soundtrack by OMNI Audio

We have received a lot of great feedback from everyone about the background music for Entanglement, including many requests for obtaining the soundtrack apart from the game.

First, I want to mention how wonderful it was working with OMNI Audio. Alistair Hirst and his team composed and created the beautiful sound effects and music for Entanglement with a very short turn-around, giving us everything we needed and more to make sure we had high quality audio for everything in the game.

To make the background audio loop less repetitive, Alistair asked us to try an interesting layering technique that I believe worked out phenomenally well. The background track in-game is actually 2 separate layers of audio. The layers are offset by 16 seconds and each layer plays a short 32 second clip randomly chosen from a list of five or six unique sound clips. So, at any given time, you are listening to one of 30 possible combinations, resulting in up to 15 minutes straight of non-repeating original audio.

When I mentioned all the requests we were receiving for a separate downloadable soundtrack, Alistair revisited the background music for us and put it into a linear form. Enjoy!

Grab the original MP3 from CDBaby for $0.99 (a higher percentage gets to us) or on iTunes, Amazon MP3, and several other distributors.