This project brings a few new challenges that are unique to this style of game, one of them being a heavy reliance on audio. In our earlier games, audio is a great tool for creating a complimentary ambiance, but in these two games it plays a key role in providing instructions and prompts to maintain game flow.
Since we're targeting desktops and mobile devices, we decided to work our way through the different platforms until we had it functioning appropriately across the board. I'm not an expert on audio file formats, containers, compression, encoding and the like, so the following is simply the series of trials and bits of information I found elsewhere on the web as we worked through the evolving landscape of HTML5 Audio.
This picture has nothing to do with HTML5 Audio. Except maybe metaphorically. |
Ogg and MP3
With our earlier projects I used a two-pronged audio approach that worked well for desktop-based HTML5 games. Using Audacity, I exported all of our audio clips as both MP3 and Ogg files. Ogg covers Firefox and Chrome and the MP3 version is used by the other browsers.
Unfortunately this would not carry over to mobile devices. Testing our games on mobile devices, we found that neither file type would play.
Playing Audio on Mobile Devices
The first issue is that iOS Safari will not play a sound that is not initiated by the user (which also means we cannot preload audio clips :-/) and will only allow one audio clip to be played at a time. We decided to use a single audio clip and employ the "audio sprite" method first heralded by Remy Sharp. He also provides a good solution to getting the audio started by the user. We tied our initial audio play to a simple "Play Game" button at the beginning of the game.
The second issue was the audio format. Ogg is not supported by either browser on Android and iOS, and the MP3 I had created using Audacity fared no better. After searching for a bit, I found that AAC is the format of choice. First I tried exporting via Audacity. This version did not work on any of our test devices. Next I tried exporting from iTunes on my main development computer (a Windows box). This version also did not work. On a whim, I tried exporting from iTunes on a MacBook Pro. For some reason, the mobile devices (both iOS and Android) were able to play this version. I have no idea why exporting using iTunes on Windows produces a different m4a file format than using iTunes on OSX - if you do, please let me know.
I thought using MP4 would release us from MP3, but quickly found that Internet Explorer did not like the mobile-friendly version I created from iTunes, so now we have three audio formats and three different sets of audio files to cover the different browsers: OGG, MP3, and M4A
Playing Audio in Google Chrome
Most browsers handle bouncing around a single audio clip relatively well using the audio sprite method, but Chrome suffered from incredible and inconsistent lag while jumping from one time to another on a single audio clip. This made it essentially unusable when attempting to compose voice-over sentences on the fly: "Can you count" long pause "4" uncomfortably long pause "seahorses?"
Enter "Audia" by Lost Decade Games. I have not had a chance to look into Google Chrome's Web Audio API and wanted a simple drop-in solution to handle the audio analogous to HTML5's audio tag. We already have a lot of code built around the Audio() class, and using Audia allowed us to insert it into our current structure with very little change to our existing code-base. I did throw a try/catch around it so Internet Explorer wouldn't complain about its getters and setters, but the rest was merely checking to see if Audia was supported and replacing "new Audio()" with "new Audia()" if so.
Audio and App Cache
Lastly, loading three different audio streams and storing them locally via the application cache is inefficient and wasteful, considering that the any given browser only has need of a single version of the audio, not all three. Fortunately, handling this client side is not difficult, although admittedly my solution is a bit hacked.
Update (1-21-2013) by Derek Detweiler: The crossed-out method mentioned below worked at one time, but as mentioned above this was a hack. It no longer works in any of the modern browsers we have tested. Our current solution uses a server-side check of the user agent to pass along the correct manifest. An example of this method could look something like this for an .htaccess file:
AddType text/cache-manifest .manifest RewriteEngine on RewriteCond %{HTTP_USER_AGENT} "firefox|opera|chrome" [NC] RewriteRule ^cache\.manifest$ ogg.manifest [L] RewriteCond %{HTTP_USER_AGENT} "android|silk|ipod|ipad|iphone" [NC] RewriteRule ^cache\.manifest$ m4aCombined.manifest [L] RewriteCond %{HTTP_USER_AGENT} "msie|safari" [NC] RewriteRule ^cache\.manifest$ mp3.manifest [L]
We're continuing to develop these two titles that my three boys are sure to enjoy, but I hope the above experience proves helpful on your own exploration of HTML5 Audio.