HTML5 Audio on Desktop and Mobile

In addition to All the King's Men, we've been spending the past few weeks working on two new games that my boys will love playing. Powered by Ready To Learn, PBS Kids has given us the opportunity to create two math games for children that work across desktop and mobile devices.

This project brings a few new challenges that are unique to this style of game, one of them being a heavy reliance on audio. In our earlier games, audio is a great tool for creating a complimentary ambiance, but in these two games it plays a key role in providing instructions and prompts to maintain game flow.

Since we're targeting desktops and mobile devices, we decided to work our way through the different platforms until we had it functioning appropriately across the board. I'm not an expert on audio file formats, containers, compression, encoding and the like, so the following is simply the series of trials and bits of information I found elsewhere on the web as we worked through the evolving landscape of HTML5 Audio.

This picture has nothing to do with HTML5 Audio.
Except maybe metaphorically.

Ogg and MP3

With our earlier projects I used a two-pronged audio approach that worked well for desktop-based HTML5 games. Using Audacity, I exported all of our audio clips as both MP3 and Ogg files. Ogg covers Firefox and Chrome and the MP3 version is used by the other browsers.

Unfortunately this would not carry over to mobile devices. Testing our games on mobile devices, we found that neither file type would play.

Playing Audio on Mobile Devices

The first issue is that iOS Safari will not play a sound that is not initiated by the user (which also means we cannot preload audio clips :-/) and will only allow one audio clip to be played at a time. We decided to use a single audio clip and employ the "audio sprite" method first heralded by Remy Sharp. He also provides a good solution to getting the audio started by the user. We tied our initial audio play to a simple "Play Game" button at the beginning of the game.

The second issue was the audio format. Ogg is not supported by either browser on Android and iOS, and the MP3 I had created using Audacity fared no better. After searching for a bit, I found that AAC is the format of choice. First I tried exporting via Audacity. This version did not work on any of our test devices. Next I tried exporting from iTunes on my main development computer (a Windows box). This version also did not work. On a whim, I tried exporting from iTunes on a MacBook Pro. For some reason, the mobile devices (both iOS and Android) were able to play this version. I have no idea why exporting using iTunes on Windows produces a different m4a file format than using iTunes on OSX - if you do, please let me know.

I thought using MP4 would release us from MP3, but quickly found that Internet Explorer did not like the mobile-friendly version I created from iTunes, so now we have three audio formats and three different sets of audio files to cover the different browsers: OGG, MP3, and M4A

Playing Audio in Google Chrome

Most browsers handle bouncing around a single audio clip relatively well using the audio sprite method, but Chrome suffered from incredible and inconsistent lag while jumping from one time to another on a single audio clip. This made it essentially unusable when attempting to compose voice-over sentences on the fly: "Can you count" long pause "4" uncomfortably long pause "seahorses?"

Enter "Audia" by Lost Decade Games. I have not had a chance to look into Google Chrome's Web Audio API and wanted a simple drop-in solution to handle the audio analogous to HTML5's audio tag. We already have a lot of code built around the Audio() class, and using Audia allowed us to insert it into our current structure with very little change to our existing code-base. I did throw a try/catch around it so Internet Explorer wouldn't complain about its getters and setters, but the rest was merely checking to see if Audia was supported and replacing "new Audio()" with "new Audia()" if so.

Audio and App Cache

Lastly, loading three different audio streams and storing them locally via the application cache is inefficient and wasteful, considering that the any given browser only has need of a single version of the audio, not all three. Fortunately, handling this client side is not difficult, although admittedly my solution is a bit hacked.

Since the app cache manifest is not checked until the HTML document is loaded, I inserted the following JavaScript code inline to dynamically assign the appropriate manifest file to the document's html tag. First I assume the browser supports Ogg and the document uses <html manifest="ogg.manifest"> where ogg.manifest lists all the files used, including the appropriate Ogg audio files. The following code replaces "ogg.manifest" with either "mp3.manifest" or "m4a.manifest" depending on what the browser requires:

var myAudio = document.createElement('audio'), isMSIE = /*@cc_on!@*/false;

if ((myAudio.canPlayType) &amp;&amp; !(!!myAudio.canPlayType &amp;&amp; "" != myAudio.canPlayType('audio/ogg; codecs="vorbis"'))){

    if(isMSIE){

        if(document.documentElement.getAttribute("manifest")) document.documentElement.setAttribute("manifest", document.documentElement.getAttribute("manifest").replace('ogg','mp3'));

    } else {

        if(document.documentElement.getAttribute("manifest")) document.documentElement.setAttribute("manifest", document.documentElement.getAttribute("manifest").replace('ogg','m4a'));

    }

}

We're continuing to develop these two titles that my three boys are sure to enjoy, but I hope the above experience proves helpful on your own exploration of HTML5 Audio.

Positioning and Mirroring Images in HTML5... ...and Dragons!

While working on our most recent game, Todd implored me to sharpen some of our sprite rendering. Fuzzy sprites are a result of antialiasing and in our case some images were undergoing this process twice. Rendering pixel to pixel makes a much sharper image, especially if the image has already been anti-aliased once. For All the King's Men, we check the screen size and automatically create all the art at the correct resolution by scaling our sprites from the original size to the appropriate size needed for the current resolution (determined in part by our canvas auto-resizing). Since we pre-render the art at the correct size, and anti-alias it in the process, the last thing we want to do is render it to the visible canvas with a pixel offset, effectively anti-aliasing it twice. Todd accuses me of implementing my awesome "Blurrification Technology" whenever he noticed me doing this. This is how I addressed it (and hopefully stopped him from using made-up words).

At right is a scaled dragon sprite that is still relatively clear, after it's initial anti-aliasing. Also note the sharp color bars I've inserted to highlight the blur. This image is rendered at (0,0), so each pixel on the destination canvas matches a pixel on the source canvas. If our source image was stored in img and the canvas context is ctx, the code might look like this:

x = 0;
y = 0;
ctx.drawImage(img, x, y);

However, in converting from game world coordinates to visual coordinates, rarely are exact integers produced. So, for example, if we offset x by 0.35, the image is anti-aliased for the final rendering, giving the blurred image at right.

x = 0.35;
y = 0;
ctx.drawImage(img, x, y);

Addressing this is simple, since we can simply round to the closest integer coordinate and we once again see our pristine dragon.

x = 0.35;
y = 0;
ctx.drawImage(img, Math.round(x), Math.round(y));

This doesn't hold true when we decide to mirror the image, however. Now we have two items to consider: the original coordinates and where we're flipping the image. Generally, we want to flip the image at it's midpoint, so we could store that in halfImageWidth. Using translate and scale to set things up, we might have:

x = 0.35;
y = 0;
flipAxis = x + halfImageWidth;
ctx.translate(flipAxis, 0);
ctx.scale(-1, 1);
ctx.translate(-flipAxis, 0);
ctx.drawImage(img, Math.round(x), Math.round(y));

Notably, we have once again offset our image to sub-pixels as seen at right. We could round the flipAxis to the closest integer, but images can be flipped between two pixels or in the exact center of a pixel to render a pixel perfect mirror image, so we do this instead:

x = 0.35;
y = 0;
flipAxis = Math.round((x + halfImageWidth) * 2) / 2;
ctx.translate(flipAxis, 0);
ctx.scale(-1, 1);
ctx.translate(-flipAxis, 0);
ctx.drawImage(img, Math.round(x), Math.round(y));

Multiplying the original flip axis (x + halfImageWidth) by two, rounding that number to the closest integer and then dividing by two gives us a flip axis that's between two pixels or directly centered on a pixel. Once again, we have a sharp, pixel-perfect rendering from the source image to the visible canvas.

Entanglement iOS - What's Going On?

Nothing has changed. :-/

To all of our wonderful Entanglement iOS fans, you may have noticed the promise of a new level "Coming Soon! Free!" that's been on the intro screen for... ...over a year. Well, it's not coming. Not because we don't want to, but because fulfilling that promise is unfortunately out of our control. Entanglement iOS was developed last year by CrateSoft and shortly after its release, the company all but disbanded. It does not appear that we will be able to work out a well-deserved future for Entanglement iOS in spite of the success you have helped make it. It is currently in the state it will remain for the foreseeable future.

If you own the iOS version of Entanglement, we want to make it up to you. We are prepared to give you the online Sakura Grove Expansion Set at no cost to you: simply send a note to us at feedback@gopherwoodstudios.com using the same email address you use to log into http://entanglement.gopherwoodstudios.com, and we will enable the expansion set for your account.

We've learned a few things from this unfortunate turn of events and we are sorry it has affected you this way. Thanks for your understanding and we hope you continue to enjoy Entanglement!

All the King's Men

Todd and I have been busy working on a new game called "All the King's Men". (Yes, we finally named it if you've been following us on Facebook or Google+) It's not ready to be officially unveiled for play just yet, but here's a short run-through from our initial prototype.

Originally the game was a simple puzzle grid where a player could choose between several pins to push off and collect gems of the same color. The end-goal was a bit vague and players scored by counting how many gems they pushed off. In the prototype photo below, the movement pins are represented by Lego bricks, the gems are Risk pieces, and the game board is the only part I can find of my marble chess set. Playing this version with friends quickly highlighted a problem: game-play felt like a sequence of solitary moves. One player's move didn't really affect the next player's options, so there was no multiplayer strategy or fun to the game.

Prototype comprised of Risk and Lego

These look nicer than Risk pieces

After trying several variations, Todd decided it might be better to have each player play on their own board and their actions could effect the other players. This turned out to be a lot more fun. We had already found that the puzzle grid game-play worked best with three types of gems, so we turned that aspect into three ways to affect your opponents, scaled the puzzle grid from 8x8 to 5x5, and removed all but one movement pin. Since the primary interaction between players happens outside the grid, it served well to simplify that portion of the game.

I love playing Castle Wars with my wife, so I thought a similar theme might work well for an overall goal and the three gems mechanic. We ultimately settled on armies, castles, and flying boulders, with the overall goal being for players to complete their castle or destroy all of their opponent's castles. To complete the setting, the player assumes the role of a medieval king standing at his war-room table, using the puzzle grid to gather his kingdom's resources and imprinting his seal on the resulting commands to send his valiant armies to battle.

Getting ready to unleash Dwayne, the vicious dragon, on the battlefield.

I'm looking forward to completing this game and making it available for play so someone with more skills than I can beat Todd.

Loading HTML5 Game Assets

Before diving into HTML5 game development, I built a few conventional web sites (like this one - nice snapshots of the Blue Ridge). With conventional web design, page assets like images are loaded once the page loads, slowly filling out the blank screen with colorful pictures. This works well for web pages, since you can immediately begin reading the content and hardly notice the small delay as the images are painted in.

However, the wait for in-game assets can be especially long when downloading 10MB worth of images and sounds for a game like Entanglement. With Entanglement, the first issue I came across was that I couldn't draw a beautiful wooden tile to the gameboard canvas if the wooden tile image hadn't been downloaded yet.

My first attempt to solve this problem was to throw a JavaScript try/catch block around the drawImage calls so they wouldn't break the game if the image wasn't available yet. This resulted in the game starting immediately, but the gameboard slowly appeared as the images were downloaded. This is fine behavior for a web page but seemed a bit ugly for a web game.

My second attempt was a bit better: I included all the images in the html document as invisible <img> tags. This ensures that the images are indeed loaded before the game began but had the unfortunate side effect that JavaScript processing didn't start until the entire page was downloaded (including all the referenced images). This caused the long wait at a "Loading..." screen that couldn't be dynamically updated with a progress bar since JavaScript wasn't running yet.

It's doing something but we're not sure what nor how long it's going to take.

Enter the Asset Loader! This tool allows larger assets to be loaded after the page loads and the JavaScript processing begins. We're in the prototype stage of our next game, and I took this opportunity to build a proper asset loader. Seth Ladd wrote a wonderful tutorial on the basic steps to set up a simple asset manager. Using his insight and including a bit more code to handle audio assets, we created an asset loader that loads assets as well as calls two functions we can supply to track progress: what to do when a single asset is finished loading and what to do when all the assets are finished loading.

function loadAssets(assetList, onAssetLoaded, onAllAssetsLoaded){...}

The next piece we want is a nice progress bar. The <progress> HTML5 element has two properties we care about for keeping tabs on our loaded assets: max and value. Using these two properties we can create a very simple progress bar class to hand off to the asset loader that might look something like this:

function progressBar(parent){
 this.element = document.createElement('progress');
 parent.appendChild(this.element);
 
 // initialize the progress bar at zero percent
 this.element.setAttribute('value', 0);
 this.element.setAttribute('max', 1000);
 
 // for this example, progress will be some value [0,1]
 this.update = function (progress) {
  this.element.setAttribute('value', progress * 1000);
 };
};

We can then set up our asset loader to update the progress bar each time an asset is finished loading, by feeding the progressBar's update function the current state of loaded assets. In our asset loader, the "ratio" value shown below is simply the number of assets loaded divided by the total number of assets.

var assetProgress = new progressBar(document.getElementById('loading-screen'));
loadAssets(assetList, function(ratio){assetProgress.update(ratio);}, startGame);

We've already implemented the progress bar in the latest builds of both Thwack!! and Entanglement if you care to see it in action!

Google In-App Payments: Convenient for buyers and developers

Todd and I are always looking for good ways to monetize our web games so we can continue doing what we love: making more games! With Entanglement, we created extended content that players can purchase if they really enjoy the game, and with our latest creation, Thwack!!, we created a reward system using tokens that can be won by playing the game or purchased for a few cents.

Back in January when we first released our Sakura Grove Expansion for Entanglement, we implemented an email and key system for purchasing: to access the expansion set for the first time, a player would make the purchase, wait for the confirmation email to arrive, and follow the instructions to get back into Entanglement with the key to unlock their purchase. Not only was the process indirect and inconvenient for purchasing a digital good, but it also took us quite a bit of time to develop and debug.

In developing our latest game, Thwack!!, we decided to implement Google In-App Payments. In-app payments are ideal for selling digital goods and the small 5% transaction fee is wonderful for the small transactions common to games.

Once our game's store interface was set up, I was pleasantly surprised to find integrating in-app payments very quick and straight-forward. In fact, I first began reading the documentation early one morning, and by following the very concise step-by-step instructions I had in-app payments implemented in Thwack!! by lunch time!

We were quite pleased with both the appearance and implementation for Thwack!! so converting Entanglement’s purchase process to Google In-App Payments was a natural next step. However, setting up in-app payments for two separate games on one seller account highlighted something I would need to change with my initially simplistic Thwack!! integration, where I had implemented the entire process, including the postback URL, through the web game itself.

Since I was adding a second game on the same seller account, I had to separate the postback process from Thwack!!. After a bit of restructuring, and less than a day’s work, I had In-App Payments running in both Thwack!! and Entanglement with a separate service that tracks orders apart from the games as shown below.

We now have Google In-App Payments running in both games in less than two days’ worth of programming with the additional benefit of a separate central location to track orders.

We found that Google In-App Payments is not only convenient for our game players by allowing them to remain in the game experience while making their purchase, it is also convenient for us as game developers by making the integration process quick and easy to implement.

HTML5 AppCache and Keeping Clients Updated

If you've played Entanglement or Thwack!!, you may notice that it takes a while to load the first time. Fortunately, if you've installed Entanglement as a web app, or played Thwack!! more than once, you may have also noticed that subsequent visits require much less time to load. This is thanks to the HTML5 application cache.

The application cache (or "appcache" for short) allows us to store the game and all of its audio and graphics files locally on your computer (here's a nice tutorial on implementing an appcache). Next time you play the game, the application cache loads the files from your computer rather than pull them from the Internet. The beauty of this is two-fold. First, the game can start loading quickly since it doesn't need to check for anything online first. Second, the game can load without a connection to the Internet at all!

One drawback to this is that if someone plays the game and then returns some time later, the next time they play the game their browser loads a version of the game that's potentially weeks or even months old. If there's an updated version, appcache will load the files in the background to prepare for the next time the game is loaded, but the version the player is currently experiencing is the old version until they reload the page. This isn't too much of a problem with Entanglement where changes from one version of Entanglement to another do not negatively effect the experience.

What a player sees if running an old version

However, with the online multiplayer aspect of Thwack!! it's a bit more disruptive. With Thwack!!, each client independently calculates the results of a given turn, and those results have to match up with each other. For instance (not connected to reality... ...okay maybe a little), if Todd decides on a whim to change all of the discs' sizes, weights, and damping, the new version he deploys will calculate a different result for a turn than an older version stored in appcache. This effectively breaks the experience for not only the player whose browser loaded an old version of Thwack!! but also all of the players connected to that particular game.

The fix is fortunately quite simple. When a new version has loaded in the background, we know that we're not running the latest version. We can check this by implementing the following event listener:

var newVersionWaiting = false;
var updateReady = function()
{
    newVersionWaiting = true;
};

window.applicationCache.addEventListener('updateready', updateReady, false);

Now that we're aware of a new version waiting on the sideline, we can notify the player that they're running an outdated version before we let them in on a multiplayer game. Additionally, we attach a page refresh call to the message dialog's "OK" button to make it even easier for them to get the most updated version. Here's a snippet from Thwack!! where "messageBox" creates the dialog you see in the screenshot above. The first parameter is the message and second parameter, a function, is executed by clicking "OK":

if(newVersionWaiting)
{
    messageBox(RELOAD_THWACK, function(){window.location.reload();});
}