Raw results

iPhone 4S (iOS5)

iPhone 4 (iOS5)

iPhone 3GS (iOS5)

Bonus: FishIETank (10 fish) (Canvas test)

One thing to note here: the 3GS has a bit of an advantage because of a non-retina screen. However, it’s still outperformed easily by the iPhone 4 and 4S.

Related

JavaScript SunSpider benchmark: iOS 4.3 vs iOS 4.0
BlackBerry Torch SunSpider results (JavaScript benchmark)
iPhone 4 SunSpider test results (22% faster than iPhone 3GS)
JavaScript SunSpider test: iOS 3.1.3 versus iOS 4 GM
JavaScript SunSpider: HTC Evo versus HTC Incredible versus Nexus One

Earlier this year I gave a talk (slides) outlining the latest and greatest in mobile performance, including a bit of my own unscientific research into carrier latency and bandwidth thanks to boomerang.js.

I realized that interest in mobile performance has exploded recently, especially with Steve Souders announcing his focus on mobile, and I thought it was time for an update, this time in blog form. Also, my old slides have been somewhat embarrassing. For some strange reason, at the time I wanted to give S5 a try – that outdated, ancient, not-performant slideshow framework. The result is a slideshow on performance that loads slowly… doh! (incidentally, I recommend deck.js as an alternative).

In any case, it was time for a roundup of mobile performance best practices, in blog form. I’m not sure if it’s properly called a manifesto, but it is what it is! Onward!

For fun: Latency and bandwidth tests

Before we start.. just a little fun data! It’s always been a pleasure to fiddle around with boomerang.js and compare results. With my latest trip to Japan, I had the opportunity to run Boomerang on Japan’s e-mobile 3G network in the remotest of places, up in Hakuba/Nagano in the Japanese alps.

When I got back, I ran the same tests from downtown San Francisco, which was closer to my California-based test server, and should’ve been faster, right? Theoretically…

Boomerang Tests (running on a California-hosted server)

Yes, e-mobile didn’t have the greatest ping, but it handily beat ATT, even though it was going across the entire Pacific Ocean! Note that it came out the best in the bandwidth tests however…

(Note: it’s unknown if or how Steve Souders’s latest research affects these findings)

There’s a lot of factors that could’ve been involved, so look to something like OpenSignalMaps for more data.

Ok, onto the tips!

Page Organization

For feature phones that have little to no caching, aggressively combine requests (deliver HTML/CSS/JS all in one package). For smart phones, take advantage of caching by mirroring desktop frontend best practices: separate HTML, CSS, and JS so they can be cached (per-session and across sessions).

There’s old research about extremely small cache sizes on iOS in particular, but this research has been followed up on by more recent research by Ryan Grove (Yahoo!) and Steve Souders (Google), which shows that we shouldn’t be so paranoid, since caching is pretty decent across all the major mobile browsers.

Of particular interest is a browser’s capability to cache files in a current session, browsing from page-to-page, which is what most users will end up doing (caching of the page across sessions is another matter). What is the maximum file size a browser will cache during a session? The results (via Browserscope) end up being encouraging:

Avoid redirects (foo.com -> m.foo.com)

If possible, perform the redirection behind the scenes on the server, which should be transparent to the user. When a user performs a Google search and clicks on your page, they’re already getting redirected once by Google (check it yourself). Your own redirects are adding a second redirect where there need not be one.

Optimize Images

Deliver appropriately-sized images to devices. The philosophy of responsive design makes it easy to simply downscale images to fit the screen, but avoid this where possible, as this means wasted bandwidth.

You can optimize images through CSS media queries or in JavaScript (see below). Though you want to reduce your dependence on cookies, it may be a good idea to store these width/height values into a cookie (or localStorage if you are fetching images in nontraditional ways) so the values can be read by the server, which can deliver appropriately-sized images. This technique has been implemented by Filament Group, so you should probably read about their experiences before trying to roll your own.

Also note that where applicable, at the expense of performance you may want to serve higher resolution images for better screens, such as for Retina displays (which can be detected with JavaScript or CSS).

JavaScript examples

window.innerHeight;       // max height actually available
window.innerWidth;        // max width actually available
window.devicePixelRatio;  // pixel density (standard is 1, high resolution is generally > 1)

(note that screen.width and screen.height are also available to tell you the dimensions of the entire screen, but this isn’t all available due to the space taken by the OS and browser chrome)

Media query examples

@media only screen and (max-width: 480px) {
  /* small screen styles */
}

@media only screen and (min-width: 481px) {
  /* large screen styles */
}

@media (-webkit-min-device-pixel-ratio: 1.5),
       (-o-min-device-pixel-ratio: 3/2),
       (min--moz-device-pixel-ratio: 1.5),
       (min-device-pixel-ratio: 1.5) {
  /* high resolution styles */
}

navigator.connection (Android only)

Use navigator.connection if it’s available to serve different assets based on connection speed (3G vs WIFI, for instance).

Here’s the contents of the navigator.connection object (with a phone running on a 3G connection):

{
  type: 4,
  UNKNOWN: 0,
  ETHERNET: 1,
  WIFI: 2,
  CELL_2G: 3,
  CELL_3G: 4
}

And an example of how you could write code for each type:

(function(){  // sandbox our code
  if(!navigator.connection) navigator.connection = {type:0, UNKNOWN: 0};  // polyfill
  var connection = navigator.connection;

  if(connection.type === connection.WIFI || connection.type === connection.ETHERNET) {
    // high bandwidth
  } else {
    // normal bandwidth
  }
})();

Here’s how we might be able to use Modernizr to help us out:

(function(){  // sandbox our code
  if(!navigator.connection) navigator.connection = {type:0, UNKNOWN: 0};  // polyfill
  var connection = navigator.connection;

  // add a custom test to Modernizr
  Modernizr.addTest('highbandwidth', function(){
    return connection.type === connection.WIFI || connection.type === connection.ETHERNET;
  });

})();

We can now target special high-bandwidth assets with CSS:

.highbandwidth .logo {background-image:url('logo-high.jpg');}
.no-highbandwidth .logo {background-image:url('logo.jpg');}

base64 encode small UI images

You can base64 encode binary image data in HTML and CSS:

HTML

<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA
AAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO
9TXL0Y4OHwAAAABJRU5ErkJggg==" alt="Red dot">

CSS

.dot {
  background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA
AAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO
9TXL0Y4OHwAAAABJRU5ErkJggg==');
}

There seems to be the mistaken belief floating around that base64 images aren’t cacheable. However, because you can embed them in your CSS, base64 encoded images will be cached along with the rest of your CSS.

You’ll probably just want to limit this to small UI icons however. I wrote a quick post on the drawbacks, and also a comparison with sprites.

If you’re using Compass/SASS, base64 encoding images is pretty trivial and easy to maintain with the inline-image helper.

Unicode and Emoji

☆ (HTML entity: #9734)
(HTML entity: #x1f468)

Before even considering base64 images, take advantage of icons that are already made for you: unicode characters and emoji (where supported). Keep in mind that these will look different across browsers, so this might not be an option for everyone.

Because of inconsistent implementations of Emoji character codes between Asian carriers, representatives from Apple and Google made a proposal to add Emoji to the Unicode standard. Unfortunately it seems that only iPhone supports them for now. Here’s a handy chart of all the Unicode emoji characters available.

Take advantage of CSS3

CSS3 offers many replacements for things we needed images for previously. RGBA values replace the need for a semitransparent image for overlays and such. Likewise, border-radius, box shadow, linear-gradients, radial-gradients all reduce the need for images.

However, do be aware that though phones support these new features, it doesn’t mean they’re necessarily ready for primetime. Something I’ve encountered recently to remind me of this fact is severe color banding issues when using a CSS radial gradient. The solution was to fake the radial gradient using two linear gradients, or to (unfortunately) use an old-fashioned image.

Avoid using cookies

Cookies get thrown into every request on a per-domain basis, so limit your usage of them. Use localStorage/sessionStorage instead, where possible.

App cache

Take advantage of the HTML5 app cache, though it’s an unwieldy beast to tame. This will mean you will need to make your site work offline, which might be tricky.

Assets that are traditionally cached get rechecked on page refresh (the server sends back a 304 response if your cache is still up-to-date). Whereas with the app cache, only one file (your manifest) is rechecked on page load.

Deferred JavaScript execution

We know that deferring downloading of scripts is advantageous (by placing scripts at the bottom of a page or by using the async attribute), but deferred execution of JavaScript is even more important. Sure, we can use the defer attribute, but that’s only really relevant for when the page is loading (defer lets the browser know that the UI doesn’t depend on the JavaScript, so it can be safely deferred).

But what about JavaScript that runs after the page loads, such as XHR and JSONP requests? What does this mean for the user? It means that the UI freezes up unexpectedly when JavaScript is being downloaded and executed in the background. But this doesn’t mean that you want to completely avoid background downloading of JavaScript. The Gmail mobile team came up with a clever solution: by commenting all of their code and dynamically eval’ing it when needed, they split up the JavaScript downloading from its execution.

Perceived performance

Do whatever you have to do to let the user know that the UI is still responding. This sometimes means faking that something is happening. Communication is key! If they clicked on a button, give some indication that they clicked on it. If they clicked on something that requires a request to the network, show a spinner right away, even if it means you haven’t even sent out a request. The user doesn’t need to know the nitty-gritty details – they just want to know that their intent was communicated.

Onclick delay

On several major mobile operating systems there’s a several hundred millisecond delay on the onclick event. This is because of the double-tap-to-zoom functionality. When a user first taps on the screen, there’s a hard-coded delay that waits for the second tap. If there’s no second tap, the onclick event is then fired. Unfortunately the only way to get around the delay is to tap into touch events instead, which is a bit more complicated than it appears on first glance.

For an overview of some of the challenges of implementing the workaround, see this article by Google’s Ryan Fioravanti.

Take advantage of hardware acceleration

Use hardware-accelerated CSS transforms where possible (translate3d, translateZ, rotate3d, and scale3d). An element that is hardware accelerated is turned into a graphic, which is perfect for the GPU to manipulate, taking away the burden from the CPU.

However, the GPU isn’t all-powerful, so don’t try to apply hardware accelerations to everything. Also, these elements still need to be refreshed periodically, and it turns out that you can make some good optimizations here. First, you’ll want to debug the composited layers on your desktop browser:

Debugging hardware acceleration

Chrome

1. Type the following in the address bar: about:flags
2. “Composited render layer borders” -> Enable

Safari

1. Open a terminal
2. $ defaults write com.apple.Safari IncludeInternalDebugMenu 1
3. $ defaults write com.apple.Safari CA_COLOR_OPAQUE 1
4. Open (or restart) Safari
5. Debug -> Show Compositing Borders

(to turn these off, run the same commands with a boolean FALSE: i.e. defaults write com.apple.Safari CA_COLOR_OPAQUE FALSE)

Keep in mind that each composited layer has a limited width and height. For instance, if you’re creating an image carousel, chances are the dimensions of the element will be too big to fit the layer into memory as one piece. This means that when the element is animated, the GPU has to break up the layer manually into several manageable chunks. It’s much better to chunk it yourself. To do this, you trigger hardware acceleration on each chunk.

So you will change this:

.carousel {
  -webkit-transform: translate3d(0,0,0);  /* or translateZ(0); */
}

To this:

.carousel {
  -webkit-transform: translate3d(0,0,0);  /* or translateZ(0); */
}

.carousel-pane {
  -webkit-transform: translate3d(0,0,0);  /* or translateZ(0); */
}

Where carousel-pane represents each child element of the carousel.

More info: (slides) WebKit in Your Living Room (Matt Seeley, Netflix)

Also see HTML5 Techniques for Optimizing Mobile Performance.

HTTP Pipelining

Take advantage of HTTP Pipelining, which is often overlooked, but has broad support on mobile. This virtually eliminates round trip times, for all but the first request.

Opera and Android support pipelining, and the newly released iOS 5 has added support for it.

Great! So how do you make sure your server is taking advantage of pipelining?

The first request to every server is sent by itself (only one request on the connection), and the browser looks for two properties in the response:

1. Use of HTTP/1.1
2. An explicit “Connection: Keep-Alive” header (required by Android)

-Blaze.io

If these criteria are met, subsequent requests will be pipelined. Sweet!

DNS Prefetching

Take advantage of DNS Prefetching. This theoretically speeds up load times, but there have been some issues, so be sure to test it.

To turn off DNS Prefetching, serve this meta tag:

<meta http-equiv="x-dns-prefetch-control" content="off">

You can also explicitly force a DNS lookup:

<link rel="dns-prefetch" href="http://www.example.com/">

Avoid library code bloat

If you’re developing for smart phones, you may be able to dramatically reduce the data over the wire by simply using new JavaScript APIs instead of a full-blown library or framework such as jQuery Mobile (which is rather a UI framework and depends on the desktop version of jQuery).

Among the things offered by newer browsers, which should reduce your dependence on a library:

• querySelector/querySelectorAll: CSS selectors that replace the need for a selector engine
• getElementsByClassName
• classList – replaces the need for hasClass, addClass, removeClass helpers (available on iOS5, but not sure what else)
• XMLHttpRequest – it’s probably time to learn how to do this natively instead of using a wrapper that takes care of IE’s old implementation that requests an ActiveX object. Cross-Origin Resource Sharing also means we can easily share resources across domains.
• HTML5 history (pushState, etc) – though an older bug means you should maybe hold off for now (note: the fix is present in iOS 5, which is based on a newer WebKit)
• HTML5 Form Validation reduces our need for JavaScript-based validation
• HTML5 input types that reduce our need for custom UI controls (date, time, slider [range], etc)

You’ll want to consider using either a library that was optimized for the browsers you’re targeting (such as Zepto.js) or simply bring in a microlibrary for specific tasks you want to perform.

Clientside databases

Most mobile browsers currently support WebSQL, which is being phased out in support of indexedDB, which is not widely implemented. However, if you use a wrapper such as Lawnchair, the transition from one to the other is relatively painless.

TODO: testing

More

(to be expanded on later)

• use standard desktop best practices (for smart phones)
• requestAnimationFrame instead of setTimeout
• setImmediate (where available) instead of setTimeout(fn(){},0)
• Ajax parsing times: you might want to use Multipart XHR so you can yield the process as it’s going through big responses (to prevent the UI locking up)
• avoid rgba, box shadows, text shadow, etc, as this greatly degrades performance (especially on animations)
• high resolution screens may suffer from animation/transition slowness – it’s been shown that manipulating the viewport tag to scale the page will speed up canvas rendering, and high-dpi scaling concerns are also evident in CSS transforms
• WebKit animations are faster than animations done via JavaScript
• if you do use JavaScript-based animations, limit the UI updates to ~17ms, which is equivalent to 60fps, which is the fastest your display will refresh anyway, so it’s pointless to try to do anything faster than that. Keep track of the time since the last UI update and don’t do anything if 17ms haven’t passed.

On the horizon…

• SPDY: already implemented in Amazon’s Silk browser for Kindle. Will possibly be on Android phones when the browser is powered by Chrome
• indexedDB replacing WebSQL

Tools

Mobile Perf bookmarklet
pcapperf (Packet Capture Web Performance Analyzer)
Blaze Mobitest – tests the loading time of your site on actual phones

More Resources

Mobile Web Application Best Practices
Mobile Performance (Steve Souders)
Strangeloop Web Performance Hub / Mobile

(odometer image via henrybloomfield on Flickr)

Ever since Steve Souders started evangelizing web performance, it’s been pounded into our heads that extra HTTP requests add a lot of additional overhead, and that we should combine them if possible to dramatically decrease the load time of our web pages.

The practical implication of this has been to combine our JavaScript and CSS files, which is relatively easy and straightforward, but the harder question has been what to do with images.

Sprites

Image sprites are a concept taken from video games: the idea is to cram a ton of image assets into one file, and rearrange a “viewport” of sorts to view only specific pieces of that file at a time. For instance, a simple sprite that holds two images might have one viewport that only looks at the top half of the sprite (image #1), and another viewport that only looks at the bottom half (image #2).

On the web side of things, this means that those multiple requests have now been combined into one request. This is nice because it saves both the overhead of additional HTTP requests as well as the overhead of setting up an image’s file header each time.

But there’s a few drawbacks with using image sprites:

• hard to maintain and update: without some tool to help, manually editing and putting together image sprites is quite a chore
• increased memory consumption (possibly very dramatic): this is often overlooked. The time to deliver the images is decreased at the expense of a bigger memory and CPU footprint, especially for large sprites and sprites with a lot of whitespace.
• bleedthrough: for sprites that don’t have much whitespace to separate images, there’s an increased chance of nearby images visibly bleeding through other elements

Data URIs and Base64 encoding

Data URIs (see this, this, and this) and Base64 encoding goes hand-in-hand. This method allows you to embed images right in your HTML, CSS, or JavaScript.

Just like sprites, you save HTTP requests, but there’s also some drawbacks:

• base64 encoding makes file sizes roughly 33% larger than their original binary representations, which means more data down the wire (this might be exceptionally painful on mobile networks)
• data URIs aren’t supported on IE6 or IE7
• base64 encoded data may possibly take longer to process than binary data (anyone want to do a study on this?) (again, this might be exceptionally painful for mobile devices, which have more limited CPU and memory) (side note: CSS background-images seem to actually be faster than img tags)

The “33% larger” claim is generally accepted truth now, despite the fact that the figure varies wildly depending on the type of content. This is exactly what I wanted to test, albeit in a pretty limited and nonscientific way.

Before I tested, I wanted to keep in mind a few unverified intuitions (which aren’t entirely my own, but seem to be ideas that are floating around out there). Here’s a few questions I had before going to test:

• Is base64 encoding with gzipping roughly equal to the original filesize of the binary file?
• Is base64 encoding best for small images?
• Is base64 encoding best for small and simple icons and not good for pictures and photos?
• Is base64 encoding best when multiple files are merged together?

There’s something else I wanted to test: whether Gzipping binary image data made much difference. I know text compresses well, but is it even worth compressing JPEG files with Gzip, for instance?

I ran three tests: one with a set of small UI icons, one with a set of small photographs, and one with a set of the same photographs in a larger size. Though my tests were by no means extensive, they do show that care should be taken in making assumptions about base64.

Just a note about the tables: they are comparing the binary form (original png or jpeg) with the base64 form as it would appear in a CSS stylesheet, and comparing each of those with their gzipped form, which is most likely how they would be sent down the wire. The CSS representation has a few practical declarations and looks something like this:

.address-book--arrow {
	background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAp1JREFUeNqEU21IU1EYfu7unW5Ty6aBszYs6MeUjGVYokHYyH5E1B9rZWFEFPQnAwmy6Hc/oqhfJsRKSSZGH1JIIX3MNCsqLTD9o1Oj6ebnnDfvvefezrnbdCHhCw/n433P8z7nPe/hBEEAtX0U7hc164uwuvVSXKwZLoOmaRDim+7m9vZa0WiEKSUFFpNpCWlmMyypqTDRuYn6t3k8vmQ2gRDCxs0t9fW45F52aBTROJLtZl7nEZad2m+KtoQCQ0FBARyOCGRZ/q92I1WgqqXlfdd95VsrK8/pChIEqqpCkiQsiCII0aBQZZoWl8lzFDwsFjMl0DBLY8Lj41hBwK4jSQrWOIphL6xYyhwJDWGo6wFSaH1Y3PTCAsITE1oyAa8flhWkbSiCLX8vun11eiGIpiJ/z2nYdx5HqLdVV7elrOzsuqysL3rmBIGiKPizKCHHWY4PLVeQbnXAdegqdhy+hu8dDTBnbqQJZJ1A7u+vz7RaiymWCZgCRSF6Edk8b9cx+B/W6WuVxPaZnyiqXoPpyUmVYvkKTIFClHigEieKjYuSvETUllaF4GAUM1NT6ooaJDKx+aDfC9fByxj90REb+9ppmIoAscH/6leg8MS9DJXPAM9xHCM443K57C6biMjcHDaVVCHw9RmCA2/RGC5C00AqXk/m4p20HZK4CM/J3Zk9n0ecMBhDQnJHcrTisyMfdQXOilrdMfxcwoHq/fg5R59TiQV3hYGKo6X2J/c7LyQIjOx9GXhOw/zoJ8wEevRGyp53o/lGMNYsBgPtEwLecwov7/jGDKa1twT6o3KpL4MdZgGsWZLtfPr7f1q58k1JNHy7YYaM+J+K3Y2PmAIbRavX66229hrGVvvL5uzsHDEUvUu+NT1my78CDAAMK1a8/QaZCgAAAABJRU5ErkJggg==);
	width: 16px;
	height: 16px;
	background-repeat: no-repeat;
}

Ok, onto the tests!

Test #1: Five 16×16 icons from the Fugue Icon set (PNG)

* All numbers are byte sizes
** Numbers in parenthesis represent actual but impractical data. Unfortunately, images cannot be combined and delivered together in their binary form.

Takeaways:

• The binaries are always smaller.
• Sometimes Gzipping makes the files larger.
• Gzipping the base64 version brings the filesize close to the size of the original binary, but this ignores the fact that the binaries get Gzipped as well. The Gzipped binaries (how they would be delivered to the client) are always smaller than the Gzipped base64 images
• Combining files together dramatically reduces filesizes.

Practically, the developer has two options: deliver 5,140 bytes to the user in 5 separate HTTP requests, or 4,423 bytes in one HTTP request (CSS with base64 encoded image data). Base64 is the clear winner here, and seems to confirm that small icons compress extremely well.

Test #2: Five Flickr 75×75 Pictures (JPEG)

Takeaways:

• (some of the same takeaways as Test #1)
• Separately, photos aren’t too much bigger when base64 encoded and Gzipped. It’s very much within reason.

Practically, the developer can deliver 39,415 bytes in 5 separate requests, or 40,312 in 1 request. Not much filesize difference here, but 1 request seems preferable when we’re talking about 40kb.

Test #3: Five Flickr 240×160 Pictures (JPEG)

Takeaways:

• (some of the same takeaways as Test #1)
• Larger photos seem to bring the Gzipped binary and Gzipped base64 filesizes MUCH closer together, making the difference very minimal

The developer must choose between delivering 130,986 bytes in 5 HTTP requests, or 133,615 bytes in one HTTP request. Any good Souders follower would opt for the one request, BUT I would be careful here…

Caution: things aren’t always as they seem

There’s a huge caveat here: it may actually be more beneficial for perceived performance to deliver the images in 5 separate requests.

Why? Because 133,615 bytes is a lot to deliver all in one package to an end user who will be staring at blank placeholders for the duration. If the 5 base64 images all come in one request, that request will have to complete before ANYTHING is shown on the screen. All 5 images go from blank placeholders to almost immediately decoded from base64 and displayed in place.

Compare this with 5 requests that are most likely made in parallel and actually give a visual indicator to the user that actual image content is being downloaded, by showing parts of the images as they’re downloaded (you can also try a throwback to progressive JPEGs – really anything will be better than just a blank screen). That’s why it might actually be beneficial for perceived performance to just load images in the good old fashioned way. They will most likely load in parallel anyway, so the extra HTTP requests may actually not really make a difference. Not to mention it will be easier to let the browser manage the cache for each file instead of having to make your JavaScript manage your cache and prevent you from downloading an image that’s already stored away in localStorage or sessionStorage.

This being said, it’s generally advisable to put your common UI icons in base64 in your CSS, then let that whole chunk get cached by the browser. Those are usually clean vector icons as well, which seem to get compressed quite well (see Test #1).

But for image content, where there is nothing to be saved but HTTP requests, you should definitely think twice about base64 encoding to save requests. Yes, you will save a few HTTP requests, you won’t really be saving bytes, and the user might actually think the experience is slower because they can’t see the image content as it’s being downloaded. Even if you shave off a few milliseconds of wait time, the perceived performance is what matters most.

(EDIT: changed the wording of the “unverified intuitions” section from “not verified” to actual questions, to make it clearer)

Raw results (iOS 4.3 beta 2 running on my iPhone 3GS)

============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total:                  5357.3ms +/- 2.3%
--------------------------------------------

  3d:                    736.9ms +/- 7.5%
    cube:                260.1ms +/- 3.5%
    morph:               206.0ms +/- 23.2%
    raytrace:            270.8ms +/- 4.4%

  access:                617.4ms +/- 4.1%
    binary-trees:        118.8ms +/- 4.7%
    fannkuch:            180.4ms +/- 0.7%
    nbody:               222.0ms +/- 11.7%
    nsieve:               96.2ms +/- 0.8%

  bitops:                288.7ms +/- 9.6%
    3bit-bits-in-byte:    38.0ms +/- 4.3%
    bits-in-byte:         70.5ms +/- 39.0%
    bitwise-and:          85.1ms +/- 1.9%
    nsieve-bits:          95.1ms +/- 2.1%

  controlflow:            34.7ms +/- 3.2%
    recursive:            34.7ms +/- 3.2%

  crypto:                308.3ms +/- 1.1%
    aes:                 175.2ms +/- 0.5%
    md5:                  79.3ms +/- 3.9%
    sha1:                 53.8ms +/- 0.8%

  date:                  706.2ms +/- 5.5%
    format-tofte:        341.3ms +/- 11.0%
    format-xparb:        364.9ms +/- 1.1%

  math:                  605.9ms +/- 1.1%
    cordic:              224.5ms +/- 1.0%
    partial-sums:        263.4ms +/- 2.3%
    spectral-norm:       118.0ms +/- 1.2%

  regexp:                152.5ms +/- 5.5%
    dna:                 152.5ms +/- 5.5%

  string:               1906.7ms +/- 0.7%
    base64:              247.9ms +/- 2.4%
    fasta:               268.5ms +/- 0.6%
    tagcloud:            370.4ms +/- 3.0%
    unpack-code:         676.2ms +/- 0.3%
    validate-input:      343.7ms +/- 1.4%

Raw results for iOS 4.0

Related

BlackBerry Torch SunSpider results (JavaScript benchmark)
iPhone 4 SunSpider test results (22% faster than iPhone 3GS)
JavaScript SunSpider test: iOS 3.1.3 versus iOS 4 GM
JavaScript SunSpider: HTC Evo versus HTC Incredible versus Nexus One

============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total:                 322.2ms +/- 4.9%
--------------------------------------------

  3d:                   55.0ms +/- 15.3%
    cube:               19.8ms +/- 12.1%
    morph:              16.6ms +/- 35.8%
    raytrace:           18.6ms +/- 22.5%

  access:               32.8ms +/- 12.4%
    binary-trees:        1.8ms +/- 30.9%
    fannkuch:           14.2ms +/- 7.3%
    nbody:              12.6ms +/- 26.7%
    nsieve:              4.2ms +/- 24.8%

  bitops:               29.4ms +/- 10.2%
    3bit-bits-in-byte:   2.4ms +/- 28.4%
    bits-in-byte:        8.0ms +/- 15.5%
    bitwise-and:         8.6ms +/- 21.9%
    nsieve-bits:        10.4ms +/- 21.7%

  controlflow:           2.4ms +/- 28.4%
    recursive:           2.4ms +/- 28.4%

  crypto:               22.0ms +/- 10.6%
    aes:                 9.8ms +/- 27.5%
    md5:                 6.2ms +/- 22.0%
    sha1:                6.0ms +/- 20.7%

  date:                 33.2ms +/- 11.7%
    format-tofte:       16.6ms +/- 20.2%
    format-xparb:       16.6ms +/- 13.6%

  math:                 32.6ms +/- 15.2%
    cordic:             12.0ms +/- 29.3%
    partial-sums:       15.2ms +/- 20.4%
    spectral-norm:       5.4ms +/- 20.6%

  regexp:               15.6ms +/- 7.1%
    dna:                15.6ms +/- 7.1%

  string:               99.2ms +/- 14.0%
    base64:              9.2ms +/- 14.8%
    fasta:              14.4ms +/- 27.7%
    tagcloud:           27.2ms +/- 23.2%
    unpack-code:        31.8ms +/- 14.5%
    validate-input:     16.6ms +/- 18.0%

Related articles

iPhone 4 SunSpider test results (22% faster than iPhone 3GS)
JavaScript SunSpider test: iOS 3.1.3 versus iOS 4 GM
JavaScript SunSpider: HTC Evo versus HTC Incredible versus Nexus One
Sencha: BlackBerry Torch: The HTML5 Developer Scorecard

The problem with Switch statements

The basic switch statement in JavaScript looks something like this:

var foo = 'c';

switch(foo) {
  case 'a':
  break;

  case 'b':
  break;

  case 'c':
  break;

  default:
}

So what’s wrong with this? The JS engine has to examine a bunch of unrelated cases until it finds the relevant one, executes the code, then breaks out of the switch because the job is done (this is why it’s important to break!). In the above example we had to go through case A and case B until finally reaching case C. What’s worse is that if it didn’t match any of these cases, the JS engine has to jump through ALL of the cases before it reaches Default, the fall-through case.

Actually it’s not so bad, as long as there are a limited number of cases. It’s probably no big deal if you only have a few cases to jump through. The problem gets bigger as your number of cases increases (some of you may know this as O(n)). What happens when there’s 10 cases? Then there’s potentially 10 checks on cases (assuming what ended up being executed was the default). 100 cases? Then potentially 100 checks.

What would be better is if there were a way to reduce the number of checks. One way would be to put the most frequently used cases at the top. This would alleviate some of the pain, but you still end up with extra processing while the JS checks each case. It would be ideal to avoid this extra processing altogether.

An alternative: The hash table

There is a way to avoid this extra processing! It’s by leading the code directly where it needs to go, without unnecessary checking of unrelated cases.

You can do this using a hash. In JavaScript we accomplish this with an object:

var foo = 'c';
var cases = {};
cases['a'] = function() {
  alert('I am A!');
};
cases['b'] = function() {
  alert('I am B!');
};
cases['c'] = function() {
  alert('I am C!');
}

if(typeof cases[foo] == 'function') {
  // only executes if we've defined it above
  cases[foo]();  // I am C!
} else {
  // default (the fallthrough)
}

There we go! No extra case checking here. We’ve led the JS straight to the code we want to execute!

Performance improvement…?

So.. this hash lookup seems faster in theory, but what about in practice? Unfortunately I ended up with some mixed results…

I created a simple performance test on jsperf.com and got these results:

* Ops/sec = Operations per second. Higher is better
** Chrome, Safari, Opera, and Firefox were tested on Mac OSX 10.6.4 2.53GHz Intel Core i5. IE tests were run on Windows 7 64bit 2.4GHz Quad Core

The Results

From the results, it looks like the hash optimization is only a benefit for Chrome, IE6-IE8, and Android. That’s quite a specific sampling. My guess is that the other browsers have implemented some sort of Switch statement optimizations that actually turn the hash optimization into an antipattern.

More info

Although I first read about this online, by no surprise this trick also appears in Nicholas Zakas’s High Performance JavaScript in a section on “Lookup Tables” (p. 72).

Thoughts

The Incredible is just slightly faster than the Evo, to the point where it’s probably negligible or within a margin of error. Both of these phones run on Android 2.1 with HTC’s Sense UI modifications, and represent the latest and greatest in Android phones available on the market today. Both run on the same 1GHz Snapdragon processor (QSD8650). The Nexus One is a bit older, and runs with an older version of the Snapdragon processor (QSD8250), however it still runs at 1GHz just like the other two phones.

As you can see the Nexus One blows away all the competition because it’s running Android 2.2 Froyo. These results were quite a shock to me and are quite impressive. These results even blow away Apple’s new iOS 4 running on my iPhone 3GS, which clocked in at a total time of 13787ms compared to the Nexus One’s startling 5452ms.

Testing methodology

Test: SunSpider 0.9.1

Devices: HTC Evo (Android 2.1), HTC Incredible (Android 2.1), HTC Nexus One (Android 2.2)

The SunSpider test was run five times on each phone. The phone was completely turned off and on before each test. The most extreme values of the five tests were thrown out, and the resulting four tests were averaged (sometimes from three tests when the values were very close together).

Raw results:

SunSpider HTC Evo results (5 tests)

SunSpider HTC Incredible results (5 tests)

SunSpider HTC Nexus One results (5 tests)

Related links

JavaScript SunSpider test: iOS 3.1.3 versus iOS 4 GM

Thoughts

After running these SunSpider tests, it looks like overall there’s significant speed gains between iOS 3.1.3 and iOS 4 GM. However, it’s concerning from these tests there were some things that actually ran slower on iOS 4. This either represents a real speed loss between the versions, a margin of error, or some flaw or inconsistency while testing. Or maybe I possibly have some wrong setting on my phone? Any input would be appreciated.

Testing methodology

Test: SunSpider 0.9.1

Device: iPhone 3GS

The test was run five separate times on the same phone for each version of the OS. The phone was completely turned off and on before each test.

The most extreme values of the five tests were thrown out, and the resulting four tests were averaged (sometimes from three tests when the values were very close together). I’m no statistics expert, so if you’d like to work it out for yourself, here are my raw test results:

SunSpider iOS 3.1.3 results (5 tests)

SunSpider iOS 4 results (5 tests)

Related links

Speed Test: iPhone 3GS Even Faster than Apple Claims

Usually developers are more interested in just getting the dang code to work, and as a result actual the testing and maintenance of JavaScript isn’t talked about too much, so I’m sure this will be new territory for many developers. And since it’s John Resig speaking, there was of course a bit about using TestSwarm, a testing distributed framework-agnostic automated testing tool (that’s a mouthful!).

Included in the talk are good things to note while testing, such as the fact unless you’re running Firefox or Chrome on Windows, all test times have a margin of error of up to 15ms (not to be confused with PPK’s observation of the delay between JavaScript computation and browser rendering).

(via YUIBlog)