Javascript performance testing

I've been horribly disappointed with IE's performance with some of my javascript, so the point that I've pretty much decided to simply hide the troublesome functions from IE users. That's nearly 85% of our audience who won't get to experience the work I've put in to some of this stuff. I thought I needed to get some actual figures on performance, so I took the worst offender and turned it into a test case. Then I thought while I'm doing that I might as well compare jQuery 1.2.6 to jQuery 1.3.1. So I made a 2nd test case.

First of all, you need to understand that I'm terribly abusing the plugin in question. I'm using treeview to do the online version of our organizational chart. There's a demo showing how it works with large lists. That demo has 290 total list items, 53 of which are expandable. Our org chart has 403 list items, 81 of which are expandable. It's large enough that I had to edit the default images to extend all the way down the fully expanded list.

I added a crude counter to the plugin. Here's the code. The lines involving Date() and getTime() and alert() functions are my additions (114 to 127 or so). It's far from perfect, but it should be equally imperfect for all browsers and therefore free of bias. What this does is whenever the expand all or collapse all function is triggered, it grabs the time early on in that process, then grabs the time again near the end of that process, computes the difference (in milliseconds) and alerts the value. In each browser I expanded the list, recorded the value, collapsed the list, recorded that value, and repeated the process until I had 15 measurements for both the expand and the collapse feature.

The Data

We'll do the Mac browsers first. I'm running on a 20 inch iMac with a 2.0 gHz Intel Core Duo processor and 4 gigs of RAM. I ran these tests under "normal" conditions. So I had other applications going and in the case of Firefox several other tabs open.

All figures are in milliseconds.

Firefox 3.0.6

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	838 840 847 826 827 821 839 835 827 827 825 839 824 816 845	1178 1184 1183 1188 1188 1184 1187 1205 1186 1190 1189 1182 1185 1182 1190	506 514 511 493 482 513 480 510 493 484 508 484 492 512 491	1006 1024 999 998 1011 998 1010 1003 1009 1008 1005 1002 1000 997 994
Min	816	1178	480	994
Max	847	1205	514	1024
Range	31	27	34	30
Median	827	1186	493	1003
Average	831.73	1186.73	498.2	1004.26

The most surprising thing here is that jQuery 1.2.6 is actually a bit faster than 1.3.1. Of course, 1.3.2 is supposed to release soon. And as we'll see that's not true for every browser. Collapsing takes longer than expanding. I'll leave explaining that to someone that knows more about jQuery DOM transversal.

Safari 3.2.1

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	262 256 267 269 264 269 266 267 270 270 267 269 292 267 267	161 159 164 163 167 164 169 165 163 164 165 164 164 166 166	311 311 326 316 314 311 311 319 312 314 310 316 318 318 317	147 146 149 151 148 148 146 149 149 149 149 151 148 152 150
Min	256	159	310	146
Max	292	169	326	152
Range	36	10	16	6
Median	267	164	314	149
Average	268.13	164.27	314.9	148.8

At first I thought the performance boost for 1.3.1 here was pretty small, but I think the fact that all the numbers involved is tiny threw me off. If you add up the median figures, you'll find a difference of about 7.5%. That's nothing compared to the relative advantage Safari already has over most of the other browsers, but I think it's still a decent speed boost for the new jQuery release.

Safri's js engine is awesome. :)

Can anyone explain why in Safari collapse is quicker than expand, but in Firefox it's the other way around?

Opera 9.63

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	692 690 737 667 711 703 715 703 704 692 719 680 731 695 682	698 709 721 709 715 730 697 695 692 696 705 704 697 713 696	875 944 904 875 872 869 864 929 905 910 891 885 926 857 918	691 711 692 692 684 692 741 688 652 687 674 707 701 692 690
Min	667	692	857	652
Max	737	730	944	741
Range	70	38	87	89
Median	703	698	891	692
Average	701.4	705.13	894.93	692.93

Opera seems slow compared to Safari, but it's on par with Firefox, and more than tolerable. We also see a decent speed boost with 1.3.1 here.

Windows Browsers

I'm running Windows XP via virtualization with 412 megs of RAM. These tests were run with no other running applications, so more "ideal" conditions than the Mac browsers got. Then again, the Macs get a ton more RAM and aren't running in a virtual machine. I tested both Firefox 2 and Firefox 3 because a decent portion of our Firefox using audience haven't upgraded yet.

Firefox 2.0.0.20

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	1018 1069 1019 1019 1019 1019 989 1049 1019 989 1029 1009 1029 999 969	1169 1109 1129 1129 1139 1159 1129 1189 1149 1129 1128 1129 1129 1189 1119	629 629 609 649 609 639 619 629 619 649 630 629 640 629 639	740 719 739 729 749 749 749 730 739 739 759 750 759 719 769
Min
Max
Range
Median
Average

Here we're starting to get slower than I'd like. A full second for anything to happen is a bit much. Then again, 403 list items in 81 lists is a lot of munch on. Once again we loose a little speed with the new jQuery release. Hmm.

Firefox 3.0.6

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	431 449 461 434 464 472 558 493 481 436 457 442 472 462 458	657 688 648 623 657 708 632 650 695 649 634 649 648 659 625	450 467 424 475 443 456 446 472 460 477 449 464 483 468 440	618 650 634 638 667 637 632 632 646 643 622 614 628 651 640
Min
Max
Range
Median
Average

Here we go! Come on Firefox users, update already!

Safari 3.2.1

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	253 326 277 257 268 288 278 269 244 271 264 281 284 281 293	141 171 148 160 147 143 147 150 151 171 163 158 154 142 147	303 300 324 327 313 312 283 297 355 332 367 332 360 317 307	130 133 129 134 135 134 129 130 132 130 133 135 148 133 129
Min
Max
Range
Median
Average

Safari continues to be awesome, even on Windows. :)

Opera 9.63

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	470 480 469 480 480 499 490 580 469 519 460 470 479 563 480	399 410 429 409 419 399 419 409 400 400 400 430 410 399 399	949 989 949 929 939 939 939 929 939 939 959 949 958 1009 989	370 380 390 409 400 390 370 390 380 400 399 420 380 400 420
Min
Max
Range
Median
Average

With Opera on Windows, we see some pretty solid speed boosts with the 1.3.1 release of jQuery.

Chrome 1.0.154.48

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	229 203 191 195 158 208 201 188 204 244 360 327 307 305 178	88 77 119 88 79 106 84 86 95 158 116 145 135 61 81	512 598 546 607 607 640 526 674 545 631 549 635 581 616 531	111 112 133 115 109 105 103 103 105 106 133 112 109 110 114
Min
Max
Range
Median
Average

Chrome also sees some good boosts from 1.3.1 and manages to out awesome even Safari. When you look at the range as a percentage, it's all over the place. Sometimes the alert box would have a check box in it that would say something to the effect of "prevent this site from producing alert boxes" and I think those tended to take longer to generate. Maybe Chrome's doing other stuff behind the scenes. I'd need to do more specific research to figure this one out.

IE 7.0.5730.13

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	2990 2897 2867 2917 3016 2877 2877 2847 2907 2897 2877 2916 2897 2888 2848	1718 1738 1832 1718 1668 1708 1678 1678 1698 1709 1728 1738 1689 1688 1708	2538 2587 2557 2557 2577 2568 2567 2557 2548 2707 2548 2708 2558 2568 2568	1248 1219 1249 1248 1279 1249 1208 1228 1258 1229 1229 1239 1389 1229 1239
Min
Max
Range
Median
Average

Ugh! I'm in browser hell! What's up with this? 2.5 to 3 seconds to expand the list? Come on!

IE6

I don't know the exact version number here because of the way the Multiple IE installer works.

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	10779 12577 13060 14085 15034 16123 17272 19549 19859 21121 23080 23249 24961 24770 25953	4935 5165 5774 5964 6314 7032 7152 8021 8231 8611 9082 9402 9892 9959 10349	11078 12617 13167 14785 15684 16876 19643 21740 20642 21947 22726 23925 36235 28810 29422	3899 4005 4356 4715 5105 5534 6453 7366 6842 7453 7801 8241 9700 9364 9849
Min
Max
Range
Median
Average

No. I was wrong before. This is browser hell. Notice how each test gains about 1 full second? I think that means we've got memory leaks. I closed and relaunched the browser between test cases to avoid skewing the results between jQuery versions.

I'll complete the number crunching and do a bit more in depth analysis tomorrow. It's 5 o'clock and I'm heading home for the day.

Content! Third-hand content, but still, content!

One of the people I encountered in all the discussion on higher ed in the web design industry is Kurt Schmidt. (ImTheSchmidt.com, isn't that one of the greatest domain names ever?)

Since then, I've been following his blog semi-daily. Earlier today he posted Make Your Site Faster - Or Else!, which is his take on info from Geeking with Greg. I started to reply to Kurt, but about the time I started my 3rd paragraph I figured I should just turn it into a blog post here. See what I mean about blogs and peer review?

The importance of speed and performance on the user experience is something that I'm currently struggling with. In the redesign I'm working on, I've made extensive use of jQuery. Everything runs super quick on all the Mac browsers. Chrome and Safari on Windows also blaze. Firefox on Windows is noticeably slower than the rest, but still tolerable. Who does that leave to drool on his shirt and crap his pants at the browser testing party?

Both IE6 and IE7 are intolerably slow. I've considered simply filtering out IE6 from all the js goodness. Since I've designed for progressive enhancement, everything will still function, just not the same way. But that still leaves IE7 users with a clunky and ungraceful experience. As much as I hate IE, I won't kid myself about it's market share.

The best solution from the users' point of view would be to test each jQuery implementation on the site separately and selectively enable/disable for IE based on performance in context. Some of the stuff I'm doing on the redesign works fine in IE. I know because I've been doing it on the current site for more than a year.

I also need to dig deeper into my jQuery-fu and make sure my calls are as efficient as possible. But some of it is straight up plug in functionality. I'm not too keen on rewriting other people's plug ins to increase efficiency. The next version of Firefox will probably incorporate ideas currently only available in Chrome's V8 js engine, making it's tolerable performance that much better. But that will probably just widen the gap between IE users and everyone else even more!

The next current version of jQuery (which I haven't updated to yet, oops) will also have a revamped selection engine that should boost performance across the board. But unless those changes impact performance in IE a hell of a lot more than everyone else, the gap will still be huge. It's the gap in the user experience that worries me.

What little testing I've done so far would seem to indicate that IE6 is bad at say 2000 milliseconds. IE7 is barely any better and a lot less consistent, with speeds ranging from 1500-2300ms. Firefox is around 700ms. Opera is 300-400ms. Safari is 200-300ms. Chrome is about 150ms (10 times better than IE7's best run!). I guess with those numbers, even if sizzle improves performance by 20% across the board, Chrome gains 30ms and IE7 gains at least 300ms. Maybe the gap will get smaller. But still, 1.2 seconds waiting for all the js to trigger is unacceptable from a user centered outlook. But why should I be punished as a designer and the 20% or so of our users who don't run IE miss out on those enhancements simply because IE can't get it's crap together? These sorts of situations are the only part of my job I hate. I'll stop there before this turns into a rant.

Well, now that I know the new version of jQuery is out, it's pointless to flap my jaw any more about this stuff until I've updated and retested. I've got work to do.

The Social Side of Social Bookmarking

There's a decent amount of discovery power available through Ma.gnolia as well. I still think Stumble Upon does it better. But it would be silly to not explore the 2nd best tool available for the job. Ironically, but doing so, I was quickly reminded of the problems with default tagging in SU. I pulled up a couple of recent bookmarks from Jeffry Zeldman.

The first is a blog post from Simon Clayson on feeding IE6 a basic style sheet using the sort of techniques that were once common for targeting Netscape Navigator 4 with a set of specific, dumbed down styles while simultaneously protecting NN4 users from that browsers botched implementation of the majority of CSS which was safe to show to less craptastic browsers. Now NN4 is little but a ghost to haunt the nightmares of us old school CSS scribes and IE6 is now the crappiest browser still in common usage. I'll probably spend the rest of December debugging the redesign in IE6. Had I found this idea a year ago, I probably would have served IE6 a very simple style sheet and skipped the debugging. In all honesty, even at this stage it may be less work to implement these ideas rather than try to “fix” IE6.

So anyway, I thought this was a potentially useful technique, so I thumbed it up. This didn't pull up the form for submitting new content to SU, so I knew someone else had already submitted this particular link. This gave me a great opportunity to see what the default tag would be. jackosborne says this page is primarily about “graphic-design”. It deals exclusively with serving specific CSS code targeted at a specific web browser. I can think of at least half a dozen tags more useful for this content than “graphic-design”. But the current SU system gives too much power to the person submitting the content. Jack's actually got more stumbles tagged “web-design” (54) than “graphic-design” (45), but apparently that's due to other people's default tags on the pages he is thumbing up. Looking at his discoveries, he's also submitted this article on Five CSS Design Browser Differences I Can Live With by Andy Clarke and Using jQuery for Background Image Animations from Snook.ca as “graphic-design”. Maybe that tagging scheme server Jack well. It makes SU virtually worthless for me when it comes to organizing and retrieving the resources I discover through it.

The other page I discovered via Zeldman is Western Civ's guide to CSS browser support. Again, this page deals exclusively with CSS and web browsers, so for my purposes it would be pretty easy to tag. It was submitted by SU user kancerman uh...wow, 3 years ago. If I'm reading this right, I'm only the 10th person to thumb this up in those 3 years. That could be because it was submitted into the category “internet-tools”. For me, that category is better suited for things like online mortgage calculators or WriteBoard. But due to the way kancerman submitted this page, “internet-tools” is the default tag. Now I can look at his entry for this page and see that his 2nd tag is in fact “CSS”, but since that's the 2nd tag on his entry, it holds no bearing for how it is tagged by default when I thumb up the page.

Maybe this problem in the design of SU is worse than I thought. Not only does the default tagging scheme make it harder for me to go back and look up stuff I have previously thumbed up without bothering to write a review and/or manually tag myself. But it also seems to have a negative impact on the effectiveness of SU to function as a discovery engine. How many times have I found a page via means other than SU, thumbed it up, didn't see the new content submission form pop up, assumed whoever beat me to the punch on submitting the content at least submitted it properly, and went on my way? How often does the average SU user do that? One thing I've noticed since I started paying attention to the default tagging scheme in SU is how often I see content that is submitted into the wrong category. If I have found this content via SU, then I can use the “report last stumble” feature. But that only works if I stumble into something in one of my defined interests that really should be tagged as some other of my defined interest. If someone submits a CSS gallery as “photography” for example. But if I get to that page without being referred there by SU, there's no way for me to bring the miscategorization to the attention of whoever addresses such things. That is most likely to happen if someone submits content that should fall within one of my defined interests as pertaining to a topic of interest that isn't on my list.

Oh look, this jQuery plug-in has been submitted under “alternative-medicine”

There's no way I can do anything about that. All I can do is tag it properly within my own account. But since very few web designers are going to be stumbling through the alternative medicine category (then again, maybe I assume too much), and very few people looking for alternative medicine information will give a rat's ass about a jQuery plug-in, very few people who care about that content will ever stumble into that content. I can't even resubmit it. Once a page is submitted, all I can do is tag and review it myself. In effect, such content is quarantined, cut off from it's true target audience. I've got to think there are ways for SU to address this. If Mac OS X can have a pretty effective summarize tool built in, can't a similar algorithm be run against the content of new submissions to SU in an attempt to verify the categorization of that content? Couldn't meta tags, key words, or the sort of tricks search engines use to categorize content be applied? I know these things aren't cheap, but they are possible, and SU has a larger user base than delicious (which may actually be a big part of the problem).

Antique Browser

What reminded me to come back and complete that last post is a response I wrote in yet another design blog. The thoughts that lead to that reply were obviously inspired by the ideas that have been kicking around in my head for the past month while my last post was locked in draft stasis. I'll go ahead and share those thoughts here as well, slightly more detailed than in my reply to Raymond's entry on hating IE6 (I took the time to do the math).

I was thinking just last night that an appropriate metaphor to explain to non-techies why IE6 sucks would be to compare it to an antique car.

The Model T was not the first car, and Mosiac was not the first browser, but both brought those particular products into the main stream an birthed industries to support them. So we’ll use those are our anchor points for comparison.

The first Model T rolled off the assembly line in 1908. So the automobile as we know it is 100 years old this year. Mosaic was released on April 22, 1993. Today being December 5th, 2008, that makes the modern web browser 15.6226243 years old. IE6 came out August 27, 2001, so it is 7.274739 years old. That means IE6 is 46.5654096% as old as the modern web.

If you were driving a car manufactured in 1963, would you expect to meet modern safety standards? Would you expect to pass modern emissions testing? You couldn’t use a modern gas pump without using an artificial additive to replace the lead that was commonly added to fuels 46 years ago. There’s a reason most people who own antique cars usually drive them only to auto shows or for the occasional pleasure cruise and don’t use them for every day driving. If you drove such a car the typical 1,000 miles or so a month, you would expect to pay much more in maintenance costs than someone driving a newer model.

The primary appeal of an antique car is the cool factor. They offer styles not available in the modern market and you just look cooler behind the wheel of a cherry antique than just about any modern automobile. No one is impressed by your antique browser. So why are you still using it?

Charging more to support craptastic and/or old browsers

Good God. I literally started writing this entry a month ago, got busy, and left it suspended in draft mode for ages. Let me brush off the dust and get this bad boy ready for publication...

This started as a reply to a post from Philip Beel, which was in turn explaining his thoughts in response to item #10 on Ten Web Development Tips I Wish I'd Known Two Years Ago over at hackification. By the time my comment hit it's 6th paragraph, I figured I'd be better off just putting my thoughts into my own blog post. Ain't the web grand?

Philip says in part:

Consider that you were sold a car, but told you could only drive it on motorways, as the wheels may fall off if you drove it on any other roads.

But I think that's the wrong way to look at the situation. My thinking is a bit closer to the hackification article, which states in part:

Explain to the client that since older browsers work in a different way to modern ones (which they do), that’s extra work and hence extra cost. (I’d suggest approximately 10% on top). Explain that IE6 has approximately 25-30% market share, and let them make the cost/benefit call. Money has an amazing focusing effect: it forces people to really think about what they want and need. (emphasis added)

As designers, we're concerned with stuff like use experience, so we tend to think of things in terms of the end user/consumer. But our clients are often business people, so I think it's more effective to talk to them in the language of business people.

I think the “buying a car that only works on motorways” example wouldn't work well on most clients. The situation is more akin to opening a new convenience store 7 years after the last gasoline powered car has rolled off the assembly line. Would you want to take on the cost of installing and maintaining a gas pump for a market segment that is no longer in the majority and will be forever dwindling? Even installing just a single pump alongside the hydrogen system (or quick charge stations or whatever the new technology happens to be) would require redoing a lot of the same type of work. The systems aren't interchangeable or compatible. There would be redundant systems; more things to build and eventually need repair.

I think most clients would understand that analogy. It probably would not be hard to come up with one more targeted towards their own business model.

But another problem is that consumers are better informed as to the cars they drive than the browsers they use. The owner and employees of the hypothetical fueling station above would probably never have to deal with an irate customer who doesn't understand why the hydrogen hose or quick charge plug won't fit into his gas tank. Sites like Browse Happy can come in handy for educating consumers, but the kind of people who are the most problematic (don't even realize browsers and other software are entities separate from “the computer”, never update whatever shipped with their operating system, etc) are the least likely to go there or truly understand the content once they get there. So the ethical considerations are still not dodged.

But I think the ethical issues are drastically different from the issues involved in getting clients to understand the cost associated with “supporting” old browsers. And a lot depends on how you define “support” as well.

If you want the 0.1% (or hopefully even lower than that) of folks still surfing on Netscape 4 to be able to use some Ajax based enhancement, that's gonna take a heck of a lot more work. What's the return on investment of that work? Expecting the 21.53% (and dropping) of IE6 users to see rounded corners exactly like what someone on Safari 3 sees is like expecting a 1993 Geo Metro to handle and perform just like a Tesla Roadster.

In my current work with jQuery, I'm noticing drastic performance difference even between IE6 and IE7. The features work in IE6, but it's not as smooth nor as quick. With the move towards just in time scripting execution in Google Chrome and future browsers, such performance differences will become even more noticeable. So what's the acceptable level of differences in appearance and performance from browser to browser? That will change from project to project, but the price tag for the work we do needs to also change to reflect the shift in work load (be it up or down).