Javascript performance testing

I've been horribly disappointed with IE's performance with some of my javascript, so the point that I've pretty much decided to simply hide the troublesome functions from IE users. That's nearly 85% of our audience who won't get to experience the work I've put in to some of this stuff. I thought I needed to get some actual figures on performance, so I took the worst offender and turned it into a test case. Then I thought while I'm doing that I might as well compare jQuery 1.2.6 to jQuery 1.3.1. So I made a 2nd test case.

First of all, you need to understand that I'm terribly abusing the plugin in question. I'm using treeview to do the online version of our organizational chart. There's a demo showing how it works with large lists. That demo has 290 total list items, 53 of which are expandable. Our org chart has 403 list items, 81 of which are expandable. It's large enough that I had to edit the default images to extend all the way down the fully expanded list.

I added a crude counter to the plugin. Here's the code. The lines involving Date() and getTime() and alert() functions are my additions (114 to 127 or so). It's far from perfect, but it should be equally imperfect for all browsers and therefore free of bias. What this does is whenever the expand all or collapse all function is triggered, it grabs the time early on in that process, then grabs the time again near the end of that process, computes the difference (in milliseconds) and alerts the value. In each browser I expanded the list, recorded the value, collapsed the list, recorded that value, and repeated the process until I had 15 measurements for both the expand and the collapse feature.

The Data

We'll do the Mac browsers first. I'm running on a 20 inch iMac with a 2.0 gHz Intel Core Duo processor and 4 gigs of RAM. I ran these tests under "normal" conditions. So I had other applications going and in the case of Firefox several other tabs open.

All figures are in milliseconds.

Firefox 3.0.6

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	838 840 847 826 827 821 839 835 827 827 825 839 824 816 845	1178 1184 1183 1188 1188 1184 1187 1205 1186 1190 1189 1182 1185 1182 1190	506 514 511 493 482 513 480 510 493 484 508 484 492 512 491	1006 1024 999 998 1011 998 1010 1003 1009 1008 1005 1002 1000 997 994
Min	816	1178	480	994
Max	847	1205	514	1024
Range	31	27	34	30
Median	827	1186	493	1003
Average	831.73	1186.73	498.2	1004.26

The most surprising thing here is that jQuery 1.2.6 is actually a bit faster than 1.3.1. Of course, 1.3.2 is supposed to release soon. And as we'll see that's not true for every browser. Collapsing takes longer than expanding. I'll leave explaining that to someone that knows more about jQuery DOM transversal.

Safari 3.2.1

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	262 256 267 269 264 269 266 267 270 270 267 269 292 267 267	161 159 164 163 167 164 169 165 163 164 165 164 164 166 166	311 311 326 316 314 311 311 319 312 314 310 316 318 318 317	147 146 149 151 148 148 146 149 149 149 149 151 148 152 150
Min	256	159	310	146
Max	292	169	326	152
Range	36	10	16	6
Median	267	164	314	149
Average	268.13	164.27	314.9	148.8

At first I thought the performance boost for 1.3.1 here was pretty small, but I think the fact that all the numbers involved is tiny threw me off. If you add up the median figures, you'll find a difference of about 7.5%. That's nothing compared to the relative advantage Safari already has over most of the other browsers, but I think it's still a decent speed boost for the new jQuery release.

Safri's js engine is awesome. :)

Can anyone explain why in Safari collapse is quicker than expand, but in Firefox it's the other way around?

Opera 9.63

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	692 690 737 667 711 703 715 703 704 692 719 680 731 695 682	698 709 721 709 715 730 697 695 692 696 705 704 697 713 696	875 944 904 875 872 869 864 929 905 910 891 885 926 857 918	691 711 692 692 684 692 741 688 652 687 674 707 701 692 690
Min	667	692	857	652
Max	737	730	944	741
Range	70	38	87	89
Median	703	698	891	692
Average	701.4	705.13	894.93	692.93

Opera seems slow compared to Safari, but it's on par with Firefox, and more than tolerable. We also see a decent speed boost with 1.3.1 here.

Windows Browsers

I'm running Windows XP via virtualization with 412 megs of RAM. These tests were run with no other running applications, so more "ideal" conditions than the Mac browsers got. Then again, the Macs get a ton more RAM and aren't running in a virtual machine. I tested both Firefox 2 and Firefox 3 because a decent portion of our Firefox using audience haven't upgraded yet.

Firefox 2.0.0.20

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	1018 1069 1019 1019 1019 1019 989 1049 1019 989 1029 1009 1029 999 969	1169 1109 1129 1129 1139 1159 1129 1189 1149 1129 1128 1129 1129 1189 1119	629 629 609 649 609 639 619 629 619 649 630 629 640 629 639	740 719 739 729 749 749 749 730 739 739 759 750 759 719 769
Min
Max
Range
Median
Average

Here we're starting to get slower than I'd like. A full second for anything to happen is a bit much. Then again, 403 list items in 81 lists is a lot of munch on. Once again we loose a little speed with the new jQuery release. Hmm.

Firefox 3.0.6

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	431 449 461 434 464 472 558 493 481 436 457 442 472 462 458	657 688 648 623 657 708 632 650 695 649 634 649 648 659 625	450 467 424 475 443 456 446 472 460 477 449 464 483 468 440	618 650 634 638 667 637 632 632 646 643 622 614 628 651 640
Min
Max
Range
Median
Average

Here we go! Come on Firefox users, update already!

Safari 3.2.1

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	253 326 277 257 268 288 278 269 244 271 264 281 284 281 293	141 171 148 160 147 143 147 150 151 171 163 158 154 142 147	303 300 324 327 313 312 283 297 355 332 367 332 360 317 307	130 133 129 134 135 134 129 130 132 130 133 135 148 133 129
Min
Max
Range
Median
Average

Safari continues to be awesome, even on Windows. :)

Opera 9.63

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	470 480 469 480 480 499 490 580 469 519 460 470 479 563 480	399 410 429 409 419 399 419 409 400 400 400 430 410 399 399	949 989 949 929 939 939 939 929 939 939 959 949 958 1009 989	370 380 390 409 400 390 370 390 380 400 399 420 380 400 420
Min
Max
Range
Median
Average

With Opera on Windows, we see some pretty solid speed boosts with the 1.3.1 release of jQuery.

Chrome 1.0.154.48

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	229 203 191 195 158 208 201 188 204 244 360 327 307 305 178	88 77 119 88 79 106 84 86 95 158 116 145 135 61 81	512 598 546 607 607 640 526 674 545 631 549 635 581 616 531	111 112 133 115 109 105 103 103 105 106 133 112 109 110 114
Min
Max
Range
Median
Average

Chrome also sees some good boosts from 1.3.1 and manages to out awesome even Safari. When you look at the range as a percentage, it's all over the place. Sometimes the alert box would have a check box in it that would say something to the effect of "prevent this site from producing alert boxes" and I think those tended to take longer to generate. Maybe Chrome's doing other stuff behind the scenes. I'd need to do more specific research to figure this one out.

IE 7.0.5730.13

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	2990 2897 2867 2917 3016 2877 2877 2847 2907 2897 2877 2916 2897 2888 2848	1718 1738 1832 1718 1668 1708 1678 1678 1698 1709 1728 1738 1689 1688 1708	2538 2587 2557 2557 2577 2568 2567 2557 2548 2707 2548 2708 2558 2568 2568	1248 1219 1249 1248 1279 1249 1208 1228 1258 1229 1229 1239 1389 1229 1239
Min
Max
Range
Median
Average

Ugh! I'm in browser hell! What's up with this? 2.5 to 3 seconds to expand the list? Come on!

IE6

I don't know the exact version number here because of the way the Multiple IE installer works.

jQuery:	1.3.1		1.2.6
	Expand	Collapse	Expand	Collapse
Tests	10779 12577 13060 14085 15034 16123 17272 19549 19859 21121 23080 23249 24961 24770 25953	4935 5165 5774 5964 6314 7032 7152 8021 8231 8611 9082 9402 9892 9959 10349	11078 12617 13167 14785 15684 16876 19643 21740 20642 21947 22726 23925 36235 28810 29422	3899 4005 4356 4715 5105 5534 6453 7366 6842 7453 7801 8241 9700 9364 9849
Min
Max
Range
Median
Average

No. I was wrong before. This is browser hell. Notice how each test gains about 1 full second? I think that means we've got memory leaks. I closed and relaunched the browser between test cases to avoid skewing the results between jQuery versions.

I'll complete the number crunching and do a bit more in depth analysis tomorrow. It's 5 o'clock and I'm heading home for the day.

Content! Third-hand content, but still, content!

One of the people I encountered in all the discussion on higher ed in the web design industry is Kurt Schmidt. (ImTheSchmidt.com, isn't that one of the greatest domain names ever?)

Since then, I've been following his blog semi-daily. Earlier today he posted Make Your Site Faster - Or Else!, which is his take on info from Geeking with Greg. I started to reply to Kurt, but about the time I started my 3rd paragraph I figured I should just turn it into a blog post here. See what I mean about blogs and peer review?

The importance of speed and performance on the user experience is something that I'm currently struggling with. In the redesign I'm working on, I've made extensive use of jQuery. Everything runs super quick on all the Mac browsers. Chrome and Safari on Windows also blaze. Firefox on Windows is noticeably slower than the rest, but still tolerable. Who does that leave to drool on his shirt and crap his pants at the browser testing party?

Both IE6 and IE7 are intolerably slow. I've considered simply filtering out IE6 from all the js goodness. Since I've designed for progressive enhancement, everything will still function, just not the same way. But that still leaves IE7 users with a clunky and ungraceful experience. As much as I hate IE, I won't kid myself about it's market share.

The best solution from the users' point of view would be to test each jQuery implementation on the site separately and selectively enable/disable for IE based on performance in context. Some of the stuff I'm doing on the redesign works fine in IE. I know because I've been doing it on the current site for more than a year.

I also need to dig deeper into my jQuery-fu and make sure my calls are as efficient as possible. But some of it is straight up plug in functionality. I'm not too keen on rewriting other people's plug ins to increase efficiency. The next version of Firefox will probably incorporate ideas currently only available in Chrome's V8 js engine, making it's tolerable performance that much better. But that will probably just widen the gap between IE users and everyone else even more!

The next current version of jQuery (which I haven't updated to yet, oops) will also have a revamped selection engine that should boost performance across the board. But unless those changes impact performance in IE a hell of a lot more than everyone else, the gap will still be huge. It's the gap in the user experience that worries me.

What little testing I've done so far would seem to indicate that IE6 is bad at say 2000 milliseconds. IE7 is barely any better and a lot less consistent, with speeds ranging from 1500-2300ms. Firefox is around 700ms. Opera is 300-400ms. Safari is 200-300ms. Chrome is about 150ms (10 times better than IE7's best run!). I guess with those numbers, even if sizzle improves performance by 20% across the board, Chrome gains 30ms and IE7 gains at least 300ms. Maybe the gap will get smaller. But still, 1.2 seconds waiting for all the js to trigger is unacceptable from a user centered outlook. But why should I be punished as a designer and the 20% or so of our users who don't run IE miss out on those enhancements simply because IE can't get it's crap together? These sorts of situations are the only part of my job I hate. I'll stop there before this turns into a rant.

Well, now that I know the new version of jQuery is out, it's pointless to flap my jaw any more about this stuff until I've updated and retested. I've got work to do.

I've been neglecting all my imaginary readers

Things have been crazy. Professionally, academically, personally; you name it. Chaos defines me. Even more than usual. I've had plenty of blog worthy thoughts in the past month, but this is the first chance I've had to sit down and actually blog, you know, as a verb.

There was an explosion of discussion over at A List Apart on higher ed and web design. And by explosion, I mean I posted a lot and the discussion was meaningful to me. I'm sure there are articles with a hell of a lot more discussion.

The articles in question are Elevate Web Design at the University Level by Leslie Jensen-Inman and Brighter Horizons for Web Education by Aarron Walter. Both are worth checking out if you haven't already.

I've tried, and failed, to articulate what my heart tells me about this issue. I've posted discussions for both those articles. I've posted to the University Web Devlopers group on Ning. I've posted on various blogs where the discussion has spilled over. I've carried on conversations via email with a few folks. The closest I've come to getting it right is probably this comment in response to Leslie's article.

I'm trying to make the case in favor of a new type of “digital apprenticeship” in the industry. I think a formal education can work in our field. No. I know it can. My bachelors was a great model and actually follows a lot of the suggestions that people have come up with in these ongoing discussions. But the shelf life of that program as I experienced it was limited to about 5 years. A change in leadership is all it took to bring it back to business as usual (in other words, the kind of troubled program Leslie was targeting with her research). I don't think our industry can afford to wait a generation for the top level of campus politics to embrace the educational philosophies required to produce marketable graduates. For the next 20 years we'll see a lot of hit or miss programs, and some campuses will be more progressive in that respect than others. There are probably half a dozen or so solid programs in existence right now that have the proper institutional support to remain viable and stable. But I don't think students should be required to seek out places like MIT to find a decent web design program. And I'm sure less prestigious/well known campuses are capable of fielding the sort of program I'm talking about, but we're still faced with the problem of how do students find out about them? I mean, the only reason I found the program at TTU is because I started out there as an engineering student.

So while I appreciate the efforts that are being made to entice higher ed to get with the program, I don't think that's a sphere where we have any real influence. A lack of solid educational material hasn't been a problem in this industry for at least a decade. The fact that most academics will pay no mind to such material until it's formalized in a book or an accepted peer reviewed journal is a failing on their part (within the context of our industry), not on our part. In my mind, the spirit of peer review is alive and well on prominent blogs. And the transparent nature of the comment and debate system removes a lot of the politics that can infect trade journals. Movements like open course ware seem to indicate that some academics understand this and are forging new educational models. But until that kind of thinking can emerge from the underground, it only serves as a further means of exclusion.

We can't make higher ed listen to us. And we can't force HR departments to not require a degree for a given position. But I don't think we have to. Before we can effectively maneuver around the momentum and bureaucracy of the old schools of thought, we may be able to build a better model ourselves. After a couple of hiring cycles of employees with graphic design or computer science degrees who can't address the full range of skills a web position requires, HR might learn to value portfolios and experience a bit more, at least for these positions. If universities start noticing problems with retention and the ability of their graduates to compete in the market, they'll come to use for a solution.

Right now I feel like we're trying to cook a more enticing meal for a man who doesn't even realize he's hungry. But he's not just hungry, he's starving. Things like the Opera Web Standards Curriculum are great. But in the mind of most academics, it's foistware. We're pitching a solution to a problem that isn't even on the radar yet.

But, I have a bit more hope now than a few weeks ago. Leslie's article on ALA was largely preaching to the choir. But she's an academic herself (and a fellow Tennessean). The research that lead her to produce the ALA article is also the sort of research that academics may be willing to pay attention to. In fact, she caught the attention of the Chronicle of Higher Education. Maybe academia is more willing to listen than I've assumed. Still, I'll believe it when I see it.