Materialized Subquery Optimization

Greatest N per group is a hard problem. Everytime I comes up in my project at work (essentially a custom customer relation management system built in CodeIgniter) I end up wrestling with trying to do things the way that feels right but MySQL ain't havin' it. I usually end up having better luck saying to Hell with it and just rolling a Select N + 1 antipattern. Shipped beats perfect. I can beat myself up about sucking at database design later. At the moment I have problems to solve.

But today I discovered something interesting. I need to export the most recent values per account out of a couple different tables. Classic greatest N per group problem. I write a fairly ugly set of nested queries with the necessary ORDER BY and GROUP BY tomfoolery to get the behavior I want out of MySQL. I run it here on my local dev instance. It executes in a little under 4 minutes. Far from great, but for what should be a one-off across 200,000 accounts it's something I can live with. I switch over to production. Wait way longer than 4 seconds. As I approach a full minute I halt the query and scratch my head. I throw an EXPLAIN in front of the query in both environments.

The first thing I notice is while my local environment produces the query execution plan almost instantly (under 1ms) over in production the explain statement takes over 10 full seconds to run. Then I notice the output is also quite different. Locally I see the select_type MATERIALIZED. I'm familiar with materialized views even though MySQL doesn't support them. (At least not in the same way systems like PostGRES or Oracle do out of the box.) And I figured this difference in the output of the associated execution plans was an important clue.

A few weeks ago my early 2011 Macbook Pro gave up the ghost. So I'm running on new hardware. MySQL doesn't ship as part of MacOS so when I set up my stuff I (perhaps naively) just rolled with the most recent version. Let's just say our production server isn't quite so up to date.

The difference in query optimization between 5.5 and 5.6 is significant. Maybe I don't suck as database design so much afterall. Maybe I've just been wrestling with the query optimizer. (I probably still suck, just — ya know — less.)

The choice quote that is most relevant to the issues I was wrestling with today:

With Subquery Materialization, a subquery is executed once and the result is stored. In MySQL 5.5, the subquery would instead be executed for each row in the outer query. Our tests show that with a scale factor of 10, DBT-3 query 18 execution time drops from months to 68 seconds.

It's possible that my painfully slow but livable ~4 minute query in 5.6 would take months or even a year or more to run under 5.5. That's insane.

Adventures in Geek Parenting: Chore Wars

The Setup

We're trying to replace assigned chores and allowances with Chore Wars. Keep in mind that allowances are already earned on a per-chore basis under the old system. You only got paid for what you did and could document. (Just like real life!) The old system had a number of assigned chores equal to each child's age and they earned $1 per chore completed for the week. We were open to negotiating extra pay for extra work were either child really needing to generate some income. But I don't know that the option was ever exercised. Maybe once by the teenager?

For Chore Wars I'm moving away from assigned chores 'cos that doesn't fit the gamification aspects. Once it's assigned it becomes a true chore, not a game-like challenge. It's possible the kids will even start to compete with each other to maximize their own earning potential. It's also possible to cooperate, but it's up to them to work out who gets credit for what within the Chore Wars system.

Also, rather than an all or nothing for the week approach, you can claim the vast majority of chores (or Adventures as Chore Wars calls them) once per day. This also means that rather than being worth $1 if completed for the week each chore is worth $0.10-$0.25 with a few particularly challenging Adventures still worth $1 per (cleaning a bathroom for example). There's also a random element in the pay out, most adventures have a range for the "gold" you earn upon completion, which the system determines randomly.

My idea here is to encourage daily tidying rather than a weekly mad-dash. One side effect being if you sweep the office today and you also swept it yesterday it should be much easier and you can earn your chore pay out that much more quickly. But we have to hit a certain critical mass of actually doing stuff before the lesson clicks.

Every time I sit the kids down and explain how things are supposed to work, we see an explosion of activity. But this quickly falters after a day or two. They say it's because I hardly ever pay out. But I say I see no point in paying out when they've barely earned $1 for the week. This weekend I made my pitch again. And the teenager has really dove into the task. The 9 year old is still kinda draggin' ass. But maybe he'll get there eventually.

Current Experiment

I say all that just as back-story for my actual purpose in writing this. In addition to the in-game gold system I'm basing pay outs on, there's an XP system and a treasure system. I'm brainstorming on ways to leverage those for added incentive to actually do chores and log them daily.

Another goal here is to build family memories. So I'm trying to organize/structure evening family activities around that goal. Then offer incentives based on the Chore Wars system. Currently my thinking goes something like this.

Organized Family Activity Time

I'd like to set aside 7 to 9pm each night for family activity time. There are some assumptions already built into this. On work days I get home around 5:30. If we can get dinner served by 6:15 or so then we can have the dishes put away by 7 and dive into family activity time. This also assumes the house is at least tidy enough to be livable if not perfect and the kids have all their school work done. If this isn't the case we all pitch in to get there. Family activity time ends at 9 so if we don't have our collective act together then we all pay a price.

Each week night has an assigned activity. I haven't gone so far as to assign these to a specific night yet, but the 5 structured activities I can come up with are:

1. Board Game Night
2. Reading Aloud Night
3. Netflix / TV Night
4. Music / Jam Session Night
5. Video Game Night

The weekly leader in terms of XP in Chore Wars gets to pick the specific thing we do within that night's activity. Saturday and Sunday nights are freestyle, and the XP leader gets to pick both the type of activity as well as the specific thing we do that night. There's also random treasure items that we collect. I'm considering allowing those to be bid to hijack the normal XP leader process. Or maybe make them worth a temporary XP bonus for the purpose of calculating the winner for the day.

Risks

We've tried stuff like this in the past and it has never stuck. Essentially it only takes one of us to go all Negative Nancy on the idea to derail it for the rest of us. Also making a schedule and sticking to it is hard, you guys! The gamification elements are new and hopefully will help motivate us to actually stick with it. I have not budgeted in time for things like an exercise routine. So getting that in there piles the assumptions even higher than they currently are. Once we try to roll this out to production we may have to reduce family time to 90 or 60 minutes. But that will limit the choices available for some of the activities. Maybe that's part of the weekend freestyle. We can play longer games or watch longer movies or whatever. That gets a little easier to pull off 'cos the school work should still be done from Friday night. We'll see.

Future Development

If the incentives for experience points catch on there's added arbitrary challenges we can introduce. The Chore Wars system already allows you to claim partial or bonus XP for any given Adventure, the mechanics of which are left up to the users. So, for example, completing a task while whistling may be worth 25% bonus XP while walking backwards throughout the task may be worth a 50% bonus. Do both and claim double XP. Introducing such things at this point would just be adding complication to a system we're still not yet fully using. But it's good to have ideas to grow on as needed.

Chore Monster

For anyone thinking of trying something similar, there is an alternative to Chore Wars that seems a bit more modern and app based: Chore Monster.

Necessary Evil's power system as an adaptive framework for super heroic gaming in Savage Worlds

One of my favorite aspects of the core Savage Worlds rules is the adaptability. There's no "exotic weapon" training, just pick your weapon (including bare hands) and make a Fighting roll. If you wanna model being particularly good with a certain weapon or fighting style there are Edges to do that.

This carries over to the magic / power system. There's no Magic Missile or Acid Arrow. You get Bolt. That can model any magical power that deals damage to a single target at range. The rest is just window dressing. And that's handled through Trappings. One character can use a fire trapping while another uses an acid trapping. The end result is still some mofo is takin' 2D6 damage from across the room. Accept no substitutes.

If you want your Trappings to have an actual mechanical effect there are ways to add that to your games. That sort of thing is usually left up to the specific setting.

Necessary Evil introduces a whole new power system to handle the sort of street level comic book action not really possible under the core rules alone. Thankfully they carry over the adaptive powers + player chosen trappings. Not only does this allow players to model a wide variety of super power concepts from a fairly concise set of rules, it often also supplies multiple possible ways of getting your intended power, depending on how picky you are about how the specific game mechanics play out.

For example, Gru's freeze ray from Despicable Me. That is most certainly a Device, which qualifies it for a 1 Power Point discount for ever 5 Power Points invested into it. (This is a fairly universal Modifier in the Necessary Evil powers system.) But how exactly should we invest those Power Points?

We could just call it Stun with a cold trapping. As a Device that costs us a single Power Point! The Shaken status isn't a perfect fit for what we see in the movies but you get what you pay for.

Another option is a level of Attack: Ranged + Elemental Trick: Cold for a final cost of 4 Power Points after the discount for Device. This version does damage and "slows" the target by lowering their initiative order, still not exactly what we see in the movie. But it fits the general idea of a "freeze ray" very well.

Or we could go with Ensnare + Ranged Attack modifier. This gets us much closer to exactly what Gru's gun does for the same 4 Power Point cost as the previous option.

Finally, we could combine Paralysis and a level of Attack: Ranged via Paralysis's Extra Range modifier for a total cost of 7 Power Points after the discount for Devices. This is the most expensive option, but also gets us closest to what we see in the movies: targets physical covered in ice and completely unable to move for at least a few seconds.

Personally I wouldn't sweat the exact mechanics of it all and just go with the 1 point Stun with cold trapping option. I mean, Gru's gonna need every extra Power Point he can muster to pay for all those Minions. ;)

Pre-Generated Characters

I kinda cheated and went with the "Race as Character Class" methodology. Not that Savage Worlds has Character classes per se. I just mean I came up with a single concept per non-Human Race. So all my Wood Elves are archers, all my Sun Elves are mages, etc. For Humans I came up with 4 for a total of 10 basic character concepts. For each of those I have a common block of stats and a short list of Edges and Hindrances that would work well with the concept.

I located art to serve as character portraits for 2 male and 2 female versions of each concept. I made choices on Edges, Hindrances, and Gear purchases inspired by the character art. At the end of this process I had 40 pre-generated characters for my 8 players to choose from. For the most part they never noticed that they are really only dealing with slight variations on 10 themes.

I named each character using the WotC Character Name Generator and wrote up a brief blurb on the personalities and combat strategies for everyone (which my players ignore for the most part, but at least they've got the option of taking some guidance in role playing and combat situations).

Basic Character Creation Guidelines

• Attributes start at d4
• 5 Attribute points
• 15 Skill points
• Take up to 4 points of Hindrances
  
  • 2 Hindrance points buys an Edge or Attribute point
  • 2 Hindrance points buys a Skill point or doubles starting funds
• $1000 starting funds

The Character Concepts

Moon Elf Assassin

Stats

Agility D8

   Fighting D8

   Lockpicking 0

   Shooting D8

   Stealth D8

Smarts D6

   Healing 0

   Notice D6

   Taunt D6

Spirit D6

   Intimidation D6

   Persuasion 0

Strength D6

Vigor D6

---

Charisma -2

Pace 10

Parry 6

Toughness 5

Potential Edges

Acrobat, Assassin, Fleet Footed, Florentine, Rich, Filthy Rich, Strong Willed

Potential Hindrances

Anemic, Arrogant, Big Mouth, Mean, Overconfident, Stubborn, Ugly, Vengeful, Wanted

Gear

$1000

Powers

Wall Walker (racial ability)

Sun Elf Mage

Stats

Agility D6

   Fighting D8

   Lockpicking 0

   Shooting D4

   Stealth D4

Smarts D6

   Healing 0

   Notice D4

   Taunt 0

Spirit D8

   Intimidation D6

   Persuasion D6 Magic D10

Strength D4

Vigor D6

---

Charisma 0

Pace 6

Parry 6

Toughness 4

Potential Edges

Adept, Arcane Background, Arcane Resistance, Brawny, Common Bond, Nobel

Potential Hindrances

Anemic, Cautious, Clueless, Curious, Pacifist, Vow

Gear

$1000

Powers

Detect/Conceal Arcana (free action), Elemental Manipulation (free action) Flight (racial ability)

Pick 3 from: Armor, Blind, Bolt, Boost/Lower Trait, Burrow, Confusion, Deflection, Entangle, Environmental Protection, Fear, Healing, Light/Obscure, Smite, Speed, Stun, Succor, Summon Ally

Power Points 20

Wood Elf Marksman

Stats

Agility D8

   Fighting D8

   Lockpicking 0

   Shooting D8

   Stealth D8

Smarts D8

   Healing D8

   Notice D8

   Taunt 0

Spirit D4

   Intimidation 0

   Persuasion 0

Strength D4

Vigor D6

---

Charisma 0

Pace 6

Parry 6

Toughness 5

Potential Edges

Level Headed (racial edge), Alertness, Brawny, Danger Sense, Double Shot, Steady Hands,

Potential Hindrances

Code of Honor, Curious, Loyal, Quirk (Annoyingly chipper, Vegetarian, Collects interesting rocks, Always barefoot), Poverty

Gear

$1000

Powers

None

Hill Dwarf Treasure Hunter

Stats

Agility D6

   Fighting D6

   Lockpicking D8

   Shooting D4

   Stealth D4

Smarts D6

   Healing D4

   Notice D8

   Taunt D6

Spirit D6

   Intimidation D6

   Persuasion 0

Strength D6

Vigor D6

---

Charisma 0

Pace 5

Parry 5

Toughness 5

Potential Edges

Artifact Hunter (Racial Edge), Alertness (Racial Edge) Brawny, Danger Sense, Scavenger, Strong Willed

Potential Hindrances

Bloodthirsty, Cautious, Code of Honor, Loyal, Stubborn

Gear

$1000

Powers

None

Mountain Dwarf Merchant

Stats

Agility D6

   Fighting D8

   Lockpicking 0

   Shooting 0

   Stealth D4

Smarts D6

   Healing D10

   Notice D4

   Taunt D4

Spirit D8

   Intimidation D4

   Persuasion D8

Strength D6

Vigor D6

---

Charisma 2

Pace 5

Parry 6

Toughness 5

Potential Edges

Arcane Resistance, Improved Arcane Resistance, Attractive, Very Attractive,

Charismatic, Connections, Healer, Nobel,

Potential Hindrances

Cautious, Code of Honor, Hard of Hearing, Overconfident

Gear

$1000

Powers

None

Ice Dwarf Tank

Stats

Agility D6

   Fighting D12

   Lockpicking 0

   Shooting D6

   Stealth D4

Smarts D4

   Healing D4

   Notice D4

   Taunt D4

Spirit D4

   Intimidation D4

   Persuasion 0

Strength D8

Vigor D8

---

Charisma 0

Pace 5

Parry 8

Toughness 7 (+1 racial bonus)

Potential Edges

Brawny, Nerves of Steel, Sweep, Trademark Weapon

Potential Hindrances

All Thumbs, Arrogant, Bloodthirsty, Clueless, Heroic, Outsider, Poverty, Ugly

Gear

$1000

Powers

None

Human Martial Artist

Stats

Agility D8

   Fighting D12

   Lockpicking 0

   Shooting 0

   Stealth D8

Smarts D4

   Healing D4

   Notice D4

   Taunt 0

Spirit D6

   Intimidation 0

   Persuasion D6

Strength D6

Vigor D6

---

Charisma 0

Pace 6

Parry 8

Toughness 5

Potential Edges

Acrobat, Ambidextrous, Brawny, Extraction, First Strike, Florentine, Martial Artist, Quick, Sweep, Two-Fisted (If using Hindrance points to raise Strength, can also take Brawler)

Potential Hindrances

All Thumbs, Arrogant, Bad Eyes, Coe of Honor, Death Wish, Heroic, Loyal, Pacifist, Poverty, Wanted

Gear

$1000

Powers

None

Human Mage

Stats

Agility D4

   Fighting D6

   Lockpicking 0

   Shooting 0

   Stealth D4

Smarts D10

   Healing D6

   Notice D8

   Taunt D6 Magic D10

Spirit D6

   Intimidation 0

   Persuasion 0

Strength D4

Vigor D6

---

Charisma 0

Pace 6

Parry 0

Toughness 0

Potential Edges

Arcane Background Level Headed, New Power, Power Points, Wizard

Potential Hindrances

Anemic, Arrogant, Cautious, Curious, Mean, Overconfident, Phobia, Quirk

Gear

$1000

Powers

Detect/Conceal Arcana (free action), Elemental Manipulation (free action)

Pick 3 from: Armor, Blind, Bolt, Boost/Lower Trait, Burrow, Burst, Confusion, Darksight, Deflection, Entangle, Environmental Protection, Fear, Healing, Light/Obscure, Smite, Speed, Stun, Succor, Summon Ally

Power Points 10

Human Ranger

Stats

Agility D8

   Fighting D6

   Lockpicking D6

   Shooting D6

   Stealth D6

Smarts D6

   Healing D6

   Notice D6

   Taunt D6

Spirit D6

   Intimidation D6

   Persuasion D6

Strength D6

Vigor D6

---

Charisma 0

Pace 6

Parry 0

Toughness 0

Potential Edges

Alertness, Ambidextrous, Brawny, Danger Sense, Extraction, First Strike, Fleet Footed, Quick, Two-Fisted,

Potential Hindrances

Clueless, Code of Honor, Heroic, Illiterate, Loyal, Outsider, Phobia, Poverty, Quirk, Stubborn

Gear

$1000

Powers

None

Human Mercenary

Stats

Agility D8*

   Fighting D8

   Lockpicking D4

   Shooting D8

   Stealth D4

Smarts D6

   Healing D4

   Notice D6

   Taunt D4

Spirit D6

   Intimidation D4

   Persuasion D6

Strength D8*

Vigor D6

---

Charisma 0

Pace 6

Parry 0

Toughness 0

Potential Edges

Brawny, Florentine, Quick, Rich, Sweep

Potential Hindrances

Arrogant, Big Mouth, Code of Honor, Greedy, Heroic, Overconfident, Quirk, Stubborn

Gear

$1000

Powers

None

Character Sheets

I used the character art as a background image and styled the various stat blocks and normal character sheet info into semi-transparent blocks as a web page. Then I converted these to PDF using Firefox (which allows me to include background images when printing, including printing to PDF).

If I've properly figured out how to share stuff on Google Drive you can see an example character sheet. This happens to be the character played by my son, the only magic user chosen.

Custom Fantasy Races and Pre-Generated Characters

Rules for making your own Races can be found in the Fantasy Companion and I think maybe in the newest version of the core rules as well. The gist is you've got a list of modifiers both positive and negative. As long as each Race balances in terms of net "points" then the relative power level should be fairly even.

I (rightly) assumed these kids would be guided by visual cues. I also assumed among common fantasy creatures Dwarves and Elves would be the easiest to locate in terms of character art. Making 3 variations of each and tossing in standard Humans gives me 7 playable Races, which is plenty for my purposes.

Speaking of character art, Google image search is your friend. And I was pleased to discover Wizards of the Coast is kind enough to provide Art Galleries and Art & Map archives. Had I known that going in I could have made Savage versions of some more oddball races like Tieflings or Warforged. But I made the Races and pre-gens before I went hunting for character portraits.

I think in my previous entry I said the Races balanced out at +2. That's wrong. My Races are +2 above the norm, which is already +2, so everything balances out at +4 per the Race creation rules.

Humans

Basic, adaptable, fantasy Humans. They get 2 free Edges during character creation at +2 points each for a total of +4. Easy peasy.

Moon Elves

Natural born assassins based on the Drow but with none of the assumed evilness of that Race.

• Base Pace 10 (+2)
• Infravision (+1)
• Wall Walker (+1)
• D6 Agility (+2)
• Allergy: direct sunlight (-1)
• -2 Charisma, generally creepy (-1)

Sun Elves

Traditional "high elves" with a natural affinity for magic.

• +10 Power Points (+2)
• Flight: Innate magical levitation (+2)
• Elemental Tolerance: +4 to resist negative environmental effects (+2)
• -1 Toughness (-2)

Wood Elves

Legolas clones. (Hey, I never claimed originality was my goal here!)

• Keen Sight (+1)
• Free d6 in Shooting (+1)
• Level Headed Edge (+3)
• Quirk: Fatigue roll if sleeping in any "unnatural" environment (-1)

Hill Dwarves

Adventurers and treasure hunters. Indiana Jones wearing a ZZ Top beard.

• Artifact Hunter (+2)
• Free d6 Lockpicking (+2)
• Alertness (+2)
• Pace 5 (-1)
• Greedy (Minor) (-1)

Mountain Dwarves

Closer to traditional Dwarves, but not at all isolationist as Dwarves are sometimes portrayed and with a dash of the natural inclination towards merchantile trade usually found in Gnomes (or Ferengi).

• +2 Charisma (+2)
• Free d6 Spirit (+2)
• Free d6 Persuasion (+1)
• Pace 5 (-1)

Ice Dwarves

Ferral warriors of the north. I think I stopped short of giving them all horned helmets.

• Hardy (+3)
• +1 Toughness (+2)
• +4 resist cold (+1
• -4 to resist heat (-1)
• Pace 5 (-1)

I had planned to include my pre-generated character ideas in this same post but checking the preview I see this is already quite a lot info so I'll stick a fork in this one and save pre-gens for next time.

Gaming with Kids: Savage Worlds System Changes

If you lack a baseline understanding of the Savage World system let me direct you to the free downloadable Test Drive Rules.

Character creation rules remain the same, although that's a bit of a moot point 'cos I made pre-generated characters. The players never had to concern themselves with character generation rules.

Character Advancement is changed in some small but significant ways. Raising an Attribute is no longer a choice limited to once per Rank. It now happens automatically upon gaining a Rank, in addition to the normal advancement.

Bonus damage uses an extra d6 per Raise — because rolling fists fulls of dice is fun. This makes things potentially more deadly in a system that already allows for a lucky roll from the lowliest of characters to take down a legend. But it's slanted in favor of Wild Cards so I knowingly throw a lot of mooks at my players to give them lots of opportunities to land some cinematically extreme damage rolls.

I'll do a full entry on the Races I created for the campaign later. For now all you need to know is there are basic Humans and 3 types each of Elves and Dwarves. I used the Race creation rules from the Fantasy Companion and everything balances out at +2.

Attributes & Skills

Attributes remain unchanged. But I pared down the Skills to just 10.

• Agility
  
  • Fighting
  • Lockpicking
  • Shooting
  • Stealth
• Smarts
  
  • Healing
  • Magic (potentially)
  • Notice
  • Taunt
• Spirit
  
  • Intimidation
  • Magic (potentially)
  • Persuasion
• Strength
• Vigor

The Magic Skill can potentially fall under either the Smarts or Spirit Attributes. In essence it replaces both Spellcasting and Faith from the core rules. I designed this game for an audience of largely unknown (to me) home schooled children so I wanted to avoid potentially loaded or off-putting words. I assume "using magic" sounds less offensive than "casting spells" or expressing "faith" in some sort of pagan deity. :)

Excluded Skills

The following Skills are simply not used at all:

• Boating
• Driving
• Piloting
• Riding
• Gambling

Skill Substitutions

The following Skills use some basic substitutions should an actual in-story need arise for them:

• Swimming — Use Strength
• Throwing — Use Shooting
• Repair — Use Smarts or Lockpicking
• Streetwise — Use Notice or Persuasion
• Tracking — Use Notice
• Survival — Use Smarts
• Climbing — Use Strength

Gear

Savage Worlds provides a fairly basic array of gear, particularly weapons. Swords are essentially limited to short, long, katana, and 2-handed varieties. A scimitar is just another word for long sword as far as this system is concerned. But my players wanted more variety, and a special name for every possible way to curve a blade. So I dug up this wiki of Pathfinder weapons and essentially mapped 40 of the names there to existing weapon stats. If nothing else it provided an effective way to separate fools I mean my players from their money. I'll eventually make item cards for each of the weapons. If anyone is interested I'll throw them up on my Google Drive and share them here.

Everything is Physical

The lack of item cards for everything alluded to above shows that I haven't quite ironed out all the details on this one yet. But my goal is to minimize book keeping as much as possible by representing everything I can with physical objects. Item cards for gear. Coin shaped beads of various colors and sizes for in-game money. Gem shaped beads to represent experience points (the in-game explanation for this is 5 gems can be traded in for training at guild houses just like 5 experience points buys you an Advance in the normal rules). Bennies are represented by various beads that represent ancient artifacts. These can be "donated" to groups who are working to rediscover the lost technologies of magical equipment for something akin to "good karma" and we retcon stuff story-wise to say a character made any needed donations on his or her last visit to town.

One interesting ramification of this is that Bennies and experience points can effectively be sold, and — in rare cases — bought. This is potentially game-unbalancing but I don't expect my group of pre-teens to figure out the min-maxing stuff well enough to take advantage. I was expecting it to present some interesting opportunities for group dynamics and strategy. For example, character advancement is potentially more powerful for a magic user where as expensive gear is potentially better in the hands of a melee based character. So far the group hasn't run into anything like that. Although they do make a strong effort to keep treasure split fairly. I'll do a full post on how I'm using beads and physical stuff in-game at a later date.

Next Time…

I think that fairly well covers my changes to the core rules. Next time, whenever that happens to be, I'll go into details on the Races, pre-generated characters, and custom character sheets I made for the game.

Teach Your Kids to Game Week

I'm coming out of the geek closet. I'm a gamer. I don't mean video games. I'm talking table top role playing. I own over 100 dice with a number of sides different from 6!

So there's this thing called Teach Your Kids to Game Week that happens every year. Well, at least for the past 2. It's going on right now. And I decided to blog about what I'm doing in honor of the event.

I've ran a few one-shot games with my 2 kids in the past. A couple of generic fantasy games and some Star Wars based stuff. But ideally a gaming group should have more than 2 players if you hope to sustain a cohesive campaign.

We home school. So I thought it could be cool to get a group of home schooling kids together for a regularly scheduled game. If you find the idea of educational benefits of gaming with kids a bit hard to swallow, the Lewisville (TX, not KY) Public Library has an active program with a website offering several links worth checking out. Or read 5 Reasons to Play D&D which apply just as well to any other role playing game.

D&D is a specific system and kinda the flagship of the hobby so it gets used in a generic sense to refer to table top role playing as a whole. For what it's worth, Savage Worlds is my system of choice because it gives me enough options to run an interesting game without forcing me into a ton of prep work or leaving me feeling like I'm simulating a video game.

My wife has the social connections with the local home schoolers so we put out the call. I ended up with 8 players, counting my own 2 kids, tweens and early teens. We meet every other Friday from 1pm to 4pm. Norma and the other moms knit and drink tea during that time. We had our 4th session last Friday (November 30th) and I think we're finding our groove.

I hope to write up session reports for the 4 sessions we've had so far, although by the time I get that done we'll probably have our 5th session in the bag too. Before any of that I'll write up a few posts about:

• System Changes — Making the already "Fast! Furious! Fun!" Savage Worlds system more palatable for young players
  
• Creating Races & Characters — I wanted easy to find character art so I went with Humans and 3 flavors each of Elves and Dwarves. Then I made several pre-generated characters.
  
• Loot — Pretty much everything is a type of treasure, including XP and Bennies (I'll explain what those are when it matters). I bought various beads from Fire Mountain Gems to represent the various types of treasure.
  
• Accessories — I use a Chessex Battlemat and minis from whatever sources I can find with an eye towards budget.
  
• Adventures — I'm using a mix of One Sheets, Dungeon Crawl Classics, and Dungeon Delve with no sweeping Save the World metaplot
  

Stay tuned.

Higher Ed Institutional Blogging Server Usage Guidelines

This was one of those projects that I thought would take half a day and ended up taking a week. If I’m going to sink that much work into it, I might as well share what I’ve learned.

I started out looking for lists of higher ed blogs. I didn’t realize at the time that I’d be documenting the process so I didn’t save the lists I ended up using. Those playing along at home can Google it. Once I found my lists, I looked for blogs that lived on a .edu top level domain. Then I looked for any links that mentioned usage guidelines or policies, went to the home page and performed site searches, played around with the wording, and whatever else I could come up with to locate the sort of info I was hunting for. Sometimes the leads I found would be walled up inside an intranet, which is understandable if not a little disappointing. I ended up with around a dozen promising leads.

• From Zero to Blogging in Five Easy Steps on the Web Musings blog from agency.nd.edu
• Blogs @ Brandeis University
• Create A Blog at Stanford
• Requesting a Blog Account from Vanderbilt Medical Center
• Vanderbilt ITS Blog FAQ
• Blogs on Spark at Tufts (I love the whole idea of Spark by the way)
• General Recommendations for Social Media from Binghamton U
• Policies and Procedures for Smathers Libraries Blog at Uni of Florida
• Wordpress Help Blog at Oregon State U
• Berkely breaks out the P-word for Use of Social Media
• Yale University Web institutional server policies
• Weblogs at Harvard Law School » Terms of Use

From those links, I collected about 60 phrases from 7 different schools (in no particular order).

1. Brandeis
2. Stanford
3. Vanderbilt
4. Tufts
5. University of Florida
6. Yale
7. Harvard

I grouped those phrases under common themes and came up with these 9 common elements.

1. The stuff you publish here is visible to the entire internet and could be cached by search engines or archiving services for a very long time.
2. This server/service exists to support the mission of the institution, if your intended purpose is at odds with this then there are many freely available blogging services available to you elsewhere.
3. Content, including comment moderation, is your responsibility (and liability), not the institution’s.
4. Don’t do anything stupid or illegal.
5. We can re-purpose your content (from RSS feeds to marketing campaigns).
6. We can delete your stuff, up to and including your entire blog/account.
7. We provide a certain level of technical service/support but there are limits. (Varies widely from one institution to the next)
8. We can change our minds about any of this with or without notice.
9. Explanation of process for obtaining an account.

Not every school covered all those points. Not surprisingly, Harvard Law has one of the most thorough documents and they’ve been nice enough to license their terms under a Creative Commons Attribution-Share Alike 3.0 United States License. If you need to put together some guidelines in a hurry I recommend starting there and adjusting the wording to meet the needs of your particular institution.

Performance: Where Do We Draw the Line?

I endorse the majority of the content of the following presentation:

The Top 5 Mistakes of Massive CSS

Dissension!

It’s not even so much that I disagree with what she’s saying up there. I just apparently have a slightly different comfort level than she does when it comes to altering my work flow for the sake of performance gains. The only point I really have exceptions to is the idea of avoiding IDs in CSS.

I think IDs make perfect sense for major page areas like #footer or for elements that are truly unique like #logo. But I’m not gonna throw the baby out with the bathwater. She’s right that there are trade offs in using IDs and we should be aware of them so that we can make informed decisions.

It only took me 159 lines of CSS to find a use of an ID that makes me question the sanity of my former self. Applying a class to objects using the same design pattern would be more elegant in that situation. I probably have a dozen or so more of those I can clean up when I find the time (HA!).

Case in Point: Facebook

The primary case study in the above presentation is Facebook. Here’s some CSS from Facebook. I see around 20 instances of IDs used in selectors; stuff like #content and #pageFooter. Even her primary example doesn’s practice what she’s preaching with full on zealotry.

We are quick to forget that presentations like this are designed for oral communication. Nicole wants the people in that room with her to leave the conference with a few big ideas stuck in their minds. But it gets recorded and published as a video. That changes the medium. The medium is the message. So the audience expectations change. We non-attendees fault the speaker for presenting things in absolute terms. We expect video content to at least pay lip service to the fairness doctrine. But that’s really not fair.

Presentational Class Names

Nicole doesn’t spend a lot of time on this, but she mentions it in passing, so I’ll take a moment to get up on this particular soap box. My biggest problem with stuff like object oriented CSS and YUI Grids is presentational class names. That clashes with philosophies like the separation of content, style, and behavior or progressive enhancement that I’m not willing to sacrifice. And not just because Zeldman told me so.

I honestly believe these ideals bring value to the work I do above and beyond simple efficiency. That probably has something to do with the fact that I work in-house. The few client based projects I’ve done over the years tend to get junked up with presentational markup a couple hours after the client takes over anyway. If that was the only sort of work I ever did, I’d probably be in love with Blueprint. I can see the value of such frameworks for quickly producing a high fidelity prototype. But just like a mechanical engineer wouldn’t put a part produced by a 3-D printer into a final product, I wouldn’t put a CSS framework into production.

But that’s an informed decision I’ve made for myself. Your millage will vary. Just try to keep yourself up to date and consider the trade offs so that you can make your own informed decisions.

White Space Matters

So far today I’ve read two really good articles dealing with web analytics: Metrics and Dimensions and Reports — Oh My! from the Omniture blog and The Complete Beginner’s Guide to Web Analytics and Measurement from UX Booth.

But seeing these two articles open in adjacent tabs lead me to do some side by side comparisons. I noticed my gut reaction the the Omniture blog was “I don’t want to read all that!” The UX Booth article was considerably longer, but it didn’t illicit such a knee jerk reaction. I think the reason for this is white space. UX Booth gives their content a bit more room to breath.

Screen Shots

The following screen shots show the first 910-ish characters of content. I attempted to preserve the white space between this content and off screen elements. I’m using PNG-8 to avoid jpg compression making the text fuzzy. The downside is the photo from Oniture ends up looking a bit off. My fault, not theirs. Also, Blogger appears to be scaling these images to be the same width. The Omniture Blog screen shot is actually about 20px wider, for what that’s worth.

Omniture Blog

UX Booth

Counting Pixels

The textual content here are within 5 characters of each other, 912 characters for Omniture vs. 907 characters for UX Booth. Both spend some screen real estate on supplementary content, a photo or a table of contents. But I find the UX Booth content much more inviting. And by my count they use about a third more pixels to present their content.

What Do You Think?

Do you notice a difference anything like what I’m talking about? Or am I “putting the ‘anal’ in analytics” to borrow a phrase from the Omniture blog? Either way, the quality of the content in both cases is top notch. Definitely worth a read if you’re into that sort of thing.

Costs and Benefits of Website Performance Optimization

My last post covered the how, and supplied just enough explanation of the benefits to keep people interested. Here I will try to go in depth with my analysis of the costs and benefits of the website performance optimization steps I have taken this week.

Costs

I spent the better part of 4 days this week focusing on site performance enhancements. Since we launched the new site in November of last year, I may have spent as much as another 4 days on similar tasks. That time would have been spread out an hour here and there rather than devoting most of a work week to it. Let’s say in total I’ve spent 2 work weeks setting up these various tweaks. Setup is a one time cost.

Some of the costs stretch back to developing the redesign. It took a lot more time to separate content, presentation, and behavior than it would have to just export from Photoshop to Dreamweaver and push the results up the server. At the time I was thinking more in terms of quality than performance. But it would paint an incomplete picture to ignore that end of the equation. There are benefits above and beyond performance for that sort of attention to detail as well. The overall code footprint for the new site is about 4% that of the old site, and normal site maintenance takes a fraction of the time it once did. I’ve blogged on that topic before, although that entry is on the WordPress server which is currently down for maintenance. (3.0 upgrade! w00t!)

My previous entry included a step by step altered workflow for doing something as simple as adding a new icon to our footer. But to be honest most of the extra steps are trivial. A couple of extra “Save As” in DreamWeaver and minifying the results. These things take seconds or even a fraction of a second. That’s nothing compared to the normal edit/test/debug/publish cycle. Occasionally we’ll hit an image sprite edit that will be complex. But most of those will be as simple as sticking the new element at the bottom of the existing file and saving under a new version number. I’d be hard pressed to assign a number to this cost. As a percentage of my time, it’s probably somewhere in the single digits, low single digits at that.

Costs in a Nutshell

Two weeks up front to set up. Another two to four weeks per year in recurring maintenance costs. And the human capital investment to have the people around to pull it off. (That means pay enough to hire someone who knows what they are doing, or put forth the time and effort to educate yourself.)

Benefits

To gauge our progress, I ran tests on WebPageTest.org as I completed each website performance optimization milestone. Full results for each test are publicly available and linked from a previous post.

As a sample, I selected 3 pages. The home page represents only itself, but alone it accounts for 35% of our page views. The Current Students page represents our role based navigation pages and the A to Z index. These are only 6 such pages, but together they account for another 28% of our page views. The Library home page represents around 250 department/office home pages which taken together account for about 21% of our page views. I didn’t worry about the remaining 16% of our traffic since it’s distributed across about 1,000 other pages.

Definitions

Start Render

The time it takes for the browser to receive enough information to start rendering the page. In other words, time our users spend staring at a blank screen.

Load Time

Time spent loading all requests, including AJAX calls made after onLoad (when the status bar says “Done”) For us this includes loading Google Analytics.

Total Requests

The number of calls between the browser and the web server. Think of the server as something like a switch board operator answering around 7 calls per second, non stop. Each request is one of these calls. Our server must get really annoyed with us.

Total kb

Total kilobytes transferred throughout the entire process. This should translate directly to bandwidth cost savings.

Estimates & Assumptions

Based on Google Analytics data collected over the past 7 months, we average around 3,000,000 page views per year. Traffic calculations for the pages represented by our test sample are outlined above. Approximately 25% of our page views are from first time visitors, placing the other 75% into the repeat views category.

Home Page

First Visit

3,000,000 total page views times 35% represented in sample times 25% first time visitors = 262,500 per year.

Metric	Monday’s Test	Thursday’s Test	Difference	Percent Change	Yearly Savings
Start Render	1.089s	0.854s	-0.235s	21.579% decrease	0.7 days
Load Time	6.350s	4.250s	-2.1s	33.071% decrease	6.4 days
Total Requests	41	26	-15	36.585% decrease
Total kb	524	389	-135	25.763 % decrease

Repeat Visit

3,000,000 total page views times 35% represented in sample times 75% repeat visitors = 787,500 per year.

Metric	Monday’s Test	Thursday’s Test	Difference	Percent Change	Yearly Savings
Start Render	0.920s	0.523s	-0.397s	43.152 % decrease	3.6 days
Load Time	2.572s	1.462s	-1.110s	43.157 % decrease	10.1 days
Total Requests	38	4	-34	89.474 % decrease
total kb	59	52	-7	11.864 % decrease

Current Students Page

First Visit

3,000,000 total page views times 28% represented in sample times 25% first time visitors = 210,000 per year.

Metric	Monday’s Test	Thursday’s Test	Difference	Percent Change	Yearly Savings
Start Render	1.075s	0.785s	-0.290s	26.977 % decrease	0.7 days
Load Time	7.162s	4.342s	-2.820s	39.374 % decrease	6.9 days
Total Requests	38	26	-12	31.579 % decrease
Total kb	521	382	-139	26.679 % decrease

Repeat Visit

3,000,000 total page views times 28% represented in sample times 75% repeat visitors = 630,000 per year.

Metric	Monday’s Test	Thursday’s Test	Difference	Percent Change	Yearly Savings
Start Render	1.222s	0.519s	-0.703s	57.529 % decrease	5.1 days
Load Time	2.625s	1.475s	-1.15s	43.81 % decrease	8.4 days
Total Requests	35	4	-31	88.571 % decrease
Total kb	66	63	-3	4.545 % decrease

Library Home Page

First Visit

3,000,000 total page views times 21% represented in sample times 25% first time visitors = 157,500 per year.

Metric	Monday’s Test	Thursday’s Test	Difference	Percent Change	Yearly Savings
Start Render	1.379s	0.943s	-0.436s	31.617 % decrease	0.8 days
Load Time	4.994s	4.983s	-0.011s	0.22 % decrease	0.02 days
Total Requests	33	22	-11	33.333 % decrease
Total kb	470	369	-101	21.489 % decrease

Repeat Visit

3,000,000 total page views times 21% represented in sample times 75% repeat visitors = 472,500 per year.

Metric	Monday’s Test	Thursday’s Test	Difference	Percent Change	Yearly Savings
Start Render	1.249s	0.515s	-0.734s	58.767 % decrease	4.0 days
Load Time	2.266s	1.472s	-0.794s	35.04 % decrease	4.3 days
Total Requests	30	3	-27	90.000 % decrease
Total kb	70	80	+10	14.286 % increase

Benefits in a Nutshell

Over the next year, visitors to our site will spend about 15 fewer days staring at a blank screen waiting for our server to respond and about 36 fewer days waiting for our pages to completely load.

Guide to Intermediate Website Performance Optimization

If you need a primer on website performance optimization, check out these best practices from Google and these best practices from Yahoo.

The Easy Stuff

As long as we’re designing our sites with web standards some of these practices will already be familiar.

• External CSS and JS files
• Avoid CSS expressions and filters
• Lighter markup compared to table based layouts (fewer DOM elements)
• Specify a character set early
• Avoid empty tags and duplicate scripts

Other best practices are super easy to implement given clean, efficient, and (mostly) standard coding practices.

• Use <link> over @import
• Put CSS at the top, scripts at the bottom
• Combine external CSS and JS files

The Not-So-Easy Stuff

Image Optimization

Unfortunately this isn’t as simple as using “Save for Web” in Photoshop. To really squeeze every byte out of your image bandwidth without quality loss requires a little research into image formats and which one is appropriate for a given situation. Optimization tools like Smush.it are nice too. You can see Google’s suggestions and Yahoo’s suggestions but I’ll summarize my understanding below.

• JPG and PNG files handle almost everything quite well
• JPG for photos, PNG for everything else (icons, backgrounds, etc)
• PNG-8 does everything GIF can do except for animations
• The last browser support hurdle for PNG is alpha channel transparency, which is not possible with GIF anyway

Minification

Things start to get a bit more involved at this stage. Even “advanced” image optimization is a fire-and-forget sort of process. You do it right the first time and reap the benefits forever after. Minification effectively adds a compile step to our workflow and a lot of us webbies aren’t used to that sort of thing.

But it’s not as difficult as it may sound. I keep a non-minified version of code around to edit and test with, then minify it before putting it into production. I started out using Yahoo’s command line compressor. Since Google Page Speed is part of my standard testing routine, it’s trivial to save the minified version straight from its report once I’m satisfied with the changes.

Visitors to www.volstate.edu are served minified JavaScript (40% smaller) and CSS (19% smaller).

CSS Sprites

You can find a ton of resources on this topic with a quick Google search, but I prefer the classics. This technique can get quite confusing, and I recommend taking the time to really understand it before implementing it. That being said, we have many online sprite tools to make the job easier. I used SpriteMe and found it useful enough. Just make sure you create a test page to run it on that makes use of every image you plan to sprite, and you may want to adjust the suggested sprites to minimize white space.

The downside is a certain loss of agility when changing or adding images to the site. If we want to add LinkedIn to our social media icons in the footer of our site, we’ll have to edit the Icons sprite rather than just uploading a new icon to the server and linking it in separately. But there are also benefits.

• Fewer HTTP requests (reduced by about a third on our site)
• Sprites are often no bigger than the combined file size of the individual files, sometimes even smaller
• Commonly used images are effectively preloaded even if not used on the initial page visited

Be warned that sprites aren’t appropriate for everything. I tried to combine our PNG logo with the JPG RODP logo and the resulting sprite was 27k larger than the individual files. I think that was due to things like gradients and shadows in the RODP logo greatly increasing the number of colors required. I stuck with jpg for that.

Compression

Now we’re moving to server configuration. I’m lucky enough to have access to and admin rights on the server, so I could do this stuff myself. In another organization I might have to rely on the server admin group for this.

I used a combination of blogs and official documentation to set this up on our IIS server. It seems to be easier on Apache. Since I’ve never set it up on Apache myself I can’t vouch for the quality of the available documentation, but once again a quick Google search should turn up a wealth of information. On either platform, the important things to remember are:

• Static, plain text content (HTML, CSS, JavaScript, etc.) should be heavily compressed and the resulting file cached server side to save on CPU cycles
• Dynamic content (PHP, ASP, etc.) should be freshly compressed with each request — no caching — and not quite as heavily compressed as static content
• Compressing binary files such as images, PDFs, Word documents, etc. costs CPU cycles on the server with little to no benefit — in some cases compressing such files can even make them larger

Like image optimization, this is something you set up right the first time and reap the benefits with no repeating costs. It’s worth it to buddy up with the server admins to get this done. Compression has netted us around 70% reduction in transferred file size for both static and dynamic content and our hardware isn’t remotely taxed even when serving up around 10,000 visits per day.

Browser & Proxy Caching

I took this step only after some heavy thought. This involves more server configuration and has a dramatic impact on workflow. Essentially we set up the server to tell the browser to not even bother asking for new copies of static content until a given date or until after X time has passed. This content is loaded from the local cache. The benefit is nearly instant page renders for repeat visitors. Of course, that means if the file has changed since it was saved in the local cache, the user won’t see that change.

On the Server

After snooping around with HTTP Live Headers I determined that our server was already sending last-modified information, although our plain text documents seem to always use the create date. Could be because I edit on a Mac and upload to a Windows server. Binary files like images are fine. Setting up content expiration on IIS was pretty easy. Be mindful that all binary files will be cached. The browser won’t even request the file for however long you specify (or until after whatever date you specify). For us that was a problem because we have a lot of PDF newsletters that are updated by simply saving over the old file on the server. So rather than enabling content expiration on the entire site I did the CSS, JavaScript, and image directories. That covers that vast majority of the content that will benefit the most from client side caching.

Workflow Changes

To get users to updated content, we have to change the file name. That means we also have to change the markup, CSS, and/or JavaScript that references these files. HTML pages won’t be cached, so in the case of markup we just have the one extra step. Of course, if a link is hard coded into a few thousand pages, that one extra step can get pretty massive. Server side includes help us avoid those kinds of problems. CSS and JavaScript files themselves will be cached. That complicates things. To return to a previous example, let’s say we add a new social media icon to our footer. Here’s the process to do something like that.

1. Edit sprite image to incorporate the new icon
2. Save the updated sprite with a new file name — I recommend version numbering
3. Edit un-minified CSS to reference the new image and adjust other styles as appropirate
4. Save CSS with a new file name
5. Test and debug cycle
6. Minify CSS
7. Update <link> in universal header include to point to updated CSS file

That’s a bit more complex than uploading the new icon and editing the CSS file as needed. If I miss something in the test and debug cycle, I can’t just make a quick edit and call it a day. I have to start the whole process over again because anyone who hit the site in the meantime will be stuck with a cached copy of those files for the next 6 months. My CSS and JavaScript files now have a change log in the comments at the top, complete with file dependencies. These sorts of workflow changes are stuff we should be doing anyway but getting serious about website performance forces us to treat web development a bit more like software engineering.

Benefits

After incorporating all these changes, first time page loads for first time visitors are about 2 seconds faster. Subsequent page loads, even for first time visitors, are about 1 second faster. That means in a year’s time our users will spend about 1 month less time waiting for our pages to load. Plus the gains that brings to the user experience (a bit harder to quantify). Plus the associated reduction in bandwidth costs (a question for our IT department). Plus possibly gaining favor with search engines.

Optimizing Site Performance

Purpose & Goal

Right now I’m running some tests through webpagetest.org to establish baselines for site performance. This week I hope to generate some CSS sprites for our common images and set up gzip and xpiration headers on the server. Those are the last 3 big things pointed out by YSlow (actually I’m using Google Page Speed, but YSlow is cool too). Hopefully soon I'll be able to run a new set of tests to compare to the baselines I'm running now.

Sampling

I'm trying to get a representative sample of of pages to test. The home page gets the most traffic, followed by the Current Students page. Pride Online has the most popular departmental home page, but it's utilitarian and not representative of other pages. That moves us down to the home page for the library.

Those 3 pages realistically represent only about 1/6 of our total pages, but it’s the most heavily visited 1/6. I could add one more test case and increase my representativeness to the majority of our pages, but we’d see diminishing returns in terms of traffic (only about 16% of total site traffic). Also, the remaining 5/6 of pages are mostly text heavy and image lite. The benefits to those pages will come from optimizing the overall site template, which will be partially reflected in the 3 chosen test cases.

Thanks to Google Analytics for the ability to access that info easily.

The Site Performance widget in Google Webmaster Tools says we gained nearly half a second in our rendering time around June 6th. That half second was enough to move us up from around the 73rd percentile to the 82nd percentile. I have no idea why we saw this jump in overall site performance but I'm glad to have it. Maybe we can even hold onto it.

The Results Are (Mostly) In

The 3rd and final test is still running as I type this. I’ll link out to the results as they become available.

• Home Page
• Current Students Page
• Library

Step 1: CSS Sprites

I'm using SpriteMe and my own best judgement to create some CSS sprites for commonly used background images. I'm assuming SpriteMe already has something like Smush.it built in.

Results

• Home Page
• Current Students page
• Library Home page

Ok, maybe I need to smush my sprites after all. I've added about 60kb to each page load while freeing up about 15 http requests. Smush.it can’t find anything to optimize, but the image optimizer in YSlow can shave about 5k total from 2 of the sprites. SpriteMe makes some less than optimal choices, even occasionally introducing bugs. But like all software it has to make certain assumptions and those don’t always pan out. I need to improve upon the default sprites while I’m at it. With these newly optimized sprites, let’s look at the trade offs.

jQuery UI Theme Icons for Play Controls

The first sprite I made was actually already made for me thanks to the jQuery UI Theme Roller. I used the blue and red versions of the UI sprite to replace the play control icons under the featured stories on the home page. The bad news is that almost doubles raw file size. In exchange, I pre-load a lot of UI elements that we're not currently using anywhere else. But I plan to make more use of the jQuery UI in the future. So I think this is a worthwhile change.

	Baseline	With Sprites	Change
HTTP Requests	8	2	-6
Total Bytes	5,310	9,724	+4,414

Stripes Sprite

Both the header and the footer have background stripes that go all the way across the page. These were tiny, 1 pixel wide repeated images. But they were easy enough to combine.

	Baseline	With Sprites	Change
HTTP Requests	2	1	-1
Total Bytes	905	122	-783

Image & Badge Drop Shadows

Some day I’ll be able to pull this off with pure CSS but in the meantime I’m stuck with background images. The one for the standard large image gets used on most pages. The one for smaller images at the same aspect ratio and the badges gets used fairly often too. The other 2 don’t see as much use. Some pages don’t use any of these, but the majority of our high traffic pages do. The image produced by SpriteMe left extra white space to the right of the 16px icons screwing up the placement. I had to edit the sprite in Photoshop. After that Smush.it actually did a better job compressing the resulting image than YSlow.

	Baseline	With Sprites	Change
HTTP Requests	4	1	-3
Total Bytes	4,018	5,092	+1,074

Icons

The image produced by SpriteMe left extra white space to the right of the 16px icons screwing up the placement. I had to edit the sprite in Photoshop. After that Smush.it actually did a better job compressing the resulting image than YSlow.

	Baseline	With Sprites	Change
HTTP Requests	18	1	-17
Total Bytes	14,589	13,677	-912

Logos

I combined our logo with the RODP logo that’s featured on every page. But that only nets us 1 less HTTP request and as you’ll see below costs us significant bytes due to the differences between .jpg and .png image formats.

	Baseline	With Sprites	Change
HTTP Requests	2	1	-1
Total Bytes	12,270	39,876	+27,606

In a Nutshell

In a best case, a single page that references all these images (which is not terribly realistic), we’ve traded 28 fewer HTTP requests for 31,399 more bytes. That’s about 22 additional HTTP packets. Based on our test cases we’re realistically saving 12 – 15 HTTP requests per page. I know we’ll see a big pay off when we start effectively caching and gzipping this stuff, but I have to admit I’m disappointed with this first round of results.

Step 1.1: Learning from Mistakes

Ok, now I understand why the RODP logo was initially flagged as not to be included in a sprite due to being a jpg. I need to roll back the Logos sprite. I also need to alter the way we’re doing the random headers. We’ve been using something based on Dan Benjamin’s method from A List Apart but in terms of site performance it’s one of our slowest resources.

Results

• Home Page
• Current Students Page
• Library Home Page

We’re starting to see some minor pay off on the repeat viewing. Start render time on the current students page is now under 1 second. That’s pretty awesome, but we’ve still got work to do.

Step 2: GZip

I followed this guide as well as some official documentation from Microsoft. This online gzip test says we’re still not using compression but my web developer tool bar begs to differ.

Results

• Home Page
• Current Students Page
• Library Home Page

I just alphabetized the properties in each CSS declaration, primarily as a way to clean up my CSS and make it easier to manage. But Google’s Page Speed documentation points out the possibility of alphabetized CSS seeing more efficient gzip compression. Checking the alphabetized, minified, and gzipped version against the just minified and gzipped version with the Web Developer Toolbar (Information —> View Document Size) shows 8k either way. It doesn’t break it down into units any smaller than that. So maybe it saved me a few bytes, but not a kilobyte. So if we saw any gzip efficiency change from alphabetizing the CSS it’s less than 12.5%. If we assume rounding, then it’s less than 6.25%.

Step 3: Browser Cache

This one will be labor intensive. Before I put in the hours to set expiration dates and last modified dates for our various static resources I want to make sure they are properly optimized. We’ve covered CSS and JavaScript, but all the images that didn’t find their way into a sprite will need to be tested. I started this entry on Monday morning. It is now 1:30 on Wednesday. I this step will probably keep me busy the rest of today and a good chunk of tomorrow.

Ok, the clean up and optimization process has saved me about 10,000,000 bytes and 400 files. Gzip has been running for a full day and server’s resources don’t seem to be at all taxed. So far so good. Time to dive into Leverage Browser Caching.

I have a problem in that IIS reports the wrong last-modified date for certain files. Images seem fine. PHP files and other plain text files are not. Maybe I’m missing something in Dreamweaver. Our server also sends E-Tag hashes, which are newer than and in theory superior to last-modified headers. But they are also redundant together. E-Tags are apparently frowned upon when using multiple servers, such as a Content Delivery Network. All our content comes from just the one server so I don’t think it’s an issue with us. At worst we’s sending a single redundant HTTP response header.

So half the browser caching equation was essentially taken care of for us by IIS’s default settings. The other end of that equation is content expiration. I did not like that IIS just gave me a single check box to turn content expiration on. I wanted to be able to set it up by file extension. No such luck. I turned it on and did some testing with the Live HTTP Headers extension for Firefox. I was able to confirm that our php pages were not caching. Which is good. The last thing I want is someone to spend the next 6 months with an unchanging copy of our home page stuck in their browser cache. CSS and JavaScript files were caching, along with images. I already use version numbering on CSS, JavaScript, and image sprites. For the most part other images will have specific names anyway, so that’s good. Then I tested PDF files and did not like what I found.

I downloaded the PDF of our campus map and saw that the response headers told the browser to cache it for the next 6 months as well. We may need to update the map in the next 6 months. Maybe an office location changes to a new building or something. Worse still, many of our PDFs are newsletters. Monthly newsletters. Some offices have an archive listing years’ worth of issues while others just link to the most recent issue. In the latter case, I just save the new issue over the old issue on the server. With browsers caching these files, that wouldn’t work anymore. I’d have to give each issue a distinct name and adjust the HTML links to match, greatly complicating that process.

I rolled back those changes and tried setting up expiration on a directory by directory basis. So far I’ve only done /images/, /css/, and /js/. That covers the vast majority of our static files. Occasionally a directory off of root will have its own images directory. For now I’m leaving those alone. The handful of child directories inside /images/ also apparently did not did inherit the expiration settings of their parent folder.

The downside to all this experimenting is I have to bounce the IIS service for changes to take effect. That only takes about 8 seconds by my count, but I’t to think of someone submitting their final form for online orientation during that tiny window.

Page Speed still says we fail the browser cache check, but that’s due to the FEMA widget currently on our site. All resources being served from volstate.edu pass. So let’s run another batch of tests.

Results

• Home Page
• Current Students Page
• Library Home Page

Conclusion

This has been less a blog entry and more 4 days of notes to myself. Tomorrow I will try to recap the high points and present my data analysis.

Why I Became a Web Geek

Amber asked for web dev origin stories. Here’s mine.

In the late 1940s, in the parts of the world that mattered, the end of WWII and the atomic bomb had brought the philosophical shift we now label post-modernism into the cultural mainstream. But in rural Tennessee, just down the road from the house I grew up in, my grandmother was being loaded onto a wagon to make the 5-mile trip “in to town.” My dad was a breech birth. Home birth had taken a turn for the worse and they needed to get a doctor involved. 40 years later my dad bought about 42 acres of farm land he occasionally plows with a team of mules just for the fun of it. This is just a hobby. He earned his living by dismantling nuclear weapons in the same facility that enriched the uranium for Fat Man and Little Boy.

This is the culture that produced me. About 5 years after I was born, satellite dishes cropped up like kudzu and started piping in the culture of the greater global village. It was amazing to watch the communication/information explosion transform this area.

For example, I found the CBC fascinating. I didn’t understand all the politics involved, but I knew they had a station that broadcast in French because that was the primary language in Quebec and that they had a station out of the North West Territories that featured a lot of content relevant to the First Peoples population there. I didn’t really see that same kind of diversity in American media. We had a lot of forced tolerance, but as we tried to blur lines based on gender and race we drew a greater distinction along class lines. The prevailing message was small towns and small people didn’t matter.

That could be do in part to the overall wealth worship fetish America had in the 80s. Compare Bill Cosby’s Dr. Cliff Huxtable to Fat Albert and the Cosby Kids from the decade before. I could (and still do) identify with the Cosby Kids despite geographic and racial boundaries. The Huxtable kids lived in a different world and that had very little to do with race. The Cosby Kids were a rare example of the not-rich and not-famous among pop culture characters; and closer to my day to day reality than Larry, Darryl and Darryl.

Getting rural areas and poor areas connected to the mass media grid gave these marginalized populations a window to the world. Getting connected to the internet would give us a voice. I wanted to be part of that.

Also, growing up on a “farm” affected my work ethic. I don’t miss manual labor under the blazing Tennessee sun. It was 80 degrees at 8 am today, and it’s not even truly summer yet. I feel most fulfilled if I can point to something at the end of the day and tell myself “I made that. That didn’t exist this morning and now it does.” Luckily I discovered that digital artifacts scratch that psychological itch just as effectively as a freshly planted row of crops. And with significantly less sun burn. If only typing code burned calories as effectively.

On top of that, I’ve been extremely lucky to be in the right place at the right time both educationally and professionally. Everything fell into place and I feel like I’ve found the perfect fit to do what I am.

My dad’s birth story sounds like a scene from a John Ford movie and I earn a living building communication systems out of ASCII characters. That’s a huge jump in just 2 generations. I have 2 kids of my own. I can’t imagine what their adult lives will be like.

Why I Stay Busy

On Friday @kdc retweeted something from @eduguru that sent me to @educheckup where I watched several videos, including the one on Miami-Dade where he mentions one of the few things they did right was pausing the story rotation when you click on a particular number to view a particular story.

Hmm… Do we do that?

No. Crap!

Ok. We can fix this. Our front page story rotation is based on the jQuery Cycle plugin from @malsup. Let’s start there.

New version is out as of last month. Might as well update. Should probably update my version of jQuery while I’m at it. We were still running 1.3. Update both scripts, re-minify our local file (plug in collection and custom code), and make sure nothing breaks.

Background color change on pager links to indicate currently active slide no longer works. Fire up FireBug. Eventually figure out activeSlide class is being applied to the li tags rather than the a tags inside the list items. For our implementation, that means the buttons were being flagged as active rather than the actual pager links. Created a custom function for updateActivePagerLink and all was well again.

Pshew! We’re in the clear. Checking the docs for Cycle. Source code reveals some new stuff that hasn't made its way onto the web documentation yet. Luckily Mike does a good job of commenting his code. The functionality I need isn’t built in, but a couple of options get close, such as pause and pauseOnPagerHover. I didn’t have either of those enabled, and pause has been built-in forever. Learning about one usability gotcha lead me to discover another.

At this point the easy, built in stuff is fixed. I turn my attention back to the initial issue. Following the conventions set up by 2 existing options within the plugin (pauseOnPagerHover and onPagerEvent) I create a new option called pauseOnPagerEvent and insert 2 new lines of code: an if statement to check the state of the new option and code lifted from pauseOnPagerHover to actually trigger the pause. Amazingly, this works on the first try.

I’m still not happy with that solution because I think a more elegant way to do it would involve onPagerEvent. I just spent another hour trying to figure out how to write my own function for that callback and failed yet again.

Nashlantis

13 inches of rain so far with another 4 or 5 on the way.

The first photo here is me mowing our lawn a couple of summers ago. I'm a tiny little dot way back there, but you can see me if you squint. Take note of the big maple tree in the left of that image, the square of concrete to the right, and the row of 2 chestnut trees further back from that.

The 2nd photo is taken from right next to the big maple tree. The 3rd photo was taken from the concrete pad. Norma was ankle deep taking that photo, even with 5 inches of concrete between her and the ground. I'm standing about 3 paces beyond the edge of the concrete. You can see how deep it is out there.

The water has actually receded since we took those photos, about a foot by depth, several yards by horizontal progress. The concrete is no longer covered, for example. But there's another storm coming and the big one that dumped all that on top of us is technically still in the watershed the feeds into the lake near here. I'm hoping the worst is past us but I'm not going to count on that.

Trusting Second Guessing My Gut

Since December I've been blogging on the new multi-user WordPress server at work, but it's currently down. Once we get it back I need to figure out how to automate cross posting. This blog space belongs to me, Derek Pennycuff. The blog on the WordPress server technically belongs to the webmaster position at Vol State. If/When I leave Vol State this will still be mine. The other will not. Echos of the knowledge management course.

The Web as She Is Used

Anyway, right now I'm struggling with some stuff that my gut tells me is right but I have no evidence to support or refute it. I'll try to lay it out here and cite some specific examples.

Email Links

The default behavior of a mailto link is to launch your default email client. Here at work that means Outlook. On a default install of Windows it usually means Outlook Express. Our audience does not make heavy use of email. Most of them probably don't have a default email client. What email they use is probably web based. Some browsers let us set our web based email client of choice as the default, but how many users actually bother to do that? Personally, I tend to use Firefox's "copy email address" feature from the contextual menu when I right click a mailto link. But I've spent enough time working tech support to know that most people don't know much about advanced features such as right click menus.

So I prefer to use the actual email address as the link text for mailto links. In addition to all of the above, it's a nice visual cue to differentiate a mailto link from a standard hyperlink.

Telephone Numbers

Making that change introduced a new problem. The easiest way for me to make this change is to add a column at the end of the table and move the email addresses there. But that puts the telephone numbers in the middle of the table. A mailto link is designed to be used online. A telephone number is non-interactive.

(Although I confirmed over the weekend that a Nexus One phone does recognize a well formed phone number on a web page and makes it behave a bit like a mailto link. Among our users that's an edge case and I have no confirmation of how other mobile devices treat phone numbers. [Several people have pointed out this is de facto standard behavior on smart phones.])

My gut tells me most people will either dial the number from the screen or write it down. Either will involve a bit of eye darting on and off the screen. Have you ever had to look away from one specific table cell and then locate it again quickly when you look back? Working in Excel can cause this just as much as tabular data on a website. It's not easy. I think putting the phone number at the end of the table helps with this a bit. This brings us to a 3rd iteration of a staff listing page.

1. 
2. 
3. 

Aesthetics vs. Usability

The list above shows a progression of what my gut tells me is increasing usability. However, my eye disagrees. Aesthetically I see each version as a step down from the previous. And without any real usability data I'm at a loss as to how to resolve this cognitive dissonance. I'm dealing with opinion either way. And since so much of the usability issue hinges on action taken off-line, picking up a phone and making a call, I can't think of a way to A/B test it.

I need to standardize these pages. Inconsistency kills both the usability and the aesthetics. All I've got to go by is my own conflicting opinions, so if you've got any ideas please share them.

Update

A strong preference for option #2 is already apparent among the people who have chimed in on this issue. Apparently I'm on to something with the email address as link text for mailto links but I'm probably over thinking the telephone number thing. I'm going to continue to mull this over mentally so feel free to continue to comment or email me if you're reading this now and have anything to share. But in the interest of productivity I'm going to make option #2 the standard format and get to work. :)