Your App is Slow: Speed Is a Feature

This is a multi-part series:

1. boiling the frog, 2. speed is a feature, 3. that darn demo dataset, 4. measure early, measure often

Most development is feature driven. A developer is on the line to complete a user story or functional requirement, and even if the application gets a little slower, she’d rather have a demo to show during sprint review, instead of watching every one else’s demo.

You’ve got that guy on your team who every few months just can’t stand it anymore and disappears into the boiler room for a week. Fixing database query plans. Adding a cache where there wasn’t one before. Adding parallelism to the calculation step. Whatever it takes. And it’s usually right before a big release, that magical period when your customers show up and tell you the app is slow. (You forgot, you were the boiling frog.) And what happens when your well-meaning developers re-plumb critical parts of the system right before a release? It might get faster, but it probably doesn’t get more stable.

The mistake was made when performance didn’t show up in the user stories. “As a user, I’d like to see my list of outstanding invoices” isn’t enough. It should end “… in less than half a second”. Or if something has gotten slow despite everyone’s best intentions, then speeding it up needs to be its own story.

Slow performance is a cruel form of technical debt. The stop-gap cures are often as bad or worse than the disease. We worked with a team whose queries were getting so slow and returning so much data that they were crashing the browser. The quick solution? Install “circuit-breakers” to limit the number of results that could be returned in a query. Too many outstanding invoices? You’re not allowed to see them at all; make a more refined query. We’ve worked with teams that installed caches and forgot about cache coherency, meaning the app would show inconsistent results in different places depending on whether the cache had been thought about or not.

Performance needed to be a feature from the start. Making it a feature usually means defining it a little more carefully, which is the subject of part 4: measure it carefully. But first a detour into a classic place crappy performance hides: bad data.