Bucket full of Bottlenecks

For years I've seen a few super-guru performance engineers impress everyone by walking into the lab and with just 2-3 clicks through the code casually proclaim: “Sure, you just need to tweak the buffer setting in the max_connections_kernel_init.xml file to match the incoming parameters for the extension listing sequencer” with an “everybody-knows-that-one” attitude.  I figured my dream of becoming that cool could never happen. Or could it?

At one time or another in my own career as a performance engineer and tester, my self-perception was that I needed to be the guy who knows a huge list of all bottlenecks, and that would be my secret to success. But I've come to see there are 2 major flaws with this objective:

1)  there are a whole lotta of bottlenecks out there, and
2)  the list of bottlenecks is always changing and evolving

What I’ve learned from spending my working life encountering massive techno-churn, new platforms, and new usage patterns is that knowing where bottlenecks come from is far more important than knowing the bottlenecks after they exist. Understanding the engineering patterns that result in a performance bottleneck means that you can help to prevent performance bottlenecks from getting into the system in the first place.

In one of the performance engineering workshops I teach (“Performance Engineering for Product Owners”), I explain the importance of writing the specific performance requirements into the user stories or specifications.  Ideally, this means as the user story is evolving to become dev-ready there’s an opportunity to engineer performance into the thinking of the design, and that will impact how a developer responds to writing code or constructing the system to deliver on that story. And product owners can get ahead of bottlenecks, too.

So, the next time you are tirelessly tracking down the root cause of a performance bottleneck and some super-guru performance engineer comes walking in to pull a "miracle" solution from their bucket full of bottlenecks – remember the relative longevity of that fix, and the limits on such specific solutions.