It's time for PerfBytes!

What is PerfBytes?  It's a live internet radio show about performance started by myself and James Pulley over a series of early-morning breakfast meetings at the IHOP in Beaverton, Oregon.  While consuming copious amounts of pancakes, omeletes (made-to-order) we engaged in a serious debate about: what's the difference between load test and stress test? It took about an hour and a half to agree that we hadn't come to an agreement on how to answer that question, so we scheduled another breakfast meeting for the next cold and rainy Tuesday morning.  And it wasn't just James and I at the table.  We had not just a few of the top minds in performance and load testing, including Rex Black, Steve Oulighan, Howard Chorney and Carlos Chidiac (just to name a few). This group of performance gurus kept meeting for breakfast (although Carlos was usually late every time!) hashing over the contemporary practices, learnings, techniques, pitfalls and tools used by load testers. It was actually the most engaging part of the rare and precious time we spent together in the Spring of 2012.

But, let me tell you this show "PerfBytes" and you might still be wondering wondering how two highly-entrenched performance geeks like us could end up having an internet radio show. The truth is, we just created one. No hassle - thanks to the folks at BlogTalkRadio.com - all we had to do was signup and voila! - PerfBytes was born. 

The name: PerfBytes.  What's in a name?  Not unobviously the word "perf" stands for performance - including testing, engineering, tuning, monitoring and all that good stuff we do professionally. The "bytes" part was more important because we wanted to share information about performance in digestible units of information, akin to a short-stack of blueberry pancakes. The format was also inspired from another hit TV show "Good Eats" by Alton Brown. We thought - why not have a show like "Good Eats" only on the subject of performance testing and load testing? I mean, really - who wouldn't love that show?!?!  

PerfBytes bi-weekly listeners and callers will learn:

•      to increase their awareness of performance subjects
•      about the Science! behind performance
•      to implement performance practices in tools
•      the real stories from industry practitioners, vendors and consultants
•      to translate practices into value for their organizations

You can find out more by visiting the PerfBytes website at www.perfbytes.com!

Or how about following us on Twitter @PerfBytes?

And of course every 2 weeks, call us during the live show http://www.blogtalkradio.com/perfbytes!

What is Realistic Performance Testing?

Today I received a comment from Dzmitry on an article we posted over at www.softwaretestpro.com, asking the following question:

Dzmitry Kashlach
9/10/12 5:31:25 AM

Thank you, Mark, for the article. I agree, that in many cases performance testing engineer cannot choose right tool because of limits in budget, company's policy and so on and so forth. BTW, do you agree, that testing in cloud is more realistic and gives us more opportunities than testing in isolated test-lab, as it is described in the following article?(http://blazemeter.com/blog/top-ten-reasons-run-load-and-performance-testing-cloud)

Essentially, I agree - testing in the real world (e.g. production, or the cloud, or the internet, or in real end-user situations) is more "realistic" than *not* testing in those contexts.  But let's be very clear about the use of the word "realistic" in regards to performance testing.

Let's start by defining an unrealistic test.  An unrealistic test could refer to any test case or condition that absolutely would not happen in the real world.  Example: testing 5,000 concurrent users hitting a system that will be installed for a small workgroup team of only 10 people maximum.  The truth is - this is still a valid test if it is based on a story hypothesis like:

“In order to determine the maximum capacity of the system beyond a team of 10 end users, as the Product Owner and System Operator, the system should not handle more than 10 users concurrently."

You might assume that running a test at 5,000 would obviously fail, right?  But what if it doesn't fail?  You see, unrealistic tests can lead you to new understandings and capacity for the performance of the system – even if that test was not conducted in the “real world” configuration. 

By contrast, a realistic test could refer to any test case or condition that plausibly could happen in the real world.  Example: during the peak rush time all 10 expected users on the system submit 5,000 records per hour.  The performance story hypothesis being:

“In order to successfully process all transactions during peak times, as the Customer and End User the system should handle more than 5,000 records per hour without failing or crashing."

This is a realistic test and it can be setup and tested in the test lab sufficiently.  In this test even if you run the user load from outside the firewall against production it doesn't make the test condition any more realistic than before.  It is valid to test performance in the lab if you are very smart about the physical configuration of the transactional paths employed in your test.  You can learn about performance and scalability, you can find bottlenecks and defects.  It is valid, realistic testing. 

But often times we aren't rich enough to setup and configure the test lab to be just like the real world.  So, it is also valid to test performance in "the real world" be it in production, or the cloud, or at the install site where the entire system is configured exactly as it will be for the customer and end-user.  Be aware that the risks of something going wrong or bad associated with testing in production are higher because you might not take the time to analyze and think through all of the implications of the transactions you are throwing at the production system.  Some of those risks include managing privacy and security in the test data, preventing external downstream processing, backing-out data and changes from test traffic, setting-up a kill switch, blocking real production use during the test time - these are just a few.

By contrast, when we do take the time to set up an internal test lab we usually perceive that as an unusual and extraordinary (and more costly) initiative.  It demands more attention to detail and configuration.  Because it boosts our attention and consciousness to those details - as humans, we are more likely to be cautious and explicit in our efforts.  That helps to reduce risk.

Quality Investors: NASA vs Wall Street

You know something’s wrong when a Mars spacecraft lands without any software glitches or crashes, while at the same time here on Earth a company’s financial trading system fails disastrously costing nearly half-a-billion dollars.  I expected the opposite, didn’t you?  These two separate events demonstrate the brutally obvious outcomes of an organization’s level of commitment to quality.  And the differences couldn’t be more stark.

                  

The Mars landing (a.k.a. the 7 minutes of terror) required about 500,000 lines of code to execute with perfect timing, slowing Curiosity’s descent from 13,000 miles per hour to less than three miles per hour in a hovering maneuver to touch down on the surface of Mars.  This system’s execution flawlessly triggered 76 explosive charges without damaging the sensitive electronics of Curiosity’s scientific lab equipment.

Alternatively, the high-frequency low-latency trading software used by Knight Capital Group might involve exponentially more code when you consider all of the end-to-end components that comprise the system’s architecture.  Knight’s software is responsible for routing trades across the various exchanges with extremely quick response times.  There are typically no explosive charges in Knight’s trading system; that is if you leave out the $440 million charge to rescue the company from a financial crash.

With similar levels of system complexity both of these organizations had a common goal: avoiding a crash.  But the difference between them is their attitude about quality.  Some think that quality is just a characteristic that must be technically validated.  But there aren’t just knobs to twist, buttons to press and formulas to compute for quality.  Quality is not just a technical aspect of software.  Quality is the result of human attention to detail and critical thinking about the outcomes and impacts of our actions.  An attitude that supports high quality comes from people who are engaged in and supported in the pursuit of excellence in their work.

The commitment to quality software starts with the attitudes of the people at the very top, through investment in resources and support for culture that rewards the individual’s behavior to prioritize and ensure quality outcomes.  There’s an expectation that if individuals are given time and respect to deliver high quality, they will be committed to doing so. As an example, with regard to the engineers working on the Curiosity program NASA Administrator Charles Bolden was quoted as saying: "This is an amazing achievement, made possible by a team of scientists and engineers from around the world and led by the extraordinary men and women of NASA and our Jet Propulsion Laboratory."  Contrast that with this statement from Knight’s CEO Thomas Joyce:  "You cannot keep people from doing stupid things…that is what happens when you have a culture of risk."  Joyce’s company lost nearly 70% of its value in 2 days.

(PHOTO CREDIT: NASA JPL/CALTECH)

The “culture of risk” Joyce refers to is about taking financial risks for gain, which includes hedging against poor quality financial decisions.  This attitude of risk-taking is counter-productive to quality.  It can permeate the culture of a company even when it comes to decisions about technology.  You can see this in the form of extreme cost-cutting on tools and expertise, a lack of training and professional development and also in the prioritization of project completion over quality.  When it comes to technology systems the most effective approach to hedging risk is to test thoroughly everything involved with the system and to not simply rationalize impending failure.

As evidence to their attitude towards quality, the engineers at NASA know how to manage technical risk.

They test.

---
Sources: Bloomberg.com & Bloomberg News, USA Today, NASA JPL & MSL, Eric Pitsenbarger

UPDATE:  this article was re-posted in modified form on the STPCON website as “Quality Risk Takers: NASA vs. Wall Street” (which is totally cool!)

Performance Tool Selection and Hyperspace

For most performance testers, our first few years of work in the industry were focused on learning the basics of automated testing tools and how to conduct and develop load tests. But what I used to hear often from testers is that even though they developed these skills, they were not included in the decision about what testing tool they must learn first -- the performance tooling solution was usually pre-determined. This may have been due to a corporate licensing decision or the specific market domination of one particular tool.

Which means that as a tester, someone above you in your organization was traditionally engaged (and probably completely overwhelmed) by the task of selecting the right tools for the job you would be performing. The reason usually given for this is that the majority of performance testing solutions have been almost exclusively commercial products, some of which required formal evaluation and validation before your company would be willing to make an investment in the vendor’s solution.

Ah, how times have changed.

Today we have a much larger array of performance testing solutions with a range of costs -- from completely free open source tools, to cloud-based tools with incremental operational licensing, and sophisticated enterprise-class tools. With more choices (in different pricing classes) the perception is that you have more freedom to choose and now may use many different tools to get the job done right. 

There has also been an increased focus in recent years on automation. Although automation is not a new idea for performance testing, it is becoming a dominant topic of conversation, so much so that the expectation now is that performance testers have a comprehensive understanding of all tools on the market, can conduct a full feature set comparison, and really know the benefit of one tool over another.

And today performance testers can be engaged much more directly with vendors than in years past; this can be challenging for an engineer with no practical experience in vendor relationship management. There are often complex policies and processes when it comes to working with your management and procurement departments. Even if you are selecting free or open-source tools, navigating all this can become a full time job!

So nowadays you are expected, as a performance tester, to know everything about all the tools, to make an objective choice for the best one to use, to decipher all of the licensing constructs, to compute and justify the costs/benefit calculations and then manage the implementation for your entire team.  With little time or education on the subject, you typically end up just “shooting from the hip” when selecting performance tools.  Thus, typically, you often shoot yourself in the foot.

And this is why Scott Barber and I have put together an STP Online Summit virtual conference series, to help you learn how to do all this. The “Survey of Performance Testing Tools” (August 21st-23rd) is a great opportunity to learn how to conduct tool evaluation and vendor management in less than three days. During the summit you will be guided by Scott and I through interviews with actual testing tool vendors, engage them with your questions, and learn how to bring all the information back together at the end to drive your decision.  It’s a crash-course on tool surveying and necessary navigation skills that you will not find anywhere else.

In closing, what does this have to do with Hyperspace?  Well, there is a great scene in Star Wars Episode IV where Han Solo turns to the inexperienced Luke Skywalker and states:

"Traveling through hyperspace ain't like dusting crops, boy! Without precise calculations we could fly right through a star or bounce too close to a supernova, and that'd end your trip real quick, wouldn't it?”

In an alternate universe (and alternate galaxy…far, far away) where Luke is a fledgling performance tester, Han Solo might have instead given the strict warning:

"Selecting the proper performance testing tool ain't like dusting crops, boy! Without precise calculations from the tools we could fly right past a false positive or come too close to overloading the wrong system component, and that'd end your testing career real quick, wouldn't it?”

Testing as an essential dependency for DevOps

This post is a response to and a comment on my colleague Paul Peissner’s blog: Expanding QA and Tester Role in the DevOps IT Era?  One of Paul’s primary points in that blog post is:

“To enable business agility, IT needs to grow QA's value, role and function, this will help with better job estimating but more importantly it will drive the right long-term response by IT when surprises hit.”

In my observation, the mission and purpose for QA professionals is far more important in the DevOps model -- QA professionals strive to proactively improve the probability of delivering a higher quality product, a more innovative product. Also, I want to correctly label DevOps as not a cultural movement; DevOps is simply a change in approach or method to delivering a product.
Part of the impetus to adopt DevOps is the pressure on business leaders to be more profitable and more responsive to market demand, through innovation. Thanks to IDEO and David Kelley, those business leaders have been sold on an idea of innovation as a product, even though I think most of them actually have a preference for profitability over innovation. But we all might agree that improving the quality of our innovative thinking is an integral part of how we build successful products. From my experience as both a tester and a product manager, I know the benefits of improved quality in products and the correlated reduction in risk to failure of the product.  And I know the importance and pressure to deliver high quality innovations and differentiation, if you hope avoid failure of the idea.
What does this have to do with DevOps?  To become more innovative, the Web Operations sub-team in IT Ops started picking up more and more “innovative” skills and recruiting web developers and super-administrators to merge onto the same team so they could move faster and be more innovative. Just being a system administrator (who keeps the monthly release train wheels turning and the gears oiled) isn’t good enough enough anymore – the business needs “more responsiveness” from you.  And just being a developer engineer waiting for requirements and specifications isn’t responsive enough either – the business needs “more innovative problem solving” from you. What happens when you start to ask a bunch of developers and system administrators to stop focusing on the product development machinery and start contributing to innovation, ideas, and responsiveness? The wheels can come off.
If this self-proclaimed DevOps movement is to provide any positive change for our industry and to the businesses we serve, then I think it’s important to recognize three underlying origins for DevOps and reflect on how QA can get involved:

Vendor Agnosticism: as a web site or system administrator being pressured to innovate working with new ideas, tools, and stuff – you might turn first to the vendors you already work with. In some cases, if there is resistance to DevOps “cultural immaturity” and a lack of DIY skill some folks might cling too tightly to their vendors as a safe-haven. As a tester, you might log a bug against an incomplete process or limitation of benefit caused by vendor-bias or preference. If the quality of your product is going to suffer because there was a top-down mandate to use ‘Vendor Product X’ or ‘Vendor Product Y’ then QA should log a bug against that biased decision or perceived mandate.

Organic Collaboration: as a tester who tries to conserve the practices that successfully find bugs and secure the high-quality standard for product development, it’s natural to question anything new and “innovative” (quotes emphasized). The idea of new can also mean higher risk of the unknown impacts on quality. This means testers are perceived as conservatives or “keepers of the old” while this new DevOps-y adoption may be perceived as risky or “leading the wrong way.” No matter the perceptions we have, we need to work together to deliver something. If the quality of the product is going to suffer because certain groups "just don't get along" then QA should log a bug against the people who aren't collaborating. Yes, it's a defect to be willfully uncooperative.

Rationalization of Failure: as a business leader investing in technology as a future for the corporation, failure is definitely not preferred -- and as shareholders, we really don’t like failure either. However, failure and recovery is now becoming an accepted and even preferred mode for operating. But if the quality of the product is at risk because we are over-accepting failure in any part of the process from inception, definition, stories, coding, automation, builds, performance, security, tools - then we should log a bug against the policy (or lack of a policy) around failure.

I agree with Paul that with the adoption of DevOps practices, testers have new kinds of bugs to find, new bugs around policies on failure, organizational cooperation and unethical vendor biases.
The opportunity for testers is to expand our scope to include both products and ideas. We can bring a professional rigor to testing ideas and innovations prior to their instantiation and forecasting into product revenue or application. We can re-emphasize our role as testers of ideas and testers of innovation, testers of risk and value.
Unfortunately, what I often see is that testers are regarded as just technical bug hunters, as servants to the code or system. This treatment stems from a limited consideration for the mind's capability to engage holistically in the creative process for the very innovation we seek. In this new world where QA is a part of DevOps, then, if you limit the role of the tester, you are inherently limiting the benefits, the innovations, and the responsiveness of DevOps.

Goin’ Mobile with an STP Online Summit

Pete Townshend wrote Goin’ Mobile (for the Who’s 'Next' album) back about the time I was born, but I've been hearing it my head lately. One lyric from the song is: “When I'm drivin' free, the world's my home” -- a reference to travelling in an RV (in the UK they might call it a caravan), your own house on wheels. The modern re-interpretation of this gypsy fad from the 1970’s is paralleled in how we now take our electronic “homes” with us on the road. Our computing experiences are increasingly goin’ mobile.

This past week I supported and presented at the Software Test Professional’s STP Online Summit: Insights on Deploying a Mobile Testing Strategy. If you’ve never attended an online STP Summit, you really should. This is different from casual webinars or online lunch-n-learns -- it’s more like a virtual one-on-one training, with dedicated panel of professionals, highly interactive discussions in the “STP Crew” site, and access to evolving content as the summit proceeds.

On Tuesday, Matt Johnston from UTest gave an introduction to SOLOMO (social, local, mobile) and overview of that market landscape, and presented a robust understanding of the pressure that testers are under to adapt their skills. The main thing that I heard was that marketing professionals don’t make good testers, even though they are increasingly becoming the business sponsors for IT projects when it comes to mobile applications. These SOLOMO business sponsors are pushing developers and testers so hard and so fast to deliver apps that testing can easily get overlooked.  Matt encouraged testers to be proactive and be protective about testing mobile.

Karen Johnson reviewed in expansive detail how the entire environment for a mobile application is completely different from the normal applications we have tested historically.  This is especially true for how we interact with mobile applications, as UX designs have a different context for usage and interaction.  She covered some great suggested books like Theresa Neil’s “Mobile Design Pattern Gallery” as a way to upgrade your testing skills to the latest mobile UX techniques.

On Wednesday Dan Bartow from SOASTA offered up something different from his usual performance presentations. This time the focus was on SOASTA's new mobile functional automation solution -- which is very unobtrusive to the mobile application running live on the device.  He showed a live demo connecting SOASTA’s CloudTest Lite to the mobile device over wifi and made it look easy!  Another key takeaway from Dan’s presentation was that automation tools are all scrambling to support more and more gestures (touching the device screen/interface) in an easy way, which reminded me of old GUI automation (which used to be the race to support both analog recording and context-sensitive, object-based record and replay.)  Mobile touch-sensitive interfaces have brought a whole new challenge to automation tools.

I presented with Srikanth Gullapalli from Cognizant on Wednesday as well, and we dove as deeply as we could into performance testing, diagnostics and profiling for mobile applications…in less than an hour. Our presentation focused on the application’s performance optimization for the local device platform, which needs to be conducted together with load testing of the back-end server and network load testing.

Dan Cornell jumped into mobile application security testing, and Dan’s presentation was the most comprehensive presentation I’ve ever encountered on the subject – amazing that he got through all of it in just one hour. One thing I got from the presentation is that with the increased use of mobile applications comes an increase in mobile data. Given that any mobile device is physically easier to steal and compromise (typically unlocked/insecure), the mobile applications we use absolutely should be more secure than desktops. He covered vulnerabilities at the service layer in use by mobile applications requiring a combination of static, dynamic and forensic analysis and testing.

On Thursday there were presentations from Peter Dimitopoulos giving a retrospective on the mobile evolution culminating in the modern mobile revolution from 2008-onward.  He pointed out that fragmentation (e.g. unmanaged diversity of devices, os, apps, ux) presents the biggest challenge to testers who seek to make a solid and lasting determination about quality for a mobile application.  (I thought: yeah, nothing about mobile testing is solid or lasting).  In light of the demand for faster turn around time on testing feedback and defects, testers are under more pressure than ever due to mobile. 

Then the biggest tweeter of Summit topics, JeanAnn Harrison, led the Summit deep into mobile GUI functional testing into the hardware; into the firmware, battery, temperature sensors, peripherals. One suggestion for testers needing to dig this deep into a mobile test is to purchase a volt meter, and maybe pickup a temperature sensor. For normal software-only testers this might seem like overkill, but consider an application that requires high power (battery) or CPU to function properly. It would be good to know which top five devices would work best to recommend to customers and which five to avoid completely. To wrap things up, Scott and the crew had an open panel discussion with the top ten tips for mobile testing.

This is my second experience participating and presenting in an STP Online Summit and there are two great things that stick with me at this point. First, it’s completely convenient for busy people who can’t take off from home or work for a week-long conference. And I like virtual conferences because I can really tune-in to the entire set of content in the presentations, crew discussion forums, twitter and other side chats. It’s a multi-media party! Second, it’s a cost-effective means for professional development in an age where most employers have limited funding or permission to send testers to get additional training in the craft. 

The Summit consists of 9 sessions that are design to dig deeper into the topic than you could get in a normal conference. If you feel like you’re stagnating or outdated in your profession as a tester, the online summit can give you that leap back into the game and moving ahead.

Or, as Pete Townshend wrote, “Keep me movin', groovin', groovin', yeah, Movin', Yeah”

Testing with Professional Integrity

Last week I witnessed what can only be described as a completely integrity-challenged moment by a performance consultant I know, who said, in reference to my current customer:  “You’ve done a nice job educating them -- or should I say, brainwashing them!”

The consultant was cheering on what they thought was an effective strategy, that of persuading the customer into an overly elaborate view about what they need to succeed in order to compete with lesser experienced consultants.  In actuality my work had been focused on helping my customer and their other consultants fully understand the value and importance of doing performance testing and engineering in the best possible way.

Later, that same person went on to claim that “there are a lot of stupid people in the world” and that consultants like us need not commit ourselves to excellence in delivery of our consulting services, that lesser quality would suffice.  This person was advocating that it might be just fine to waste the customer’s time and money by delivering less than what was required by or expected by the customer. And sadly, this far from the first time I have witnessed this kind of unethical approach to consulting.

If you haven’t yet read Matt Heusser’s excellent posts on integrity in IT (Part 1 and Part 2), I encourage you to do so. He highlights a specific example from New York City (CityTime/TechnoDyne), an appalling tale of a team who decided to shakedown the city for $700M by taking advantage of the customer’s lack of technological acumen.

This is on an entirely different level from the snakeoil-selling forwarned by technology guru Clifford Stoll -- the CityTime scam went beyond the simple seductions of technology, and right off the deep end into egregious illegality. For a consultant with no ethics, it seems, any rule will get bent just to make few bucks.

So, after considering my interaction with the performance consultant who actively encouraged customer manipulation, and after reading Matt's great posts, and after reflecting on the many ethical lapses I have been witness to while working in this field over the last 20 years, I’d like to put on the table the three key practices in technology consulting which are perfect examples of a lack of professional integrity:

#1 – Unscrupulous persuasion. The consultant engages in dishonest tactics to “win the customer” by leaving out important details about their lack of talent, lack of legal licensing, lack of real experience or lack of professional certification. Conversely, embellishing on any of these subjects dishonestly is equally damaging to customers.

To operate with integrity, most professional testers will recommend that customers obtain second or third opinions, consult with other sources or professionals, and also thoroughly check consultant references.

#2 – Deliberate under-delivery. The consultant delivers less than excellent quality, when they know that they should be producing higher quality deliverables, but they leave out some of the required work in favor of a work extension or to ensure continued dependency on their own work product (e.g. leaving comments out of code, inadequate documentation).

To operate with integrity, a professional tester will provide industry-standard examples of the deliverables to the customer and commit to the same or better level of quality.

#3 – Willful perpetuation of ignorance. The consultant withholds important details about the work or the subject area from the customer so that the consultant may gain favor or financial gain. In the example above, the performance consultant expressed support for the idea of under-educating the customer about all aspects of the subject or situational context.

To operate with integrity, a consultant must provide as much information as possible to assist the customer, as well as admit with honesty the limits of their own scope of knowledge.

Consulting companies often have mechanisms to ensure that they are operating with professional integrity. Customers should always ask to see the employee code of ethics, standards of business conduct, or the legal statement of integrity in their contracts.

As an independent consultant, there are several ways I demonstrate my commitment to professional integrity. My contributions to and membership in industry forums shows that I am aware of and accountable to many other individuals in my own profession. As a member of the Software Test Professionals organization I am committed to learning and keeping up to speed on the state of the art practices for my profession. I consider myself a representative both of this profession and of its professional ethics; I encourage all of us to do the same.

I also just wrote this blog entry -- to put it on the record.

A Theory of Optimistic Performance

About 15 years ago my manager was quoted in an article where he said that software testers are inherently pessimistic people. The idea has resonated with me over the years, and it has shifted in my mind as I have shifted in my career from being a full-time quality analyst to becoming a test consultant specializing in performance testing. 

Initially, I had agreed with him, based solely on my experience in enterprise IT departments (and relatively little worldly experience.) Starting out as an internal quality analyst I spent much of my time learning how to break software and processes, learning how to find bugs. That’s what I got paid to do. But I also spend quite a bit of time being confused and frustrated in that role.

When I became a consultant, things flipped around. I spent more of my time focusing on the optimistic side of testing -- the benefits, the value, and the return on investment. Testing had to be a positive investment for my customers. And I realized that becoming a testing evangelist distanced me from those old feelings of frustration or disappointment, because in a consultative role, I had to be inspiring and persuasive.  

Then I went back to working as a tester again, at Microsoft, and I spent time every week with real customers engaged in the real world, one full of frustrating bugs, disappointing circumstance, and looming failures. This dose of reality brought me back down to earth. There it was again, that frustration and pessimism. But this time, during those short-cycled and intense performance optimization engagements, I saw customers change from a state of frustration to one of celebration. Reaching new peaks for performance with customers was exciting and positive, and often the result of  tireless efforts. 

And this new view on the experience led me to create my own theory about testers.

We are not inherently pessimistic people. Instead, we are actually relentlessly hopeful that we won't be disappointed. What I mean is, we aren't frustrated with the world as it is around us, and we also aren't longing for a more traditional approach to something. Testers don't lament about the past. Instead, when we are testing, we are all just temporarily frustrated, because our hopeful vision for an improved future just isn’t quite here. Yet.  

This is especially true for performance testers, where we attempt to manifest success for a system that’s limping along slowly and wrapped-up around itself, and we think “I know it’s just a few more tweaks and we’ll uncover the real power of this system!” 

So, the next time you get some complaint or push back on a performance defect where the developer is growing tired of your presumed negative or pessimistic attitude, be the optimistic tester you actually are. Remind them of your feelings about the future. Tell them about how you envision this bug -- and so many more -- being fixed. Let them in on the secret that you are not complaining about what they did in their code at some point in the past, but that you are thinking about how to celebrate the success that lies just around the corner.

Bucket full of Bottlenecks

For years I've seen a few super-guru performance engineers impress everyone by walking into the lab and with just 2-3 clicks through the code casually proclaim: “Sure, you just need to tweak the buffer setting in the max_connections_kernel_init.xml file to match the incoming parameters for the extension listing sequencer” with an “everybody-knows-that-one” attitude.  I figured my dream of becoming that cool could never happen. Or could it?

At one time or another in my own career as a performance engineer and tester, my self-perception was that I needed to be the guy who knows a huge list of all bottlenecks, and that would be my secret to success. But I've come to see there are 2 major flaws with this objective:

1)  there are a whole lotta of bottlenecks out there, and
2)  the list of bottlenecks is always changing and evolving

What I’ve learned from spending my working life encountering massive techno-churn, new platforms, and new usage patterns is that knowing where bottlenecks come from is far more important than knowing the bottlenecks after they exist. Understanding the engineering patterns that result in a performance bottleneck means that you can help to prevent performance bottlenecks from getting into the system in the first place.

In one of the performance engineering workshops I teach (“Performance Engineering for Product Owners”), I explain the importance of writing the specific performance requirements into the user stories or specifications.  Ideally, this means as the user story is evolving to become dev-ready there’s an opportunity to engineer performance into the thinking of the design, and that will impact how a developer responds to writing code or constructing the system to deliver on that story. And product owners can get ahead of bottlenecks, too.

So, the next time you are tirelessly tracking down the root cause of a performance bottleneck and some super-guru performance engineer comes walking in to pull a "miracle" solution from their bucket full of bottlenecks – remember the relative longevity of that fix, and the limits on such specific solutions.

STPCON 2012 Reflections on Excellence

If you’re like me, when you get a chance to attend a professional conference you look forward to meeting up with other people in “your tribe” (as some say).  I love these opportunities to connect with colleagues and peers who I’ve not seen for months or years sometimes.  Sometimes it’s people whom I have never had the pleasure of meeting in-person.  And always I’m meeting new people – with new ideas and new solutions that open my mind to new things.  My experience in the Spring of 2012 at the Software Testing Professionals conference (STPCON) in New Orleans (NOLA) was an excellent example of all these reasons.  Here’s a rapid-fire reflection on the folks I met and why it’s so valuable to be at this conference with our tribe!

For starters, I met up with Matt Heusser right away and that’s like diving into the deep end of the STPCON pool – he’s so intense, passionate and excited about testing it’s hard to believe.  During our late-night dining at Daisy Dukes, he invented an alcoholic milkshake made of warmed milk and Kahlua.  (Apparently not bad!)  Oh yeah, we also got caught up on work for future conferences and also giving a referral to a customer near his family in Salem, OR.  I got to chat with Jon Bach for 10 minutes before his presentation and introduced myself as a fellow responder to Alberto Savoia’s “Test is Dead” industry-bashing scandal.  Jon’s treatment of the subject in his keynote was a very professional response to the naiveté’ of Alberto and James Whittaker. 

Then, in my first track presentation on Conscientious Testing I noticed a familiar face, Ben Simo (@QualityFrog) in the back – lurking, but also smiling a nodding quite a bit at the review of how to be an ethically aware software tester.  At the same time my excellent friend Scott Moore was also in the session bringing up points of reference from his experience in performance test consulting from the last 20 years.  I stayed on for Scott’s session on Peak Performance Planning which was excellent!!  Later that day I had my second session on Performance Testing Metrics and Measures – which was exhausting, packed, standing-room only – and I got to sync-up with my mentor and peer Scott Barber.  Basically, if you have a job today in performance testing or as a load tester or performance engineer of any kind…you probably owe some thanks to Scott for his undying commitment to our discipline and industry.  That evening I got to hang out with the team from HP Software (John, Sylvia, Heather and Kristin) - chatting about the latest & greatest news from our favorite old ex-Mercury product vendor.  Scott Moore also taught us a new way to properly taste single-malt scotch.

The next morning I participated in Breakfast Bytes – taking an interactive survey of participants on the topic of Performance Engineering vs. Performance Testing titles and careers (to be summarized in a later blog post, coming soon).  I got to meet up with Lanette Creamer, who I recalled from the last STPCON in Fall of 2011 in Dallas, TX.  Lanette is a very sharp tester and presenter with a great sense of humor to keep you going!  At the reception that evening I got to have a marvelous chat with Fiona Charles about her own writing on ethics in software testing. She’s also worked closely with Jerry Weinberg and sent me a few references and readings to consider.

I finally met Dan Bartow from SOASTA in-the-flesh – and he’s really a great guy; enthusiasm for performance is not his weakness in any way.  Brad Johnson and Dan went with a group of us out to the Old Absinthe hotel and like true testing geeks we just hung-out and told stories about testing.  Sure we had some absinthe (the less-potent, legal kind) which was tasty actually.  As the conference continued, there were presentations from several other excellent testing guru’s out there like Kerry Fields, Griffin Jones and Brian Lynch – each of whom gave marvelous presentations, with depth and experience.

But there was a moment that really struck me at the very last day when I saw Ben Simo again and he responded to me:  “I realized – I need to do this.”  [referencing the idea of getting out and presenting, speaking and sharing and participating in the community].  I smiled and said, “Absolutely you should!“

What Ben was experiencing is the kind of buzz at the STPCON event, which is highly positive and uplifting, and that is being surrounded and immersed in testing excellence.  It’s so different from the conflict-ridden existence we typically face each day in our jobs as testers.  It’s different from the politics or the negativity we sometimes are forced to deal with in our work, in business or on our teams.  At STPCON all of that noise just fades into the background and you are surrounded by testers!  You’ll find (like me) that every conversation is filled with opportunity to learn and to share something excellent about software testing. 

See you in Miami – STPCON FALL 2012!!

PS – special thanks to Abbie, Fiona, Rich, Matt and the STP Crew!  This couldn’t happen with your excellent support.