A software testing metaphor, or why I hate the literature so much

My father was (and is) a competent golfer. My brother was good enough to flirt with the bottom rung of pro status. I have always been terrible. I never learned how to control the swing. I whack way too hard at the ball. I don't have any "feel" for the club, or the swing. The rare times I make reasonably good contact, my typical ball flight is somewhere between a high fade and an outright slice.

With me so far?

In an effort to help me out, someone bought me a golf technique book by Ben Hogan. I really tried to read through it, think about what Hogan was trying to explain, practice, and so on. But it just didn't work.

I think Hogan's book failed to help me for one specific reason and one general reason. The specific reason is that Ben Hogan was a fairly short and very strong man whose main problem was a hook. So the grip, timing, and other techniques he used were primarily to prevent a hook. The general reason is that I am so bad a golfer that I still have no idea what a good golf swing should feel like - what is the best tempo, how should the club feel in my hands, what does it feel like to bring the club face squarely in contact with the ball, and so on. I also have no idea what it feels like to miss this idea swing in any of the several different ways that I can miss. How can I tell that I am lunging my shoulders forward, when not only do I not really know where my shoulders should be, but I also don't really feel small differences in where my shoulders are?

A little while ago, we bought Wii Fit Plus. One of the things on it is a golf simulator. I doubt that it is particularly precise, and I suspect that it might not even be accurate. (Obligatory digression: the important difference between "accuracy" and "precision" is lost on many. My high school chemistry teacher John Belk taught it to me and I urge you to go research it. Maybe finish reading this post first.) But the point is that I think I am beginning to learn, in a very broad sense, what a reasonable golf swing feels like, and what it feels like to miss reasonableness in different ways. Maybe if I continue practicing with the Wii for a while, I might be able to go out to the real physical driving range and react properly to mistakes in my swing. The thing is, even when I get to this point, I am quite sure that Ben Hogan's book is not going to be the right book for me, because of our physical differences.

I got my hands on "Think Like a Grandmaster" by Kotov fairly early in my chess career. The first time I read it, it was clearly over my head. I put it down and didn't pick it back up for a long time. Now I can tell that I would benefit from it -- and in fact it would be one of the best books I could study at this point, if I had the time or the interest.

OK, let me bring this discussion over to software testing where it belongs.

One of the most interesting (and fraught) things about software testing and quality assurance is that you think "outside the box", and at many different levels, to provide information to the people who are trying to build a software product. Just for starters on why it's so fraught: I have been doing this job for at least fifteen years now, and have had some success at it, and I still do not know where "testing" ends and "quality assurance" begins. I think that some people think they know, but I have never been able to agree with any particular definition, or to even find the effort at definition useful. But anyway, in general, what you usually do as a software QA engineer (whether you know you are doing this or not) is conduct experiments. A hypothesis comes to mind -- this line of code is not doing what it's supposed to, this component is not doing what we say it's doing, this subsystem is written in a way that is insufficiently generalizable, this feature is not delivering enough value, this way we're doing work is inefficient. You design and execute an experiment. The result of this experiment is increased knowledge about the product. This increased knowledge is the primary output of your work. There are many side effects: the line of code is fixed to do what it's supposed to, the component (or its documentation) is changed so that it is doing what we say it is doing, the subsystem is refactored, the feature's requirements are re-thought, the process is improved. But the doing of those things is usually someone else's job. The QA engineer's responsibility usually ends with the discovery and analysis of the problem.

Now let me bring up something that I have just hated for years. There are hundreds of books out there that tell you how to do software QA. There are organizations that will, for the right price, sit you through classes that indoctrinate you in how to think about the QA job, hand you a certificate, and encourage you to use it to impress the rubes and convince yourself that you're Doing Something Professional. But -- and here's my point finally -- most of those books and courses are like Ben Hogan's book, only worse. Not only do most software QA books only apply if your situation is enough like the author's, but it's not as easy to determine whether the author is really someone whose advice you really ought to be taking. Going back to the golf books for a moment: if you read a book by Ben Hogan, or Jack Nicklaus, or Tiger Woods, you'd probably start off giving the author the benefit of the doubt and would assume his advice was worthwhile until proven otherwise. If you read a book by someone whose name you did not know, but who had coached several professional golfers, you might think that the advice was probably going to be good, but you might want to come up with a way to find out whether the golfers' success was due to their own abilities or the author's coaching before you really put faith in the advice. In software, there are so many more factors and people involved in a product's success than there are in golf, that even if an author was the QA Director of a hugely successful project, it will be very difficult to determine whether the project succeeded because of, or in spite of, that person's contribution.

I have read dozens of books that were like bad versions of Ben Hogan's golf book: of uncertain value, and almost definitely not relevant to my situation.

But I have also discovered some software testing/QA books like "Think Like a Grandmaster" by people like Grandmaster Kotov. The people generally identify with a crowd calling itself the "Context-Driven School of Software Testing". See this place called Satisfice by a guy named James Bach, for example. I mean this comparison to be not just glib, but also deep: Most of what Bach talks about is "meta" - more about the thought process than any specific thing you're doing. The book "Lessons Learned in Software Testing" (by Bach, Cem Kaner, and Bret Pettichord), is a lot like "Think Like a Grandmaster" - and it is therefore hugely more useful and relevant than the 95% of the literature that is like bad versions of Ben Hogan's golf book. It was a real revelation to run across the Context-Driven crowd.

I have a whole lot more to say about software testing and QA -- both in general and in many specifics. It's safe to say I have a lot more to say than I can actually say. I should have started blogging eight or ten years ago, but I was always writer's-blocked. So I'm hoping that the posts labeled "softwaretesting" will mainly serve me as therapy. Maybe I'll eventually be able to say something clear and useful as well!