Confessions of a beta tester

January 25, 1999
by Zac Belado

or "How do you know until you try it?"

Beta testing software, like many things in life, is often like orchids; each experience is distinct and you can spend a few months working and still have nothing to show for it. Despite the jokes that we all make, some beta software is very prone to crashes, lock-ups, data corruption and a whole raft of "things that are not good". If you think the first release version of some software is buggy, you have not seen a double-digit developers release of a product (typically beta testers don't see a piece of software until the build numbers are well into the hundreds). Which tends to make the early beta testing process a bit dodgy at times.

I've spent the last few months beta testing two different software products for a company whose name you might know. And no, it isn't Microsoft, although aren't we all Microsoft beta testers in a way? This situation was a bit different than many of the other beta testing experiences I have had for several reasons. One, I actually use the products on a daily basis and two, I actually had time to test them this time. Consequently I was able to learn a few new "truths" about the testing process and thought that I might pass them along in hopes that it might explain how software ships with bugs still in it.

Truth 1: Some bugs are plain weird.

One of the bugs I found was, while not totally bizarre, certainly not something you'd think you'd run in to. Typically I test software by just using it and by deliberately testing a set of properties or features. For example, I might try testing and setting all the properties of an object. Or I might take an object and use it in the officially defined ways in which it can be used. If all that works then it's time to try and cause errors. If a piece of software isn't supposed to do a certain thing, then it had better actually give you an error if you try. But, one of the bugs I found though was so precise and convoluted that I think it probably would have escaped notice if I hadn't stumbled over it.

Without mentioning the software, it involved taking a text container, placing it in a document, and then, without deselecting it or selecting another object first, using the keyboard shortcut to open the container's properties, turning the scroll bars on, and then clicking on the "Okay" button. This resulted in the product freezing.

If you didn't do every single step in the same order, or if you selected another of the container's properties it wouldn't crash. Just this single path of actions that you probably wouldn't do more than once in a year resulted in a crash. I can't think that I would have come across this if I was looking for it.

These are, of course, often the things that get picked up in about 20 minutes after the product's release.

Truth 2: Nothing tests like a release.

The reason for that so many bugs get found immediately after release is twofold; bad luck and having an installed base of "testers" 1000 times larger than your beta test group. If a bug might only show up once in a year for an individual user, it will show up in about three hours after release. No matter how bizarre the bug.

The release of Quake II is a perfect example of this sort of thing. The product went through extensive internal testing, was farmed out to a third party firm for "professional testing", was released, as a test version, to people on the net and despite all this, id software logged over 1000 bugs in the first week of its release.

Quite a few of these were hardware incompatibilities that are inevitable in the PC market. Some of them were also plain-jane bugs that escaped the testers. The final market for the product is much, much larger than the beta test group so even vastly less efficient usage of the product is bound to turn up bugs that the testers didn't find.

Truth 3: Some bugs aren't bugs

Just because you think it's a bug doesn't mean that the software engineers think it's one as well.

"Bugs" like this come in two types. Bugs that were in the last version that weren't fixed and are now accepted and bugs that are just a result of you having a different expectation of the way the software works.

Let's say, for example, that I think a certain button on a tool palette should not become inactive in certain circumstances. If the software engineer disagrees with me, then...

This isn't necessarily a bad thing. People approach a complex entity like a piece of software in different ways. People also have different expectations of what proper functionality is. I'm sure, though, that the same software engineer would re-think her position if 10 or 15 testers had the same complaint. It's a bit frustrating (even more so when you expect you're going to have to put up with the silly thing for at least a year to 18 months) but understandable.

The same thing does applies if all you are reporting is a pre-existing bug. This problem actually disturbs me quite a bit. Not every company does it but some companies play "actuarial accountant" and figure that if they haven't had to fix a bug through a full release cycle then they can safely get way with not having to fix it in the new one. People have either not found the bug (highly unlikely) or have just gotten used to it. It's the digital version of the squeaky wheel getting grease: the bug that generates complaints gets fixed. To their credit, this is not a practice that I have ever seen a programmer or QA engineer happy with, but then, they don't make the final decisions. Often times I think that these decisions are made by people that don't even use the software. But that would be cynical and bitter of me and I'm trying to turn a new leaf.

Truth 4: Software has many entry points

There is nothing like writing your own application to make you aware of just how many different ways there are to do things. Even simple things like entering a date or a price can be approached in a vast number of ways. Data can be entered in different formats, fields can be entered by clicking on them or by using the Tab key, data can be entered partially and then re-entered, data can have an unforeseen format.

Software gets complicated when you add a GUI to it. Unlike text based interfaces, a GUI allows the user an enormous amount of freedom and also increases the number of possible entry points for errors and bugs. If I have a text based system, I can ask for a series of pieces of information one at a time and test them as I get them from the client. If the application uses a GUI, I can test the condition of each piece of information at one of many points. Or I can miss the opportunity to test it as well. To further complicate things people will often use software in ways you hadn't even thought.

Software gets really complicated when it is used to make other software. I sit back and am constantly amazed that pieces of software like Visual Basic, REALBasic and Director can even compile an app.

So it shouldn't really surprise anyone that bugs can and do creep into products.

Truth 5: Software happens.

I read an editorial in the latest (February 99) issue of "Computer Games Strategy Plus" that bemoaned the "when it's done" development philosophy that is the current trend among game designers (which is, if you're not familiar with it, that the product will ship "when it's done.")

Given some of the software I have seen this last year I can't possibly agree with the author. A product shouldn't be rushed to make an arbitrary sales deadline or to be available for inclusion in the quarter's financial results so that the stock price doesn't take a tumble. Far too many products ship too soon and the decision to do so is not made by the people responsible for the product or for any reason other than greed.

I can certainly count several recent products that could have, and should have, spent several more months in testing before their release. In fact, I only received two products last year that I was really satisfied with; both of which were games and both took more time to develop than many commercial development environments.

During the test cycle for one of the last two applications I tested, we (the testers) were told that the latest build was going to be the last one before shipping. I, for one, was stunned speechless. The product was nowhere near ready for release. Posted bugs were still causing problems in that release and I was finding new bugs. But a decision had been made.

Thankfully cooler, and saner, heads prevailed and the product saw two more major builds before its final release.

Bugs are a fact of life. And, despite the negative tone, I think crappy initial releases of products will also be the norm in the coming year. I already have a general rule about not buying the initial release of any game (or the first five releases if it's by Epic Megagames) and I think it is a rule I will be extending to almost all commercial software. There are some obvious exceptions: Adobe of course, and some companies that already have had the rule applied to them, Microsoft and Quark. But I think the general trend will be for more product to be pushed out the door before it's due.

Too bad the Gallo Brothers don't make software.

Zac Belado is a programmer, web developer and rehabilitated ex-designer based in Vancouver, British Columbia. He currently works as an Application Developer for a Vancouver software company. His primary focus is web applications built using ColdFusion. He has been involved in multimedia and web-based development, producing work for clients such as Levi Straus, Motorola and Adobe Systems. As well, he has written for the Macromedia Users Journal and been a featured speaker at the Macromedia Users Convention.