In Defense of Using Your Users as Testers
Editorial note: I originally wrote this post for the NDepend blog. You can check out the original here, at their site. While you’re there, download a trial of NDepend and take it for a spin; you can try it for free.
In most shops of any size, you’ll find a person that’s just a little too cynical. I’m a little cynical myself, and we programmers tend to skew that way. But this guy takes it one step further, often disparaging the company in ways that you think must be career-limiting. And they probably are, but that’s his problem.
Think hard, and some man or woman you’ve worked with will come to mind. Picture the person. Let’s call him Cynical Chad. Now, imagine Chad saying, “Testing? That’s what our users are for!” You’ve definitely heard someone say this at least once in your career.
This is an oh-so-clever way to imply that the company serially skimps on quality. Maybe they’re always running behind a too-ambitious schedule. Or perhaps they don’t like to spend the money on testing. I’m sure Chad would be happy to regale you with tales of project manager and QA incompetence. He’ll probably tell you about your own incompetence too, if you get a couple of beers in him.
But behind Chad’s casual maligning of your company lies a real phenomenon. With their backs against the wall, companies will toss things into production, hope for the best, and rely on users to find defects. If this didn’t happen with some regularity in the industry, it wouldn’t be fodder for Chad’s predictable jokes and complaints.
The Height of Unprofessionalism
Let’s now forget Chad. He’s probably off somewhere telling everyone how clueless the VPs are, anyway.
Most of the groups that you’ll work with as a software pro would recoil in horror at a deliberate strategy of using your users as testers. They work for months or years implementing the initial release and then subsequent features. The company spends millions on their salaries and on the software. So to toss it to the users and say “you find our mistakes” marks the height of unprofessionalism. It’s sloppy.
Your pride and your organization’s professional reputation call for something else. You build the software carefully, testing as you go. You put it through the paces, not just with unit and acceptance tests, but with a whole suite of smoke tests, load tests, stress tests and endurance tests. QA does exploratory testing. And then, with all of that complete, you test it all again.
Only after all of this do you release it to the wild, hoping that defects will be rare. The users receive a polished product of which you can be proud — not a rough draft to help you sort through.
Users as Testers Reconsidered
But before we simply accept that as the right answer and move on, let’s revisit the nature of these groups. As I mentioned, the company spends millions of dollars building this software. This involves hiring a team of experienced and proud professionals, among other things. Significant time, money, and company stake go into this effort.
If you earn a living as a salaried software developer, your career will involve moving from one group like this to another. In each of these situations, anything short of shipping a polished product smacks of failure. And in each of these situations, you’ll encounter a Chad, accusing the company of just such a failure.
But what about other situations? Should enlisting users as testers always mean a failure of due diligence? Well, no, I would argue. Sometimes it’s a perfectly sound business or life decision.
Open Source Side Projects
First up, let’s consider a passion project that you might do on the side. You build some library, framework, or set of language extensions that make life easier for developers. It started out as something you built for yourself, but then you threw it up on Github. If others might find it useful, why not let them have at it?
But caveat emptor, you warned from the beginning. You write unit tests and put it through the paces a little bit, but that’s hardly comprehensive. If others find defects or issues, they can let you know or they can create pull requests.
This is perhaps the most straightforward possible example of using your users as testers. And I doubt anyone would have an issue with the approach in this context.
Startups, Especially Bootstrapped
Let’s take it a step further. With the example of you doing something for the love of the game, one can point to the lack of money at stake. (Though philosophically, I would dispute this, since you could easily attempt later to capitalize a side project). Let’s put some money into the pot.
How about the case of a startup, particularly one that bootstraps? Here, you do have real money at stake. You just don’t have a whole lot of it, which can lead to difficult decisions. A large organization won’t have to choose between spending money on testing and paying the owner’s car payment. A bootstrapped startup may.
Another factor also comes into play. With a startup, users have a much higher tolerance for a lack of polish. They tend to understand what they’re getting, warts and all. And so this combination of factors can make a startup the perfect candidate for enlisting their users as testers.
A Carefully Considered Tradeoff
In both of the last two situations, most people will sympathize. In both cases, there’s a sense that “proper” testing simply isn’t feasible. So you forgive the lack.
But I’d say the approach can even make sense in cases where the organization could spare the testing effort. I’ll grant that this will feel less sympathetic to anyone looking at it, but it can still make sense.
A more established organization might look at two factors: cost of the testing/quality effort and reputation. They would then ask themselves whether reputation damage from a shoddy output would cost more than the effort to test. Usually, the investment in testing wins out. That’ll be true for most groups you work with. But it’s not always true.
For example, imagine a company maintaining a legacy application. It has released two newer products since that one, and it has exhorted its users to migrate to newer versions. Most have, but a few staunch holdouts remain. Maybe this company starts launching relatively untested patches when patches are required. Will this affect their reputation? Negligibly. They’re concerned with the latest software, as are most of their users.
Open and Honest User Engagement
I’ll close by mentioning a fourth situation where users as testers makes senses. But while this case can stand alone, you can also apply it to the previous three examples or really to any example you might think of. I’m talking about a situation where you have an engaged user community and you’re upfront and honest with them.
When they opt in and care about the product, many users don’t mind serving as testers. That’s kind of the whole idea behind beta testing. Enlisting these users goes over well because they want to be involved and because they understand the rules of the road. There’s a world of difference between a “careful, this is a beta” release and rolling out buggy software as if it were polished.
I think everyone could agree on rolling out perfect software for an excellent user experience. But shipping software is hard and never goes according to plan. You have to use the tools at your disposal, which can include strategically enlisting users to help with quality. But if you choose to go this route, please be honest with your users about that — or you’ll find yourself deserving of everything Chad has to say about you.