Why the Statement “I Don’t Source Control My Unit Tests” Makes Me Happy
An Extraordinary Claim
I was talking to a friend the other day, and he relayed a conversation that he’d had with a co-worker. They were discussing writing unit tests and the co-worker claimed that he writes unit tests regularly, but discards them and never checks them into source control. I was incredulous at this, so I asked what the rationale was, to which my friend cited his co-worker’s claims (paraphrased) that “[unit tests] are fragile, they take too long to maintain, they prohibit adapting to new requirements, errors are caused by user change requests not incorrect code, and they’re hard to understand.” (emphasis mine, and I’ll return to this later).
This struck me as so bizarre — so preposterous — that I immediately assumed I was out of my league and had missed an industry sea-change. I remember a blog post by Scott Hanselman entitled “I’m A Phony. Are You?” and I felt exactly what he meant. Clearly, I’d been exposed. I was not only wrong about unit tests, but I was 180 degrees, polar opposite, comically wrong. I had wasted all of this time and effort with TDD, unit test maintenance, unit tests in the build, etc when what I really should have been doing was writing these things once and then tossing them since their benefit is outweighed by their cost.
Given that I don’t have direct access to this co-worker, I took to the internet to see if I could locate the landmark post or paper that had turned the tide on this subject and no doubt a whole host of other posts and papers following suit. So, I googled “Don’t Check In Unit Tests”. Your mileage may vary because I was logged into gmail, but I saw old posts by Steve Sanderson and Jeff Atwood encouraging people to write tests, a few snarky posts about not testing, and other sorts of things you might expect. I had missed it. I tried “Don’t version control unit tests” and saw similar results. A few variants on the same and similar results too. Apparently, most of the internet was still thinking inside of the box that the co-worker had escaped with his bold new vision.
Finally, I googled “Should I source control unit tests?” and found one link from programmer’s stack exchange and another from stackoverflow. In a quick and admittedly very rough count of people who had answered and/or voted one way or the other, the vote appeared to be about 200 to 0 for source controlling versus leaving out, respectively. The answer that most succinctly seemed to summarize the unanimous consensus was Greg Whitfield’s, which started with:
Indeed yes. How could anyone ever think otherwise?
Hmmm…
There’s Really No Argument
With my confidence somewhat restored, I started thinking of all the reasons that source controlling unit tests makes sense:
- Provides living ‘documentation’ of intentions of class author.
- Prevents inadvertent regressions during changes.
- Allows fearless refactoring of your code and anyone else’s.
- Incorporation into the build for metric and verification purposes.
- Testing state at the time of tagged/branched versions can be re-created.
- (As with source controlling anything) Hard drive or other hardware failure does not result in lost work.
I could probably go on, but I’m not going to bother. Why? Because I don’t think that the rationale are actually rationale for not checking in unit tests. I think it’s more likely that this person who “writes tests but doesn’t actually check them in” might have had a Canadian girlfriend as a teenager. In other words, I sort of suspect that these “disposable” unit tests, whose existence can neither be confirmed nor denied, may not necessarily actually exist and so the rationale for not checking them in becomes, well, irrelevant.
And, look at the rationale. The first three clauses (fragile, too long to maintain, prohibit new work) seem to be true only for someone who writes really bad unit tests (i.e. someone without much practice who may, I dunno, be discouraged during early attempts). Because while it’s true that unit tests, like any other code, constitute a maintenance liability, they need not be even remotely prohibitive, especially in a nicely decoupled code base. The fourth clause (errors are everyone’s fault but the programmer) is prima facie absurd, and the fifth and most interesting clause seems to be an unwitting admission of the problem – a difficulty understanding how to unit test. This is probably the most important reason for “not checking in” or, more commonly and accurately, not writing unit tests at all — they’re hard when you don’t know how to write them and haven’t practiced it.
So Why the Happy?
You’re probably now wondering about the post title and why any of this would make me happy. It’s simply, really. I’m happy that people feel the need to make excuses (or phantom unit tests) for why they don’t automate verification of their work. This indicates real progress toward what Uncle Bob Martin describes in a talk that he does called “Demanding Software Professionalism” and alludes to in this post. In his talk, Bob suggests that software development is a nascent field compared to others like medicine and accounting and that we’re only just starting to define what it means to be a software professional. He proposes automated verification in the form of TDD as the equivalent of hand-washing or double-entry bookkeeping respectively in these fields and thinks that someday we’ll look back on not practicing TDD and clean coding the way we now look back on surgeons that refused to wash their hands prior to surgery.
But having a good idea and even demonstrating that it works isn’t enough. Just ask Ignaz Semmelweis who initially discovered, and empirically demonstrated that hand-washing reduced surgery mortality rates, only to be ridiculed and dismissed by his peers in the face of cold-hard evidence. It wasn’t until later, after Semmelweis had been committed to an insane asylum and died that his observations and hypothesis got more backers (Lister, Pasteur, et al) and a better marketing campaign (an actual theoretical framework of explanation called “Germ Theory”). In Semmelweis’s time, a surgeon could just call him a crank and refuse to wash his hands before surgery. Decades later, he would have to say, “dude, no, I totally already washed them when you weren’t looking” if he was feeling lazy. You can even ask his Canadian girlfriend.
At the end of the day, I’m happy because the marketing for TDD and clean coding practice must be gaining traction and acceptance if people feel as though they have to make excuses for doing it. I try to be a glass half full kind of guy, and I think that’s the glass half full outlook. I mean one person not wanting to automate tests doesn’t really matter in the scheme of things at the moment since there is no shortage of people who also don’t want to, but it’s good to see people making excuses/stories for not wanting to rather than just saying “pff, waste of time.”
(And, for what it’s worth, I do acknowledge the remote possibility that someone actually does write and discard unit tests on a regular and rigorous basis. I just don’t think it’s particularly likely.)
Testing is definitely important, but unit tests are often too fine-grained to catch the really significant (and costly) system-wide problems that plague most development. TDD proscribes a hyper-active approach to testing, resulting in significantly more initial effort and a considerable bloat to the test base. Some tests should certainly stay around, but some are just scaffolding and once they’ve shown an absence of problems they become unnecessary baggage. They can also provide a dangerous false confidence for practices like refactoring. A test is nothing more than a different way of stating a problem, if both the code and the test… Read more »
I don’t believe anyone says it is the job of unit test to catch the system-wide problems. Unit test should not be the complete test plan for a project. Our group does unit test ( that we check into source control ๐ ) but has a QA group that does automation test and some manual test against the complete build. Even at the developer level we do ‘smoke test’ after the end of a good TDD session where we are totally confident that our code will work, we start the application and run it. Example: With unit test I can… Read more »
Unit tests for something complex like a parser are a definite necessity. But I’ve always felt that unit tests on more mundane code that also must be tested at the system level, is redundant. A lot of code is just glue or the required noise to get the whole thing working. One big thing I’ve found over the years is that ‘reuse’ heavily cuts down on testing effort. If the system is built with tonnes of redundant ‘brute forced’ code, then the little inconsistencies require way more testing or the system is fragile and unstable. If the underlying code is… Read more »
It took me a while to wrap my head around the idea that TDD is really a design methodology that produces test artifacts rather than a pure testing methodology, but once that became clear to me, I began to understand that not all tests were long for this world. For instance, I might write some truly obtuse test just to get a new class and constructor that I wanted, only to delete or change that test minutes later. So I certainly agree that, just as with normal source code, the fact that it’s been written doesn’t necessarily means it belongs… Read more »
You mean the pretty tiles? That’s a new Blogger template. I added it a few months back. It is quite amusing, there are several different displays and I find resizing the window very entertaining. (for some reason this comment box is ignoring returns and enter, so I can’t start seem to start new paragraphs. I’m back a few versions in Ubuntu and having some issues with packages, so I haven’t upgraded my Firefox for a while. So I guess it’s one big paragraph for me ๐ From my experiences, the focus on software quality fluctuates quite a bit. A couple… Read more »
There are definitely some interesting cycles. The one that always strikes me is the thick versus thin client that recently swung again with the popularity of the phone “app”. Still, I’d like to think that even accounting for cycling there is some progression. Given the incredible advances in processing power, storage, and technology in general, it seems like we ought to be able to have our cake and eat it too in the “thoroughness vs quick to market” tradeoff, at least as compared to years past. (That comment by your prof seems like the ultimate in “get off my lawn!”… Read more »
Your side comment is really (really) funny ๐ Mostly because he did resemble that cranky old neighbor that is always yelling at the kids and spoiling all of the fun, but I had never really thought of it that way before. Hardware has made stunning advances and I am often amazed at a lot of the new toys (my watch is an iPod nano with 70 albums on it for my listening pleasure ๐ Software is way larger and the amount of ‘canned’ pieces available is also stunning. But, the time it takes to get out a quality commercial product… Read more »
[…] I can TDD ’til the cows come home and check everything in so that nobody accuses me of having a Canadian girlfriend. In other words, I get to have my cake and eat it […]
[…] s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(po, s); })(); Recently, I posted my incredulity at the idea that someone would go to the effort of writing unit tests and not source control them […]
[…] most of those who don’t do it say things like “I’d like to start” or they make up weird excuses. Unit testing is The Way Forward (caps intentional) and commenters like these are standing around, […]
[…] I think the feeling described strongly parallels the feeling that a developer has when the subject of unit testing comes up. Maybe it’s broached in an interview by the interviewer or interviewee. Perhaps it comes up when a new team member arrives or when you arrive as the new team member. Maybe it’s at a conference over drinks or at dinner. Someone asks what you do for unit testing, and you scramble to hide your dirty little secret–you don’t have the faintest idea how it works. I say that it’s a dirty little secret because, these days, it’s generally… Read more »
[…] a year ago, I blogged about a guy who made a silly claim that he wrote a lot of unit tests but didn’t chec…. The reasoning behind this, as detailed in the post, was completely fatuous. But that’s to be […]