DaedTech

Stories about Software

By

Testable Code is Better Code

It seems pretty well accepted these days that unit testing is preferable to not unit testing. Logically, this implies that most people believe a tested code base is better than a non-tested code base. Further, by the nature of testing, a tested code base is likely to have fewer bugs than a non-tested code base. But I’d like to go a step further and make the case that, even given the same amount of bugs and discounting the judgment as to whether it is better to test or not, unit-tested code is generally better code, in terms of design and maintainability, than non-unit-tested code.

More succinctly, I believe that unit testing one’s code results in not just fewer bugs but in better code. I’ll go through some of the reasons that I believe that, and none of those reasons are “you work out more bugs when you test.”

It Forces Reasoning About Code

Let’s say that I start writing a class and I get as far as the following:

public class Customer
{
    public int Id { get; set; }

    public string LastName { get; set; }

    public string FirstName { get; set; }
}

Pretty simple, but there are a variety of things right off the bat that can be tested. Can you think of any? If you don’t write a lot of tests, maybe not. But what you’ve got here is already a testing gold mine, and you have the opportunity to get off to a good start. What does Id initialize to? What do you want it to initialize to? How about first and last name? Already, you have at least three tests that you can write, and, if you favor TDD and don’t want nulls or zeroes, you can start with failing tests and make them pass.

It Teaches Developers about the Language

A related point is that writing unit tests tends to foster an understanding of how the language, libraries, and frameworks in play work. Consider our previous example. A developer may go through his programming life in C# not knowing what strings initialize to by default. This isn’t particularly far-fetched. Let’s say that he develops for a company with a coding standard of always initializing strings explicitly. Why would he ever know what strings are by default?

If, on the other hand, he’s in the practice of immediately writing unit tests on classes and then getting them to pass, he’ll see and be exposed to the failing condition. The unit test result will say something like “Expected: String.Empty, Was: null”.

And that just covers our trivial example. The unit tests provide a very natural forum for answering idle questions like “I wonder how x works…” or “I wonder what would happen if I did y…” If you’re working on a large application where build time is significant and getting to a point in the application where you can verify an experiment is non-trivial, most likely you leave these questions unanswered. It’s too much of a hassle, then the alternative, creating a dummy solution to test it out, may be no less of a hassle. But, sticking an extra assert in an existing unit test is easy and fast.

Unit Tests Keep Methods and Classes Succinct and Focussed

public void ChangeUiCar()
{
    try
    {
        Mouse.OverrideCursor = Cursors.Wait;
        MenuItem source = e.OriginalSource as MenuItem;

        if (source == null) { return; }

        ListBox ancestor = source.Tag as ListBox;

        if (ancestor == null) return;

        CarType newType = (CarType)Enum.Parse(typeof(CarType), ancestor.Tag.ToString());

        var myOldCar = UIGlobals.Instance.GetCurrentCar();
        var myNewCar = UIGlobals.Instance.GetNewCar(newType);

        if (myNewCard.Manufacturer == "Toyota" || myNewCar.Manufactuer == "Hyundai" || myNewCar.Manufacturer == "Fiat")
        {
            myNewCar.IsForeign = true;
        }
        else if (myNewCar.Manufactuer == "Ford" ||
            (myNewCar.Manufacturer == "Jeep" && myNewCar.WasMadeInAmerica)
            || (myNewCar.Manfacturer == "Chrysler" && myNewCar.IsOld))
        {
            myNewCar.IsForeign = false;
        }

        try
        {
            UpdateUiDisplay(myNewCar, true, false, 12, "dummy text");
        }
        catch
        {
            RevertUiDisplay(myOldCar, true, false, 0, "dummy text");
        }

        if (myNewCar.HasSunRoof || myNewCar.HasMoonRoof || myNewCar.HasLeatherSeating || myNewCar.HasGps ||
            myNewCar.HasCustomRims || myNewCar.HasONBoardComputer)
        {
            bool isLuxury = CarGlobals.Instance.DetermineLuxury(myNewCar);
            if (isLuxury)
            {
                if (myNewCar.IsForeign && myNewCar.IsManualTransmission)
                {
                    myNewCar.DisplaySportsCarImage = true;
                }
                if (myNewCar.IsForeign)
                {
                    myNewCar.DisplayAmericanFlag = false;
                    if (myNewCar.HasSpecialCharacters)
                    {
                        myNewCar.DisplaySpecialCharacters = UIGlobals.Instance.CanDisplayHandleSpecialCharacters();
                        if (myNewCar.DisplaySpecialCharacters)
                        {
                            UpdateSpecialCharacters(myNewCar);
                        }
                    }
                    else
                    {
                        UIGlobals.Instance.SpecialCharsFlag = "Off";
                    }
                }
            }
        }

    }
    finally
    {
        Mouse.OverrideCursor = null;
    }
}

This is an example of a method you would never see in an actively unit-tested code base. What does this method do, exactly? Who knows… probably not you, and most likely not the person or people that ‘wrote’ (cobbled together over time) it. (Full disclosure–I just made this up to illustrate a point.)

We’ve all seen methods like this. Cyclomatic complexity off the charts, calls to global state sprinkled in, mixed concerns, etc. You can look at it without knowing the most common path through the code, the expected path through the code, or even whether or not all paths are reachable. Unit testing is all about finding paths through a method and seeing what is true after (and sometimes during) execution. Good luck here figuring out what should be true. It all depends on what global state returns, and, even if you somehow mock the global state, you still have to reverse engineer what needs to be true to proceed through the method.

If this method had been unit-tested from its initial conception, I contend that it would never look anything like this. The reasoning is simple. Once a series of tests on the method become part of the test suite, adding conditionals and one-offs will break those tests. Therefore, the path of least resistance for the new requirements becomes creating a new method or class that can, itself, be tested. Without the tests, the path of least resistance is often handling unique cases inline–a shortsighted practice that leads to the kind of code above.

Unit Tests Encourage Inversion of Control

In a previous post, I talked about reasoning about code in two ways: (1) command and control and (2) building and assembling. Most people have an easier time with and will come to prefer command and control, left to their own devices. That is, in my main method, I want to create a couple of objects and I want those objects to create their dependencies and those dependencies to create their dependencies and so on. Like the CEO of the company, I want to give a few orders to a few important people and have all of the hierarchical stuff taken care of to conform to my vision. That leads to code like this:

class Engine
{
    Battery _battery = new Battery();
    Alternator _alternator = new Alternator();
    Transmission _transmission = new Transmission();
}

class Car
{
    Engine _engine = new Engine();
    Cabin _cabin = new Cabin();
    List _tires = new List() { new Tire(), new Tire(), new Tire(), new Tire() };

    Car()
    {

    }

    void Start()
    {
        _engine.Start();
    }
}

So, in command and control style, I just tell my classes that I want a car, and my wish is their command. I don’t worry about what engine I want or what transmission I want or anything. Those details are taken care of for me. But I also don’t have a choice. I have to take what I’m given.

Since my linked post addresses the disadvantages of this approach, I won’t rehash it here. Let’s assume, for argument’s sake, that dependency inversion is preferable. Unit testing pushes you toward dependency inversion.

The reason for that is well illustrated by thinking about testing Car’s “start” method. How would we test this? Well, we wouldn’t. There’s only one line in the method and it references something completely hidden from us. But, if we changed Car and had it receive an engine through its constructor, we could easily create a friendly/mock engine and then make assertions about it after Car’s start method was called. For example, maybe Engine has an “IsStarted” property. Then, if we inject Engine to Car, we have the following simple test:

var myEngine = new Engine();
var myCar = new Car(myEngine);
myCar.Start();

Assert.IsTrue(myEngine.IsStarted);

After you spend some time unit testing regularly, you’ll find that you come to look at the new keyword with suspicion that you never did before. As I write code, if I find myself typing it, I think “either this is a data transfer object or else there better be a darned good reason for having this in my class.”

Dependency-inverted code is better code. I can’t say it any plainer. When your code is inverted, it becomes easier to maintain and requirements changes can be absorbed. If Car takes an Engine instead of making one, I can later create an inheritor from Engine when my requirements change and just give that to Car. That’s a code change of one modified line and a new class. If Car creates its own Engine, I have to modify Car any time something about Engine needs to change.

Unit Testing Encourages Use of Interfaces

By their nature, interfaces tend to be easier to mock than simple instances–even virtual ones. While I can’t speak to every mocking framework out there, it does seem to be a rule that the easiest way to mock things is using interfaces. So when you’re testing your code, you’ll tend to favor interfaces when all things are equal, since that will make test writing easier.

I believe that this favoring of interfaces is helpful for the quality of code as well. Interfaces promote looser coupling than any other way of maintaining relationships between objects. Depending on an interface instead of a concrete implementation allows decoupling of the “what” from the “how” question when programming. Going back to the engine/car example, if I have a Car class that depends on an Engine, I am tied to the Engine class. It can be sub-classed, but nevertheless, I’m tied to it. If its start method cannot be overridden and throws exceptions, I have to handle them in my Car’s start method.

On the other hand, depending on an engine interface decouples me from the engine implementation. Instead of saying, “alright, specific engine, start yourself and I’ll handle anything that goes wrong,” I’m saying, “alright, nameless engine, start yourself however it is you do that.” I don’t necessarily need to handle exceptions unless the interface contract allows them. That is, if the interface contract stipulates that IEngine’s start method should not throw exceptions, those exceptions become Engine’s responsibility and not mine.

Generally speaking, depending on interfaces is very helpful in that it allows you to make changes to existing code bases more easily. You’ll come to favor addressing requirements changes by creating new interface implementations rather than by going through and modifying existing implementations to handle different cases.

Regularly Unit Testing Makes You Proactive Instead of Reactive About the Unexpected

If you spend a few months unit testing religiously, you’ll find that a curious thing starts to happen. You’ll start to look at code differently. You’ll start to look at x.y() and know that, if there is no null check for x prior to that call, an exception will be thrown. You’ll start to look at if(x < 6) and know that you’re interested in seeing what happens when x is 5 and x is 6. You’ll start to look at a method with parameters and reason about how you would handle a null parameter if it were passed in, based on the situation. These are all examples of what I call “proactive,” for lack of a better term. The reactive programmer wouldn’t consider any of these things until they showed up as the cause of a defect.

This doesn’t happen magically. The thing about unit tests that is so powerful here is that the mistakes you make while writing the tests often lead you to these corner cases. Perhaps when writing the test, you pass in “null” as a parameter because you haven’t yet figured out what, exactly, you want to pass in. You forget about that test, move on to other things, and then later run all of your tests. When that one fails, you come back to it and realize that when null is passed into your method, you dereference it and generate an unhandled exception.

As this goes on over the course of time, you start to recognize code that looks like it would be fragile in the face of accidental invocations or deliberate unit test exercise. The unit tests become more about documenting your requirement and guarding against regression because you find that you start to be able to tell, by sight, when code is brittle.

This is true of unit tests because the feedback loop is so tight and frequent. If you’re writing some class without unit tests, you may never actually use your own class. You write the class according to what someone writing another class is going to pass you. You both check in your code, never looking at what happens when either of you deviate from the expected communication. Then, three months later, someone comes along and uses your class in another context and delivers his code. Another three months after that, a defect report lands on your plate, you fire up your debugger, and figure out that you’re not handling null.

And, while some learning will occur in this context, it will be muted. You’re six months removed from writing that code. So while you learn in principle that null parameters should be handled, you aren’t getting feedback. It’s essentially the difference between a dieter having someone slap his hand when he reaches for a cookie, or weigh him six months later and tell him that he shouldn’t have eaten that cookie six months ago. One is likely to change habits while the other is likely to result in a sigh and a “yeah, but are ya gonna do?”

Conclusion

I can probably think of other examples as well, but this post is already fairly long. I sincerely believe that the simple act of writing tests and getting immediate feedback on one’s code makes that person a better programmer more quickly than ignoring the tests. And, if you have a department where your testers are all writing tests, they’re becoming better designers/programmers and adopting good practices while doing productive work and raising the confidence level in the software that they’re producing.

I really cannot fathom any actual disadvantage to this practice. To me, the “obviously” factor of this is now on par with whether or not wearing a seat belt is a good idea.

By

Getting Started With Moles

Microsoft Moles seems pretty cool, so I’ve decided to set it up and play with it. However, because it’s a research project, the documentation isn’t exactly thorough. I found some videos and various tutorials but nevertheless had a lot of digging in order to get it working. I’m going to document some of the hurdles here for posterity and for myself, in case I need to do this again.

Installation

First off, I’ve been using Pex for a while now, so I just assumed that I’d have Moles installed. It appeared as though this wasn’t the case when I watched a Moles startup video. You see, I didn’t have the template that he references in the video, and I got no joy from doing “devenv /installvstemplates” (which, by the way, is very handy if you find that you’re missing some template that should be installed for whatever reason). As it turns out, the reason I don’t have it is because it’s apparently now deprecated and not the way things are done. I didn’t find this out until after I had un-installed and re-installed Pex and Moles, so I don’t know whether or not Moles was originally functioning with Pex. Probably, but who knows?

Getting Started

After that, I browsed a few posts like this one and saw that the thing I should be going for is getting these M types up and recognized by Intellisense. For instance, I want to test a class that’s using the System.IO.Directory class, so what I want is “MDirectory” for creating Moles. I got the sense that the thing to do here was to right-click on the library reference in question (i.e. “System”) and add a Moles assembly. This is wrong. Very wrong. I got about 25 compiler errors after I tried to build.

By trial and error (read: luck), I discovered that you want to right-click on “References” in your test project and “Add Moles Assembly For mscorlib.”

Adding Moles for mscorlib

Type was not instrumented…huh?

Once I’d gotten my M types in place, I was set to go. So I coded up a test that should pass if I implemented the delegate logic correctly, and I ran it. At this time, I got an exception in the test run. In opening the test result details, the Moles team was actually helpful enough to include this comment in the results:

“To resolve this issue, add the following attribute in the test project:

using Microsoft.Moles.Framework;
[assembly: MoledType(typeof(System.IO.Directory))]”

That’s pretty helpful. So, okie-dokie…and, viola! It passed.

Actual Use

If you understand functions and delegates, there’s not a lot to actual use. Once I’d gotten the annoyances out of the way, using this was actually very easy, and I found it to be quite powerful.

One thing I noticed is that it does introduce a bit of overhead when you start your test run. I’m keeping my eye on this to see if it’s a one-time, up-front cost, or if it scales and causes your test runs to be slower as you use it more.

Moling your own classes

This was fairly straightforward if you’re moling your public classes. You just right-click on your project under test in the test project’s references and select “Add Moles Assembly.” It gets a little more interesting if you want to mole internal classes.

To do this, open up the AssemblyInfo.cs class of your project under test and add an [assembly: InternalsVisibleTo(“{YourProjectNamespace}.Moles”)]. In order to know exactly what to add, just mimic the new reference that appears in your test project after you add a new moles assembly.

Moling constructors

For most instance based methods, the declaration looks something like:

MStreamReader.AllInstances.ReadToEnd = (o) => { return @"C:\ and some other stuff"; };

That is, you’ll get pretty used to the “AllInstances,” it appears. But one gotcha is that you mole a constructor like this:

MStreamReader.ConstructorString = (sr, s) => { };

That’s not really that big of a gotcha, but I’m just kind of documenting here all the things that I had to go poking around to figure out.

By

MS Test Report Generator

I’ve been working on a side project for a while, and today I uploaded it to Source Forge. The project is a tool that takes XML results generated by MS Test runs and turns them into HTML-based reports. I believe that TFS will do this for you if you have fully Microsoft-integrated everything, but, in the environment I’m currently working in, I don’t.

So I created this guy.

Back Story

I was working on a build process with a number of constraints that predated my tenure on the process itself. My task was to integrate unit tests into the build and have the build fail if the unit tests were not all in a passing state. The unit test harness was MS Test and the build tool was Final Builder.

During the course of this process, some unit tests had been in a failing state for some period of time. Many of these were written by people making a good faith effort but nibbling at an unfortunate amount of global state and singletons. These tests fail erratically and some of the developers that wrote them fell out of the habit of running the tests, presumably due to frustration. I created a scheme where the build would only run tests decorated with the “TestType” attribute “Proven”. In this fashion, people could write unit tests, ‘promote’ them to be part of the build, and have it as an opt-in situation. My reasoning here is that I didn’t want to deter people who were on the fence about testing by having them be responsible for failing the build because they didn’t know exactly what they were doing.

After poking around some, I saw that there was no native final builder action that accomplished what I wanted– to execute a certain subset of tests and display a report. So I created my own batch scripts (not included in the project) that would execute the MS Test command line executable with filtering parameters. This produces an XML based output. I scan that output for the run result and, if it isn’t equal to “Passed”, I fail the build. From there, I generate a report using my custom utility so that people can see stats about the tests.

Report Screenshot

Design

During the course of my spare time, I decided to play around with some architectural concepts and new goodies added to C# in the .NET 4.0 release. Specifically, I added some default parameters and experimented with co-variance and contra-variance. In terms of architecture, I experimented with adding the IRepository pattern to an existing tiered architecture.

On the whole, the design is extensible, flexible, and highly modular. It covered about 99.7% by unit tests and, consequently, was complete overkill for a little command line utility designed to generating an HTML report. However, I was thinking bigger. Early on, I decided I wanted to put this on Source Forge. The reason I designed it the way I did was to allow for expansion of the utility into a GUI-driven application that can jack into a server database and maintain aggregate unit testing statistics on a project. Over the course of time, you can track and get reports on things like test passing rate, test addition rate, which developers write tests, etc. For that, the architecture of the application is very well suited.

The various testing domain objects are read into memory, and the XML test file and HTML output file are treated as just another kind of persistence. So in order to adapt the application in its current incarnation to, say, write the run results to a MySQL or SQL server database, it would only be necessary to add a few classes and modify main to persist the results.

Whether I actually do this or not is still up in there air, and may depend upon how much, if any, interest it generates on Source Forge. If people would find it useful, both in general and specifically on projects I work on, I’m almost certain to do it. If no one cares, I have a lot of projects that I work on.

How to Access

You can download a zipped MSI installer from SourceForge.

If you want to look at the source code, you can do a checkout/export on the following SVN address: https://mstestreportgen.svn.sourceforge.net/svnroot/mstestreportgen

That will provide read-only access to the source.

By

Test Driven Development

All In?

It seems to me that most or many treatises on best practices for software engineering in this day and age advocate for Test Driven Development (TDD). I have read about this methodology, both on blogs and in a book that I ordered from Amazon (Test Driven Development By Example). I have put it into practice in varying situations, and I enjoy the clarity that it brings to the development process once you manage to wrap your head around the idea of writing non-compiling, non-functioning tests prior to writing actual code.

That said, it isn’t a process I prefer to follow to the letter. I have tried, and it doesn’t suit the way I do my best development. I have read accounts of people saying that it simply takes some getting used to and that people object to it because they want to “just code” and are less productive downstream in the development process because of this, but I don’t think that’s the case for me. I don’t delude myself into thinking that I’m more efficient in the long run by jumping in and not thinking things through. In fact, quite the opposite is true. I prototype with throw-away code before I start actual development (architecture and broad design notwithstanding–I don’t architect applications by prototyping, I’m referring to adding or reworking a module or feature in an existing, architected application or creating a new small application)

Prototyping and TDD

As far as I can tell, the actual process of TDD is agnostic as to whether or not it is being used in the context of prototyping. One can develop a prototype, a one-off, a feature, or a full-blown production application using TDD. My point in this post is that I don’t find TDD to be helpful or desirable during an exploratory prototyping phase. I am not entirely clear as to whether this precludes me from being a TDD adherent or not, but I’m not concerned about that. I’ve developed and refined a process that seems to work for me and tends to result in a high degree of code coverage with the manual tests that I write.

The problem with testing first while doing exploratory prototyping is that the design tends to shift drastically and unpredictably as it goes along. At the start of the phase, I may not know all or even most of the requirements, and I also don’t know exactly how I want to accomplish my task. What I generally do in this situation is to rig up the simplest possible thing that meets one or a few of the requirements, get feedback on those, and progress from there. As I do this, additional requirements tend to be unearthed, shifting and shaping the direction of the design.

During this prototyping phase, manual and automated refactoring techniques are the most frequently used tools in my toolbox. I routinely move methods to a new class, extract an interface from an existing class, delete a class, etc. I’ve gotten quite good at keeping version control up to date with wildly shifting folder and file structures in the project. Now, if I were to test first in this mode of development, I would spend a lot of time fixing broken tests. I don’t mean that in the sense of the standard anti-unit-testing “it takes too long” complaint, but in the sense that a disproportionate number of tests I would write early on would be discarded. And while the point of prototyping is to write throw-away code, I don’t see any reason to create more code to throw away than necessary to help you see the right design.

TDD and Prototyping in Harmony

So, here, in a nutshell, is the process that I’ve become comfortable with for new, moderately sized development efforts:

  1. Take whatever requirements I have and start banging out prototype code.
  2. Do the simplest possible thing to satisfy the requirement(s) I’ve prioritized as highest (based on a combination of difficulty, prerequisites and stakeholder preference).
  3. Eliminate duplication, refine, refactor.
  4. Satisfy the next requirement(s) using the same criteria, generalizing specific code, and continuing to refine the design.
  5. Repeat until a rough design is in place
  6. Resist the urge to go any further with this and start thinking of it as production code. (I usually accomplish this by doing the preceding steps completely outside of the group’s version control.)

At this point, I have a semi-functional prototype that can be used for a demo and for conveying the general idea. Knowing when to shift from prototyping to actual development is sort of an intuitive rather than mechanical process, but usually for me, it’s the point at which you could theoretically throw the thing into production and at least argue with eventual users that it does what it’s supposed to. At this point, there is no guarantee that it will be elegant or maintainable, but it more or less works.

From there, I start my actual development in version control. This is when I start to test first. I don’t open up the target solution and dump my prototype files in wholesale or even piecemeal. By this point, I know my design well, and I know its shortcomings and how I’d like to fix and address those. I also generally realized that I’ve given half of my classes names that no longer make sense and that I don’t like the namespace arrangements I’ve set up. It’s like looking at your production code and having a refactoring wish-list, except that you can (and in fact have to) actually do it.

So from here, I follow this basic procedure:

  1. Pick out a class from the prototype (generally starting with the most abstract and moving to the one that depends most on the other ones).
  2. For each public member (method or property), identify how it should behave with valid and invalid inputs and with the object in different states.
  3. Identify behaviors and defaults of the object on instantiation and destruction.
  4. Create a test class in version control and write empty test methods with names that reflect the behaviors identified in the previous steps.
  5. Create a new class in version control and stub in the public properties and methods from the previous steps.
  6. Write the tests, run them, and watch them fail
  7. Go through each test and do the simplest thing to make it pass.
  8. While making the test pass, eliminate duplication/redundancy and refine the design.

This is then repeated for all classes. As a nice ancillary benefit, doing this one class at a time can help you catch dependency cycles (i.e. if it’s really the case that class should A has a reference to class B and vice-versa, so be it, but you’ll be confronted with that and have to make an explicit decision to leave it that way).

Advantages to the Approach

I sort of view this as having my cake and eating it too. That is, I get to code “unencumbered” by testing concerns in the beginning, but later get the benefits of TDD since nothing is actually added to the official code base without a test to accompany it. Here is a quick list of advantages that I see to this process:

  • I get to start coding right away and bang out exploratory code.
  • I’m not writing tests for requirements that are not yet well understood or perhaps not even correct.
  • I’m not testing a class until I’m sure what I want its behavior to be.
  • I separate designing my object interactions from reasoning about individual objects.
  • I get to code already having learned from my initial design mistakes or inefficiencies.
  • No code is put into the official codebase without a test to verify it.
  • There is a relatively low risk of tests being obsolete or incorrect.

Disadvantages

Naturally, there are some drawbacks. Here they are as I see them:

  • Prototyping can wind up taking more time than the alternatives.
  • It’s not always easy to know when to transition from prototype to actual work.
  • Without tests, regressions can occur during prototyping when a violent refactoring sweeps out some of the good with the bad.
  • Without tests, some mistakes in your prototype design might make it into your official version recreation.

I think that all of these can be mitigated and I firmly believe that the advantages outweigh the disadvantages.

When not to do this

Of course, this method of development is not always appropriate. During a defect fixing cycle, I think tried and true TDD works the best. Something is wrong, so write a test for the right thing and modify the code until it happens — there’s no need to prototype. This process is also inappropriate if you have all of your requirements and a well-crafted design in place, or if you’re only making small changes. Generally, I do this when I’m tasked with new feature or project implementation that has a high level design already, and is going to take weeks or months.

By

Microsoft Pex

I’ve recently started to play around with Microsoft Pex, and I’m discovering (1) that there is a non-trivial learning curve; and (2) that it seems like a pretty powerful tool. As best I can tell at this early stage (and admittedly, I’m certainly a neophyte), there seem to be two major components. The first is the notion of parameterized unit testing (PUT) and the second is automated test generation.

For anyone not familiar with the idea of parameterized unit testing, it’s an idea that rather than the first bit of code below, you can do something that conceptually mirrors the second bit of code, except with better syntax. In the .NET world, I believe that MBUnit supplies this syntax natively (in the form of “row test”) and that users are left to reconstruct it with NUnit and MS Test. I’m too far removed at the moment from unit testing suites in Java or C++ to comment with any authority on those languages and PUTs.

public void SingleValueTest()
{
    var myFoo = new Foo();
    Assert.AreNotEqual(1, myFoo.FooIntProperty);
}

public void ParameterizedTest(int rangeStart, int rangeStop)
{
    var myFoo = new Foo();
    for(int index = rangeStart; index < rangeStop; index++)
        Assert.AreNotEqual(index, myFoo.FooIntProperty);
}

This second example is likely something that we’ve all done, with or without some supporting syntax of the test library. Obviously, unit test methods don’t take parameters, so you might have coded up some standard helper like the method above and call it from actual, decorated test methods (or, if you’re like me, you created an assert extension class that grows as you go). Pex creates a series of method decorating attributes and some of them appeared oriented toward generating tests with these semantics–they pepper your SUT (system under test) with ranges of values and see how it behaves.

But the really snazzy part of Pex that you can’t mimic with helper methods is that it analyzes your code, looking for interesting values. That is, you don’t supply “rangeStart” and “rangeStop” to Pex–it deduces them on its own, based on static and dynamic analysis of your code, and tries them out. The first time I ran Pex, it found all sorts of exception-generating conditions in code that I had thought to be fairly bulletproof, given that I had added a whole slew of contracts using Code Contracts to check preconditions and invariants and enforce post conditions. While somewhat humbling, that is extremely helpful. It’s better than the best possible manual code review (for seeing potential exceptions, at least), and better than the manual tests you generate. It’s automatically executing your code and guiding it towards exception-generating conditions.

Making things even better is the fact that Pex presents the results and gives you the option to actually generate unit tests (in your framework of choice) based on the exceptions it finds. It refers to this as promoting the tests. So the process flow is that you run a “Pex Exploration” which does dynamic analysis (mixed, I believe, with some forms of static analysis) to find exceptional conditions. It then shows you a list of the results of its parameterized runs and whether they passed, failed, or were inconclusive (there is a configurable timeout for each exploration) and allows you to pick and choose which results that you want to promote to unit tests saved in files in your codebase.

One thing that I find a bit off-putting is the fact that the tests it generates are strewn across a number of “.g.cs” partial classes, at least with MS Test. That is really just a matter of personal taste, and I can probably get used to it. It seems generate the partials based on the method in your code that it’s exploring and handle the setup for the call there, and then invoke the main class to actually call the method in question. This is kind of confusing, as the setup occurs in a more specific class and the actual test assert is somewhere else. However, when you open the test from the test result window (which is what I’ve been doing), you can set a breakpoint and step through the heavily attribute-decorated Pex boilerplate, winding up in your own code and seeing the exceptions. And that’s really what matters.

As I play more with this apparently slick and powerful tool, I’ll have more to say about it. I’m playing with it in conjunction with Code Contracts, and there seems to be a natural synergy between these products. I’ll have more to post on both subjects in this series of posts.