A Look at Some Unit Test Framework Options for .NET
Editorial note: I originally wrote this post for the Stackify blog. You can check out the original here, at their site. While you’re there have a look at their offerings, Prefix and Retrace.
If you enjoy the subject of human cognitive biases, you should check out the curse of knowledge. When dealing with others, we tend to assume they know what we know. And we do this when no justification for the assumption exists.
Do you fancy a more concrete example? Take a new job and count how many people bombard you with company jargon and acronyms, knowing full well you just started a few hours ago. This happens because these folks cannot imagine not knowing these things without expending considerable mental effort.
Why do I lead with this in a post about unit test frameworks? Well, it seems entirely appropriate to me. I earn my living as an IT management and strategy consultant, causing me to spend time at many companies helping them improve software development practice. Because of this, I have occasion to see an awful lot of introductions to unit testing. And these introductions usually subconsciously assume knowledge of unit testing.
“It’s easy! Just pick a unit test runner and a coverage tool, and get those setup. Oh, you’ll also probably want to pick a mocking framework, and here are some helpful Nuget packages. Anyway, now just write a test. We’ll start with a calculator class…”
Today, I will do my best to spare you that. I have some practice with this, since I write a lot, publish courses, and train developers. So let’s take a look at test frameworks.
What Are Unit Tests?
Thought you’d caught me there, didn’t you? Don’t worry. I won’t just assume you know these things.
Let’s start with unit testing in its most basic form, leaving all other subjects aside. You want to focus on a piece of functionality in your code and test it in isolation. For example, let’s say that we had the aforementioned Calculator class and that it contained an Add(int, int) method. Let’s say that you want to write some code to test that method.
public class CalculatorTester { public void TestAdd() { var calculator = new Calculator(); if (calculator.Add(2, 2) == 4) Console.WriteLine("Success"); else Console.WriteLine("Failure"); } }
No magic there. I just create a test called “CalculatorTester” and then write a method that instantiates and exercises Calculator.Add(). You could write this knowing nothing about unit testing practice at all. And, if someone had told you to automate the testing of Calculator.Add(), you may have done this exact thing.
Congratulations. You have written a unit test. I say this because it focuses on a method and tests it in isolation.
What Are Unit Test Frameworks?
Well, as you can imagine, having this sort of testing across your entire codebase could prove cumbersome. You’d write lots of classes just like this one and then… what? You’d look at all of the console output for failures, only to discover that failure proved pretty inscrutable.
Most likely you would then have the clever idea to include the name of the test in the output so that you could see what had failed. From there, you might start to add some sort of GUI-like feedback to the results, to present them in a more readable fashion. From there, who knows? Maybe you write a Visual Studio plugin or maybe you find a way to incorporate this test suite into your build?
Well, it turns out that if you did all that stuff, you would have built yourself a unit testing framework. Let’s take a look now at what some code written for an actual unit test framework (MS Test) looks like.
[TestClass] public class CalculatorTests { [TestMethod] public void TestMethod1() { var calculator = new Calculator(); Assert.AreEqual<int>(4, calculator.Add(2, 2)); } }
Unlike the last snippet, we have a bit of magic here. Notice the attributes, TestClass and TestMethod. Those exist simply to tell the framework to pay attention to them when executing the unit test suite. When you want to get results, you invoke the unit test runner, and it executes all methods decorated like this, compiling the results into a visually pleasing report that you can view.
So, with that background established, let’s take a look at your major test runner options.
MSTest
First, since I’ve alreadry mentioned it, I’ll lead with MSTest. MSTest was actually the name of a command line tool for executing tests, so we’re really talking about something called the “Visual Studio Unit Testing Framework.” But veterans will colloquially call it MSTest, so let’s use that here.
MSTest ships with Visual Studio, so you have it right out of the box, in your IDE, without doing anything. This lack of friction to getting started is arguably its killer feature. The preferred .NET pattern for unit tests is to have a test project for each production (regular) project in your codebase. With MSTest, getting that setup is as easy as File->New Project. Then, when you write a test, you can right click on it and execute, having your result displayed in the IDE. Pretty neat, huh? And you can also see code coverage without installing any other tools.
On the con side, one of the most frequent knocks on MSTest is that of performance. People find the experience can be sluggish. On top of that, many people struggle with interoperability. After all, Microsoft makes it to integrate with other Microsoft/Visual Studio stuff, so making it work with third party things would probably not rate as a priority.
NUnit
Next up, I’ll talk about NUnit. Unit test frameworks have a history dating back almost 30 years, so they long predated .NET. NUnit started out as a port from Java’s JUnit, but the authors eventually redid it to be more C# idiomatic. So the tool has the rich history of unit testing behind it, but with an appropriately C# flavor.
Because of its independent history, NUnit also has the distinction of interoperating nicely with other tools, such as non-Microsoft build platforms and custom test runners. On top of that, NUnit also has a reputation for fast testing running, and it has some nice additional features as well, including test annotations allowing easy specification of multiple inputs to a given test.
The main downside here is that it doesn’t integrate into Visual Studio the way that MSTest does. Using it means doing extra work, and installing extra tools, regardless of how easy those tools’ authors may make the process.
xUnit.NET
The last option that I’ll cover is xUnit.NET (I’ll just call it xUnit for brevity, not to be confused with the xUnit category of test tools). One of the creators of the idiomatic version of NUnit went on to create xUnit. xUnit has a relatively innovative for users to reason about their tests, dividing tests into “facts” and “theories” to distinguish between “always true” and “true for the right data,” respectively.
xUnit earns points for creating extremely intuitive terminology and ways to reason about the language of tests. On top of that, the tool has a reputation for excellent extensibility. Another awesome feature of xUnit is actually not a feature of the software, but a feature of the authors. They have a reputation for commitment, responsiveness and evangelism.
On the con side, some users seem to wish the tool had more documentation. Beyond that, the community seems pretty enthusiastic and it’s hard to find a lot of detractors. This may speak to an implicit con, however. It has a small but loyal core of users because it requires a different way of thinking about some things and perhaps a bit more learning.
Just Get Started
I’ll close here by stating something emphatically. If you’re looking to start your unit testing journey, do NOT get bogged down in trying to pick one of these frameworks. Just pick one — roll some dice, flip a few coins — whatever.
They have their pros and cons, their detractors and supporters. But every one of them is better, by far, than continuing not to write unit tests. You can always re-evaluate and switch later if you need to. I’ve used a lot of different unit test frameworks in my career and never felt locked in or concerned about it.
So get started as quickly as you can. And before you know it, you’ll suffer the curse of knowledge with unit testing and not be able to conceive of someone not versed in the pros and cons of their favorite testing framework.
Wait…WHAT!? You’re still seeing professional developers out there who don’t know how to write a unit test? Unit testing has been around for a long time too (I think you had 30 years but could be longer). Begets so many questions: What’s the root cause of this calamity? Are folks not seeing value there? What are the outcomes after you introduce it to them? What do they do instead of unit tests to ensure the correctness of their code?
Examining the phenomenon might be worth a post in and of itself, but I think it fundamentally has something to do with coming to write code inductively from “hello world.” In other words, imagine knowing nothing about programming. Your first foray might be HTML or some kind of markup. You add a few tags, see Hello World in the browser, and eureka! Or, maybe you start with a shell script and do the same. From there, you build and grow your understanding a little at a time, and each time you add to your work, you seek visual confirmation or… Read more »
I can’t imagine that anyone – college or self taught – learns to program by writing unit tests. Catch 22! For example, if hello world output iff some condition held true it still wouldn’t constitute a unit test…the unit test of a hello world is that it outputs “Hello World!”. Perhaps all hello worlds should first teach the user how to output then teach the user how to verify programmatically…that kind of tutorial would have more professional merit! As a bonus, the next tutorial should teach how to write the test first (if I can get away with suggesting TDD… Read more »
@Phil – I agree with your observation that there’s a gap for the many who aren’t explicitly introduced to Unit Testing. But I think this gap is slowly being closed by those who create the tooling and templates. For instance, most of the newer “new project” templates in Visual Studio will offer to stub out a Unit Test project, complete with a few basic methods and tests for them. Also, on the web client-side, tools like the Angular CLI also include unit tests when creating a new component or module. Hopefully this sort of effort will eventually assist those who… Read more »