DaedTech

Stories about Software

By

Static Analysis and The Other Kind of False Positives

Editorial Note: I originally wrote this post for the NDepend blog.  You can check out the original here, at the NDepend site.  If you’re a fan of static analysis tools, do yourself a favor and take a look at the NDepend offering while you’re there.

A common complaint and source of resistance to the adoption of static analysis is the idea of false positives.  And I can understand this.  It requires only one case of running a tool on your codebase and seeing 27,834 warnings to color your view on such things forever.

There are any number of ways to react to such a state of affairs, though there are two that I commonly see.  These are as follows.

  1. Sheepish, rueful acknowledgement: “yeah, we’re pretty hopeless…”
  2. Defensive, indignant resistance: “look, we’re not perfect, but any tool that says our code is this bad is nitpicking to an insane degree.”

In either case, the idea of false positives carries the day.  With the first reaction, the team assumes that the tool’s results are, by and large, too advanced to be of interest.  In the second case, the team assumes that the results are too fussy.  In both of these, and in the case of other varying reactions as well, the tool is providing more information than the team wants or can use at the moment.  “False positive” becomes less a matter of “the tool detected what it calls a violation but the tool is wrong” and more a matter of “the tool accurately detected a violation, but I don’t care right now.”  (Of course, the former can happen as well, but the latter seems more frequently to serve as a barrier to adoption and what I’m interested in discussing today).

Is this a reason to skip the tool altogether?  Of course not.  And, when put that way, I doubt many people would argue that it is.  But that doesn’t stop people form hesitating or procrastinating when it comes to adoption.  After all, no one is thinking, “I want to add 27,834 things to the team’s to do list.”  Nor should they — that’s clearly not a good outcome.

Fumigation

With that in mind, let’s take a look at some common sources of false positives and the ways to address them.  How can you ratchet up the signal to noise ratio of a static analysis tool so that is valuable, rather than daunting?

Start with the Critical

One of the most obvious things you can do out of the gate is to prioritize.  Many code analysis tools, like defect tracking systems, will weight their findings according to some scale of importance or criticality.  If this is available (e.g. NDepend’s “Critical Rules”), it makes for an excellent starting point with static analysis.

Adjust the tool, in the early stages, to show only the highest priority items.  This will cut the initial figure substantially and you can have a lot more confidence that you’re not chasing down trivialities when you should be focused on something else.  Create a plan or timeline for bringing the critical count down to zero, and then move on with a victory behind you.

Choose What Matters To You

It may be that the tool you’re using doesn’t actually have any concept of severity.  Or, perhaps you’ve just followed the advice in the previous section and are looking for a new way to slice your work.  Maybe you just flat out disagree with some of the critical items highlighted by the tool (though be careful with this).

Whatever the case may be, at some point you should sit down, go through all of the tool’s rules one by one, and decide which ones you view as important.  Given them ranks of your own on some sliding scale, and then enable them in waves.  Once you’ve knocked out the priority 1 issues, move on to 2, and then 3, and so on.  Almost every static analysis tool I’ve encountered lets you at least turn rules on and off, so use that to your advantage.

Jettison Items That You Don’t Like

Up to this point, it’s been a question of prioritization.  But at some point, you may encounter rules that aren’t relevant to you, are not at all important to you, or with which simply disagree (again, careful here).  You are not going to be in lockstep with every maker of every tool out there on what makes sense for your code.

If there’s a warning that “best practice” dictates interfaces begin with the letter “I” and that is just not in your coding standards, turn that rule off and don’t look back.  Maybe you don’t care about usings/includes/packages being sorted alphabetically at the top of the file and don’t want to hear about it.  That’s fine.  Allow yourself some leeway with turning rules off indefinitely.

Adjust Thresholds

With many tools, you have more options than just turning a rule on or off.  Frequently, there are adjustable thresholds.  For instance, the tool may warn you if a method is longer than 10 lines out of the box.

Perhaps that’s a little aggressive for your taste, at least in the short term.  After all, that produces 7,499 warnings when you turn it on.  But if you set that threshold to 50, that number dips to a much more manageable 23 warnings.  And a 23 item to-do list is one that doesn’t fall into “that’d be great but never gonna happen.”  You can handle 23.

Tune the tool’s thresholds using the same reasoning that you used earlier with prioritizing issues.  The general idea is to segment the work into manageable chunks by adjusting the settings.  Thresholds help here as much as turning off swaths of checks.

But Do Your Homework

I have offered words of caution a few times, and I will close by elaborating on those.  I hope that reading this has made the prospect of static analysis seem less daunting (or even onerous) to you.  In my experience, sometimes all that’s needed is permission to view the tools as aides and enablers rather than judges and juries.  It’s okay if you turn some things off!

And yet, you need to be really careful not to take this too far.  Imagine the reaction in the beginning where someone dismissed the tool because it’s “way too nitpicky.”  Now imagine that same person with carte blanche to turn off warnings with which he disagreed.  He may tune the tool from the original 27,834 to a manageable 4 items that he grudgingly concedes may not be quite perfect, unlike the other 27,830 cases where the tool was just clearly wrong.

Don’t be this person.

ExpertBeginner

When you disagree with the tool, the tool is probably right.  Think back to a time when you couldn’t get your code to compile and became convinced that the compiler was buggy.  Was it actually?  Or did it maybe turn out that you’d actually made some kind of mistake.  The people that write compilers tend to know the language pretty well.  And so too do the people who write analysis tools.  They have a lot of experience, a lot of study and a lot of data backing them.

There’s a lot of commonality with these tools and the compilers with warnings as well.  You ignore the warnings, to some extent, at your peril.  When you ignore compiler and analysis warnings, potential problems hide in your blind spot, waiting to reveal themselves.  It may be that some particular bit of noise is truly just that, but it may also be that you have an actual problem but lack the time to deal with it right now.

It’s okay not to have time to deal with things, and it’s okay to acknowledge that you can’t fix everything all at once.  Use the tools at your disposal, including static analysis tools, to support you as you create a manageable workload and steady improvement.  But don’t let the possibility of being temporarily overwhelmed stop you from improving at all.

4 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Jonathan Snow
Jonathan Snow
7 years ago

Interesting thoughts. Thanks for sharing.

Erik Dietrich
7 years ago
Reply to  Jonathan Snow

Thanks! I’m glad you liked.

Thomas Lassanske
Thomas Lassanske
7 years ago

Even after setting desired thresholds and enabling or disabling certain checks, there may be a fair number of cases where the tool spits out warnings or even errors that are not actually a problem for your code (after careful analysis), but you still want to know about new findings in the future. Thus, I prefer to make changes or rearrange code to change the tool’s result so that we can achieve “green results across the board”, so that future runs will flag new cases that aren’t lost in a stream of “we already know about that” results. This also paves… Read more »

Erik Dietrich
7 years ago

Broken window theory. Definitely sounds like good practice to me.