DaedTech

Stories about Software

Apr
21

By

How to Keep Method Size Under Control

Category: Abstractions Tags: , | 13 Comments

Do you ever open a source code file and see a method that starts at the top of your screen and kind of oozes its way to the bottom with no end in sight? When you find yourself in that situation, imagine that you’re reading a ticker tape and try to guess at where the method actually ends. Is it a foot below the monitor? Three feet? Does it plummet through the floor and into the basement, perhaps down past the water table and into the earth’s mantle?

TickerMonitor

Visualized like this, I think everyone might agree that there’s some point at which the drop is too far, though there’s likely some disagreement on where exactly this is. Personally, I used to subscribe to the “fits on a screen” heuristic and would only start looking to pull out methods if it got beyond that. But in more recent years, I think even smaller. How small? I dunno–five or six lines, max. Small enough that you’ll only ever see one try-catch or control flow statement in there. Yeah, seriously, that small. If you’re thinking it sounds kind of crazy, I get that, but give it a try for a while. I can almost guarantee that you’ll lose your patience for looking at methods that cause you to think, “wait, where was loopCounter declared again–before the second or third while loop?”

If you accept the premise that this is a good way to do things or that it might at least be worth a try, the first thing you’ll probably wonder is how to go about doing this from a practical standpoint. I’ve definitely encountered people and even whole groups who considered method sizes like this to be impractical. The first thing you have to do is let go of the notion that classes are in some kind of limited supply and you have to be careful not to use too many. Same with modules, if your project gets big enough. The reason I say this is that having small methods means that you’re going to have a lot of them. This in turn means that they’re going to need to be spread to multiple classes, and those classes will occupy more namespaces and modules. But that’s okay. If you encounter a large application that’s well designed and factored, it’s that way because the application is actually a series of small, focused components working together. Monolithic doesn’t scale well.

Getting Down to Business

If you’ve prepared yourself for the reality of needing more classes organized into more namespaces and modules, you’ve really overcome the biggest obstacle to being a small-method coder. Now it’s just a question of mechanics and practice. And this is actually important–it’s not sufficient to just say, “I’m going to write a lot of methods by stopping at the fifth line, no matter what.” I guarantee you that this is going to create a lot of weird cross-coupling, unnecessary state, and ugly things like out parameters. Nobody wants that. So it’s time to look to the art of creating abstractions.

As a brief digression, I’ve recently picked up a copy of Uncle Bob Martin’s Clean Code: A Handbook of Agile Software Craftsmanship and been tearing my way through it pretty quickly. I’d already seen most of the Clean Coder video series, which covers some similar ground, but the book is both a good review and a source of new and different information. To be blunt, if you’re ever going to invest thirty or forty bucks in getting better at your craft, this is the thing to buy. It’s opinionated, sometimes controversial, incredibly specific, and absolute mandatory reading. It will change your outlook on writing code and make you better at what you do, even if you don’t agree with every single point in it (though I don’t find much with which to take issue, personally).

The reason I mention this book and series is that there is an entire section in the book about functions/methods, and two of its fundamental points are that (1) functions should do one thing and one thing only, and (2) that functions should have one level of abstraction. To keep those methods under control, this is a great place to start. I’d like to dive a little deeper, however, because “do one thing” and “one level of abstraction per function” are general instructions. It may seem a bit like hand-waving without examples and more concrete heuristics.

Extract Finer-Grained Details

What Uncle Bob is saying about mixed abstractions can be demonstrated in this code snippet:

public void OpenTheDoor()
{
    GrabTheDoorKnob();
    TwistTheDoorKnob();
    TightenYourBiceps();
    BendYourElbow();
    KeepYourForearmStraight();
}

Do you see what the issue is? We have a method here that describes (via sub-methods that are not pictured) how to open a door. The first two calls talk in terms of actions between you and the door, but the next three calls suddenly dive into the specifics of how to pull the door open in terms of actions taken by your muscles, joints, tendons, etc. These are two different layers of abstractions: one about a person interacting with his or her surroundings and the other detailing the mechanics of body movement. To make it consistent, we could get more detailed in the first two actions in terms of extending arms and tightening fingers. But we’re trying to keep methods small and focused, so what we really want is to do this:

public void OpenTheDoor()
{
    GrabTheDoorKnob();
    TwistTheDoorKnob();
    PullOpenTheDoor();
}

private static void PullOpenTheDoor()
{
    TightenYourBiceps();
    BendYourElbow();
    KeepYourForeArmStraight();
}

Create Coarser Grained Categories

What about a different problem? Let’s say that you have a method that’s long, but it isn’t because you are mixing abstraction levels:

public void CookQuesadilla()
{
    ChopOninons();
    ShredCheese();

    GetOutThePan();
    AddOilToPan();
    TurnOnTheStove();

    SprinkleOnionsAndCheeseOnTortilla();
    PutTortillaInPan();
    CookUntilFirm();
    FoldTortillaAndCookUntilBrown();
    FlipTortillaAndCookUntilBrown();
    RemoveCookedQuesadilla();

    RemovePanFromStove();
    ScrubPanWithBrush();
    ServeQuesadillas();
}

These items are all at the same level of abstraction, but there are an awful lot of them. In the previous example, we were able to tighten up the method by making the abstraction levels consistent, but here we’re going to actually need to add a layer of abstraction. This winds up looking a little better:

public void CookQuesadilla()
{
    PrepareIngredients();
    PrepareEquipment();
    PerformActualCooking();
    FinishUp();
}

private static void PrepareIngredients()
{
    ChopOninons();
    ShredCheese();
}
private static void PrepareEquipment()
{
    GetOutThePan();
    AddOilToPan();
    TurnOnTheStove();
}
private static void PerformActualCooking()
{
    SprinkleOnionsAndCheeseOnTortilla();
    PutTortillaInPan();
    CookUntilFirm();
    FoldTortillaAndCookUntilBrown();
    FlipTortillaAndCookUntilBrown();
    RemoveCookedQuesadilla();
}
private static void FinishUp()
{
    RemovePanFromStove();
    ScrubPanWithBrush();
    ServeQuesadillas();
}

In essence, we’ve created categories and put the actions from the long method into them. What we’ve really done here is create (or add to) a tree-like structure of methods. The public method is the root, and it had thirteen children. We gave it instead four children, and each of those children has between two and five children of its own. To tighten up methods, it’s perfectly viable to add “nodes” to the “tree” of your call stack. While “do one thing” is still a little elusive, this seems to be carrying us in that direction. There’s no individual method that you look at and think, “boy, that’s a lot of stuff going on.” Certainly its a matter of some art and taste, but this is probably a good way to think of it–organize stuff into hierarchical method categories until you look at each method and think, “I could probably memorize what that does if I needed to.”

Recognize that Control Flow Uses Up an Abstraction

So far we’ve been conceptually figuring out how to organize families of methods into well-balanced tree structures, and that’s taken us through some pretty mundane code. This code has involved none of the usual stuff that sends apps careening off the rails into bug land, such as conditionals, loops, assignment, etc. Let’s correct that. Looking at the code above, think of how you’d modify this to allow for the preparation of an arbitrary number of quesadillas. Would it be this?

public void CookQuesadillas(int numberOfQuesadillas)
{
    PrepareIngredients();
    PrepareEquipment();
    for(int i = 0; i < numberOfQuesadillas; i++)
        PerformActualCooking();
    FinishUp();
}

Well, that makes sense, right? Just like the last version, this is something you could read conversationally while in the kitchen just as easily as you do in the code. Prep your ingredients, then prep your equipment, then for some integer index equal to zero and less than the number of quesadillas you want to cook, increment the integer by one. Each time you do that, cook the quesadilla. Oh, wait. I think we just went careening into the nerdiest kitchen narrative ever. If Gordon Ramsey were in charge, he'd have strangled you with your apron for that. Hmm... how 'bout this?

public void CookQuesadillas(int numberOfQuesadillas)
{
    PrepareIngredients();
    PrepareEquipment();
    PerformActualCooking(numberOfQuesadillas);
    FinishUp();
}

private static void PerformActualCooking(int numberOfQuesadillas)
{
    for (int index = 0; index < numberOfQuesadillas; index++)
    {
        SprinkleOnionsAndCheeseOnTortilla();
        PutTortillaInPan();
        CookUntilFirm();
        FoldTortillaAndCookUntilBrown();
        FlipTortillaAndCookUntilBrown();
        RemoveCookedQuesadilla();
    }
}

Well, I'd say that the CookQuesadillas method looks a lot better, but do we like "PerformActualCooking?" The whole situation is an improvement, but I'm not a huge fan, personally. I'm still mixing control flow with a series of domain concepts. PerformActualCooking is still both a story about for-loops and about cooking. Let's try something else:

public void CookQuesadillas(int numberOfQuesadillas)
{
    PrepareIngredients();
    PrepareEquipment();
    PerformActualCooking(numberOfQuesadillas);
    FinishUp();
}

private static void PerformActualCooking(int numberOfQuesadillas)
{
    for (int index = 0; index < numberOfQuesadillas; index++)
        CookAQuesadilla();
}

private static void CookAQuesadilla()
{
    SprinkleOnionsAndCheeseOnTortilla();
    PutTortillaInPan();
    CookUntilFirm();
    FoldTortillaAndCookUntilBrown();
    FlipTortillaAndCookUntilBrown();
    RemoveCookedQuesadilla();
}

We've added a node to the tree that some might say is one too many, but I disagree. What I like is the fact that we have two methods that contain nothing but abstractions about the domain knowledge of cooking and we have a bridging method that brings in the detailed realities of the programming language. We're isolating things like looping, counting, conditionals, etc. from the actual problem solving and story telling that we want to do here. So when you have a method that does a few things and you think about adding some kind of control flow to it, remember that you're introducing a detail to the method that is at a lower level of abstraction and should probably have its own node in the tree.

Adrift in a Sea of Tiny Methods

If you're looking at this cooking example, it probably strikes you that there are no fewer than eighteen methods in this class, not counting any additional sub-methods or elided properties (which are really just methods in C# anyway). That's a lot for a class, and you may think that I'm encouraging you to write classes with dozens of methods. That isn't the case. So far what we've done is started to create trees of many small methods with a public method and then a ton of private methods, which is a code smell called "Iceberg Class." What's the cure for iceberg classes? Extracting classes from them. Maybe you turn the first two methods that prepare ingredients and equipment into a "Preparer" class with two public methods, "PrepareIngredients" and "PrepareEquipment." Or maybe you extract a quesadilla cooking class.

It's really going to vary based on your situation, but the point is that you take this opportunity pick nodes in your growing tree of methods and sub-methods and convert them into roots by turning them into classes. And if doing this leads you to having what seems to be too many classes in your namespace? Create more namespaces. Too many of those in a module? Create more modules. Too many modules/projects in a solution? More solutions.

Here's the thing: the complexity exists no matter how many or few methods/classes/namespaces/modules/solutions you have. Slamming them all into monolithic constructs together doesn't eliminate or even hide that complexity, though many seem to take the ostrich approach and pretend that it does. Your code isn't somehow 'simpler' because you have one solution with one project that has ten classes, each with 300 methods of 7,000 lines. Sure, things look simple when you fire up the IDE, but they sure won't be simple when you try to debug. In fact, they'll be much more complicated because your functionality will be hopelessly interwoven with weird temporal couplings, ad-hoc references, and hidden dependencies.

If you create large trees of functionality, you have the luxury of making the structure of the tree the representative of the application's complexity, with each node an island of simplicity. It is in these node-methods that the business logic takes place and that getting things right is most important. And by managing your abstractions, you keep these nodes easy to reason about. If you structure the tree correctly and follow good OOP design and practice, you'll find that even the structure of the tree is not especially complicated since each node provides a good representative abstraction for its sub-tree.

Having small, readable, self-documenting methods is no pipe dream. Really, with a bit of practice, it's not even very hard. It just requires you to see code a little bit differently. See it as a series of hierarchical stories and abstractions rather than as a bunch of loops, counters, pointers, and control flow statements, and the people that maintain what you write, including yourself, will thank you for it.

Apr
17

By

Language Basics from Unit Tests

Category: Language Agnostic Tags: , |

Let’s say that in a green field code base someone puts together a type that conceptually is a collection of non-integer values. For the sake of discussion, let’s call it a graph. A graph object might store a series of two-element tuples or perhaps a series of some value type like “point.” The graph might then perform operations on this data, such as IncreaseX() or IncreaseY() or Invert() or Divide()–operations that iterate through the points and do things to them. The actual mechanics of this don’t matter a whole lot. It’s the concept that’s important.

Now let’s say that in the graph the internal representation of the points is a floating point data type such as, well, float. I’m going to save the nuance of floating point arithmetic for a future practical math post, but suffice it say that floats can exhibit some weird-seeming behavior when it comes to comparisons, truncation/rounding, certain kinds of casting and type representations, etc. 

[TestMethod, Owner("ebd"), TestCategory("Proven"), TestCategory("Unit")]
public void Mind_Equals_Blown()
{
    float x = 0.2f;
    float y = 0.1f;
    float z = x + y;

    Assert.IsTrue(z == x + y);  //What the - why does this fail?!?
}

And let’s also say that the person responsible for authoring this graph class hasn’t read a practical math post about floating point arithmetic and is completely oblivious to these potential pitfalls.

And, finally, let’s say that this graph class becomes a mainstay of the business logic in a particular application. It’s modified, extended, and relied heavily upon without a whole lot of attention paid to its internal workings. At least until stuff mysteriously doesn’t work. But when that happens, the culprit isn’t immediately obvious, so strange work-arounds and cargo-cult, oddball solutions spring up when symptoms occur. Extension methods are written, and sometimes entirely different modules are added to the code base because the existing one is “tricky” or “not to be trusted.”

At the application level, this causes maintenance issues, a lot of heated and fruitless arguments, and voodoo approaches to code. From a user interface perspective, this causes quirky behavior. Occasionally a linear graph is completely displaced out of the graph and rendered on some menu somewhere, or the screen goes blank for a few seconds and then the display is restored. Defects and defect reports are created and developers dispatched to track down the issue, but after a few days of fruitless efforts, some project manager quietly sets the defect’s priority from “critical” to “cosmetic” and the software is shipped. It’s embarrassing, but whatcha gonna do. Ya know, computers have a mind of their own sometimes!

MessedUpGraph

Catching it Early

What if, instead of doing things the old-fashioned but all-too-common way, the authors of this code had been writing unit tests and/or practicing TDD? Well, there’s a very good chance that the issue stemming from the graph library is caught immediately as its API methods are being fleshed out from a functionality perspective. There’s a good chance that someone is writing a test and gets to the point that we were at in the code sample above, where they are utterly dumbfounded as to why 1+1 does not equal 2 in float land.

And then, good things happen. The developer in question takes to google or stack overflow, or perhaps he talks to other, more experienced developers on his team. He then gets an explanation, learns something about the language, and leaves the code in a correct state. Contrast this with the non-tested approach of “code it up, build a bad house on the bad foundation, and then ship the result because it’s too late.”

And what if the TDD/unit tests don’t expose this issue? Well, what they’ll do in either case is decouple the code base. So when the issue eventually does crop up via weird GUI behavior, it will be much easier to isolate. When it’s isolated, it will be much easier for the unit-test-savvy developers to write a test that exposes the defect to learn the lesson and fix the issue. It’s still a win.

The point about unit tests helping catch errors and leading to a more decoupled design is hardly controversial. But the benefits go beyond that. Unit tests provide a fast feedback loop for all points in the code base, which lends itself very well to poking and prodding things and experimenting. And that, in turn, leads to better understanding of not only the code, but also the language. If you can execute and get feedback on code extremely quickly, you’re much more likely to ask questions like, “I wonder what happens if I do x…” and then to do it and see. And that sort of experimentation, much like immersion in natural language, leads much more quickly to fluency.

Apr
15

By

What Drives Waterfall Projects?

Category: Anti Patterns, The Life of a Programmer Tags: , |

To start off the week, I have a satirical post about projects developed using the waterfall ‘methodology.’ (To understand the quotes, please see my post on why I don’t think waterfall is actually a methodology at all). I figured that since groups that use agile approaches and industry best practices have a whole set of xDD acronyms, such as TDD, BDD, and DDD, waterfall deserved a few of its own. So keep in mind that while this post is intended to be funny, I think there is a bit of relevant commentary to it.

Steinbeck-Driven Development (SDD)

For those of you who’ve never had the pleasure to read John Steinbeck’s “Of Mice and Men,” any SDD practitioner will tell you that it’s a heartwarming tale of two friends who overcome all odds during the Great Depression, making it cross-country to California to start a rabbit petting zoo. And it’s that outlook on life that they bring to the team when it comes to setting deadlines, tracking milestones, and general planning.

scan0003

Relentlessly optimistic, the SDD project manager reacts to a missed milestone by reporting to his superiors that everything is a-OK because the team will just make it up by the time they hit the next one. When the next milestone is missed by an even wider margin, same logic applies. Like a shopping addict or degenerate gambler blithely saying, “you gotta spend money to make money,” this project manager will continue to assume on-time delivery right up until the final deadline passes with no end in site. When that happens, it’s no big deal–they just need a week to tie up a few loose ends. When that week is up, it’ll just be one more week to tie up a few loose ends. When that week expires, they face reality. No, just kidding. It’ll just be one more week to tie up a few loose ends. After enough time goes by, members of the team humor him with indulgent baby talk when he says this: “sure it will, man, sure it will. In a week, everything will be great, this will all be behind us, and we’ll celebrate with steaks and lobster at the finest restaurant in town.”

Spoiler alert. At the end of Steinbeck’s novel, the idyllic rabbit farm exists only in the mind of one of the friends, shortly before he’s shot in the back of the head by the other, in an act that is part merciful euthanasia and part self-preservation. The corporate equivalent of this is what eventually happens to our project manager. Every week he insists that everything will be fine and that they’re pretty close to the promised land until someone puts the project out of its misery.

Shooting-Star-Driven Development (SSDD)

Steinbeck-Driven Development is not for everyone. It requires a healthy ability to live in deluded fantasy land (or, in the case of the novel, to be a half-wit). SSDD project managers are not the relentless optimists that their SDD counterparts are. In fact, they’re often pretty maudlin, having arrived at a PM post on a project that everyone knows is headed for failure and basically running out the clock until company bankruptcy or retirement or termination or something. These are the equivalents of gamblers that have exhausted their money and credit and are playing at the penny tables in the hopes that their last few bucks will take them on an unprecedented win streak. Or, perhaps more aptly, they’re like a lonely old toy-maker, sitting in his woodshop, hoping for a toy to come to life and keep them company.

This PM and his project are doomed to failure, so he rarely bothers with status meetings, creates a bare minimum of power points, and rarely ever talks about milestones. Even his Gantt charts have a maximum of three nested dependencies. It’s clear to all that he’s phoning it in. He knows it’s unlikely, but he pins his slim hope to a shooting star: maybe one of his developers will turn out to be the mythical 100x developer that single-handedly writes the customer information portal in the amount of time that someone, while struggling to keep a straight face, estimated it would take to do.

As projects go along and fall further and further behind schedule and the odds of a shooting star developer become more and more remote, the SSDD project manager increasingly withdraws. Eventually, he just kind of fades away. If Geppetto were a real life guy, carving puppets and asking stars to make them real children, he’d likely have punched out in an 19th century sanitarium. There are no happy endings on SSDD projects–just lifeless, wooden developers and missed deadlines.

Fear-Driven Development (FDD)

There is no great mystery to FDD projects. The fate of the business is in your hands, developers. Sorry if that’s a lot of pressure, but really, it’s in your hands.

The most important part of a FDD project is to make it clear that there will be consequences–dire consequences–to the business if the software isn’t delivered by such and such date. And, of course, dire consequences for the business are pretty darned likely to affect the software group. So, now that everyone knows what’s at stake, it’s time to go out and get that complex, multi-tiered, poorly-defined application built in the next month. Or else.

Unlike most waterfall projects, FDD enters the death march phase pretty much right from the start of coding. (Other waterfall projects typically only start the death march phase once the testing phase is cancelled and the inevitability of missing the deadline is clear.) The developers immediately enter a primal state of working fourteen hours per day because their very livelihoods hang in the balance. And, of course, fear definitely has the effect of getting them to work faster and harder than they otherwise would, but it also has the side effect of making the quality lower. Depending on the nature of the FDD project and the tolerance level of the customers for shoddy or non-functional software, this may be acceptable. But if it isn’t, time for more fear. Consequences become more dire, days become longer, and weekends are dedicated to the cause.

The weak have nervous breakdowns and quit, so only the strong survive to quit after the project ends.

Passive-Aggressive-Driven Development (PADD)

One of the most fun parts of waterfall development is the the estimation from ignorance that takes place during either the requirements or design days. This is where someone looks at a series of Visio diagrams and says, “I think this project will take 17,388.12 man-hours in the low risk scenario and 18,221.48 in the high-risk scenario.” The reason I describe this as fun is because it’s sort of like that game you play where everyone guesses the number of gumballs in a giant jar of gumballs and whoever is closest without going over wins a prize. For anything that’s liable to take longer than a week, estimation in a waterfall context is a ludicrous activity that basically amounts to making things up and trying to keep a straight face as you convince yourself and others that you did something besides picking a random number.

Well, I broke this task up into 3,422 tasks and estimated each of those, so if they each take four hours, and everything goes smoothly when we try to put them all together with an estimate for integration of… ha! Just kidding! My guess is 10,528 hours–ten because I was thinking that it’d have to be five digits, the fve because it’s been that many days that we’ve been looking at these Gantt charts and sequence diagrams, and twenty-eight because that was my number in junior high football. And you can’t bid one hour over me because I’m last to guess!

But PADD PMs suck all of the fun out of this style of estimation by pressuring the hours guessers (software developers) into retracting and claiming less time. But they don’t do it by showing anger–the aggression is indirect. When the developer says that task 1,024, writing the batch file import routine, will take approximately five hours, the PADD PM says, “Oh, wow. Must be pretty complicated. Jeez, I just assumed that a senior level developer could bang that out in no more than two. My bad.” Shamed, the developer retracts: “No, no–you’re right. I figured the EDI would be more complicated than it was, so I just realized that my estimate is actually two hours.”

Repeated in aggregate, the PADD PM is some kind of spectacular black belt/level 20/guru/whatever metric is used to measure PM productivity, because he just reduced the time to market by 60% before a single line of code was ever written. Amazing! Of course, talk at the beginning of the project is cheap. The real measure of waterfall project success is figuring out who to blame and getting others to absorb the cost when the project gets way behind schedule. And this is where the PADD master really shines.

To his bosses, he says, “man, I guess I just had too much faith in our guys–I mean, I know you hire the best.” To the developers, he says, “boy, your estimates seemed pretty reasonable to me, so I would have assumed that everything would be going on time if you were just putting in the hours and elbow grease… weird.” To the end-users/stakeholders, he says, “it’s strange, all of our other stakeholders who get us all of the requirements clearly and on time get their software on time–I wonder what happened here.”

There’s plenty of blame to go around, and PADD PMs make sure everyone partakes equally and is equally dissatisfied with the project.

Apr
10

By

Exception Handling Basics

Category: .NET Tags: , | 4 Comments

The other day, I was reviewing some code, and I saw a series of methods conforming to the following (anti) ‘pattern’

public class CustomerProcessor
{
    public void ProcessCustomer(Customer customer)
    {
        try
        {
            if (customer.IsActive)
                ProcessActiveCustomer(customer);
        }
        catch (Exception ex)
        {
            throw ex;
        }
    }

    private void ProcessActiveCustomer(Customer customer)
    {
        try
        {
            CheckCustomerName(customer);
            WriteCustomerToFile(customer);
        }
        catch (Exception ex)
        {
            throw ex;
        }
    }

    public void CheckCustomerName(Customer customer)
    {
        try
        {
            if (customer.Name == null)
                customer.Name = string.Empty;
        }
        catch (Exception ex)
        {
            throw ex;
        }
    }

    private void WriteCustomerToFile(Customer customer)
    {
        try
        {
            using (StreamWriter writer = new StreamWriter(@"C:\temp\customer.txt"))
            {
                writer.WriteLine(customer.Name);
            }
        }
        catch (Exception ex)
        {
            throw ex;
        }
    }
}

Every method consisted of a try with the actual code of interest inside of it and then a caught general exception that was then thrown. As I looked more through the code base, it became apparent to me that this was some kind of ‘standard’ (and thus perhaps exhibit A of how we get standards wrong). Every method in the project did this.

If you’re reading and you don’t know why this is facepalm, please read on. If you’re well-versed in C# exceptions, this will probably be review for you.

Preserve the Stack Trace

First things (problems) first. When you throw exceptions in C# by using the keyword “throw” with some exception type, you rip a hole in the fabric of your application’s space-time–essentially declaring that if no code that’s called knows how to handle the singularity you’re belching out, the application will crash. I use hyperbolic metaphor to prove a point. Throwing an exception is an action that jolts you out of the normal operation of your program by using a glorified “GOTO,” except that you don’t actually know where it’s going because that’s up to the code that called you.

When you do this, the .NET framework is helpful enough to package up a bunch of runtime information for troubleshooting purposes, including something called the “stack trace.” If you’ve ever seen a .NET (or Java) site really bomb out, you’ve probably seen one of these–it’s a bunch of code with line numbers that basically tells you, “A called B, which called C, which called D … which called Y, which called Z, which threw up and crashed your program.” When you throw an exception in C# the framework saves the stack trace that got you to the method in question. This is true whether the exception happens in your code or deep, deep within some piece of code that you rely on.

So, in the code above, let’s consider what happens when the code is executed on a machine with no C:\temp folder. The StreamWriter constructor is going to throw an exception indicating that the path in question is not found. When it does, you will have a nice exception that tells you ProcessCustomer called ProcessActiveCustomer, which called WriteCustomerToFile, which called new StreamWriter(), which threw an exception because you gave it an invalid path. Pretty neat, huh? You just have to drill into the exception object in the debugger to see all of this (or have your application configured to display this information in a log file, on a web page, etc.).

But what happens next is kind of a bummer. Instead of letting this exception percolate up somewhere, we trap it right there in the method in our catch block. At that point, we throw an exception. Now remember, when you throw an exception object, the stack trace is recorded at the point that you throw, and any previous stack trace is blown away. Instead of it being obvious that the exception originated in the StreamWriter constructor, it appears to have originated in WriteCustomerToFile. But wait, it gets worse. From there, the exception is trapped in ProcessActiveCustomer and then again in ProcessCustomer. Since every method in the code base has this boilerplate, every exception generated will percolate back up to main and appear to have been generated there.

To put this in perspective, you will never be able to see or record the stack trace for the point at which the exception was thrown. Now, in development that’s not the end of the world since you can set the debugger to break where thrown instead of handled, but for production logging, this is awful. You’ll never have the foggiest idea where anything is coming from.

How to fix this? It’s as simple as getting rid of the “throw ex;” in favor of just “throw;” This preserves the stack trace while passing the exception on to the next handler. Another alternative, should you wish to add more information when you throw, would be to do “throw new Exception(ex)” where you pass the exception you’ve caught to a new one that you’re creating. The caught exception will be preserved, intact, and can be accessed in debugging via the “InnerException” property of the one you’re now throwing.

public class CustomerProcessor
{
    public void ProcessCustomer(Customer customer)
    {
        try
        {
            if (customer.IsActive)
                ProcessActiveCustomer(customer);
        }
        catch (Exception ex)
        {
            throw;
        }
    }

    private void ProcessActiveCustomer(Customer customer)
    {
        try
        {
            CheckCustomerName(customer);
            WriteCustomerToFile(customer);
        }
        catch (Exception ex)
        {
            throw;
        }
    }

    public void CheckCustomerName(Customer customer)
    {
        try
        {
            if (customer.Name == null)
                customer.Name = string.Empty;
        }
        catch (Exception ex)
        {
            throw;
        }
    }

    private void WriteCustomerToFile(Customer customer)
    {
        try
        {
            using (StreamWriter writer = new StreamWriter(@"C:\temp\customer.txt"))
            {
                writer.WriteLine(customer.Name);
            }
        }
        catch (Exception ex)
        {
            throw;
        }
    }
}

(It would actually be better here to remove the Exception ex altogether in favor of just “catch {” but I’m leaving it in for illustration purposes)

Minimize Exception-Aware Code

Now that the stack trace is going to be preserved, the pattern here isn’t actively hurting anything in terms of program flow or output. But that doesn’t mean we’re done cleaning up. There’s still a lot of code here that doesn’t need to be.

In this example, consider that there are only two methods that can generate exceptions: ProcessCustomer (if passed a null reference) and WriteCustomerToFile (various things that can go wrong with file I/O). And yet, we have exception handling in every method, even methods that are literally incapable of generating them on their own. Exception throwing and handling is extremely disruptive and it makes your code very hard to reason about. This is because exceptions, as mentioned earlier, are like GOTO statements that whip the context of your program from wherever the exception is generated to whatever place ultimately handles exceptions. Oh, and the boilerplate for handling them makes methods hard to read.

The approach shown above is a kind of needlessly defensive approach that makes the code incredibly dense and confusing. Rather than a strafing, shock-and-awe show of force for dealing with exceptions, the best approach is to reason carefully about where they might be generated and how one might handle them. Consider the following rewrite:

public class CustomerProcessor
{
    public void ProcessCustomer(Customer customer)
    {
        if(customer == null)
            Console.WriteLine("You can't give me a null customer!");
        try
        {
            ProcessActiveCustomer(customer);
        }
        catch (SomethingWentWrongWritingCustomerFileException)
        {
            Console.WriteLine("There was a problem writing the customer to disk.");
        }
    }

    private void ProcessActiveCustomer(Customer customer)
    {
        CheckCustomerName(customer);
        WriteCustomerToFile(customer);
    }

    public void CheckCustomerName(Customer customer)
    {
        if (customer.Name == null)
            customer.Name = string.Empty;
    }

    private void WriteCustomerToFile(Customer customer)
    {
        try
        {
            using (var writer = new StreamWriter(@"C:\temp\customer.txt"))
            {
                writer.WriteLine(customer.Name);
            }
        }
        catch (Exception ex)
        {
            throw new SomethingWentWrongWritingCustomerFileException("Ruh-roh", ex);
        }
    }
}

Notice that we only think about exceptions at the ‘endpoints’ of the little application. At the entry point, we guard against a null argument instead of handling it with an exception. As a rule of thumb, it’s better to handle validation via querying objects than by trying things and catching exceptions, both from a performance and from a readability standpoint. The other point of external interaction where we think about exceptions is where we’re calling out to the filesystem. For this example, I handle any exception generated by stuffing it into a custom exception type and throwing that back to my caller. This is a practice that I’ve adopted so that I know at a glance when debugging if it’s an exception I’ve previously reasoned about and am trapping or if some new problem is leaking through that I didn’t anticipate. YMMV on this approach, but the thing to take away is that I deal with exceptions as soon as they come to me from something beyond my control, and then not again until I’m somewhere in the program that I want to report things to the user. (In an actual application, I would handle things more granularly than simply catching Exception, opting instead to go as fine-grained as I needed to in order to provide meaningful reporting on the problem)

Here it doesn’t seem to make a ton of difference, but in a large application it will–believe me. You’ll be much happier if your exception handling logic is relegated to the places in the app where you provide feedback to the user and where you call external stuff. In the guts of your program, this logic isn’t necessary if you simply take care to write code that doesn’t contain mistakes like null dereferences.

What about things like out of memory exceptions? Don’t you want to trap those when they happen? Nope. Those are catastrophic exceptions beyond your control, and all of the logging and granular worrying about exceptions in the world isn’t going to un-ring that bell. When these happen, you don’t want your process to limp along unpredictably in some weird state–you want it to die.

On the Lookout for Code Smells

One other meta-consideration worth mentioning here is that if you find it painful to code because you’re putting the same few lines of code in every class or every method, stop and smell what your code is trying to tell you. Having the same thing over and over is very much not DRY and not advised. You can spray deodorant on it with something like a code snippet, but I’d liken this to addressing a body odor problem by spraying yourself with cologne and then putting on a full body sweatsuit–code snippets for redundant code make things worse while hiding the symptoms.

If you really feel that you must have exception handling in every method, there are IL Weaving tools such as PostSharp that free you from the boilerplate while letting you retain the functionality and granularity you want. As a general rule of thumb, if you’re cranking out a lot of code and thinking, “there’s got to be a better way to do this,” stop and do some googling because there almost certainly is.

Apr
8

By

Up or Not: Ambition of the Expert Beginner

Category: The Life of a Programmer Tags: | 11 Comments

In the last post, I talked about the language employed by Expert Beginners to retain their status at the top of a software development group. That post was a dive into the language mechanics of how Expert Beginners justify decisions that essentially stem from ignorance–and often laziness, to boot. They generally have titles like “Principal Engineer” or “Architect” and thus are in a position to argue policy decisions based on their titles rather than on any kind of knowledge or facts supporting the merits of their approach.

In the series in general, I’ve talked about how Expert Beginners get started, become established, and, most recently, about how they fend off new ideas (read: threats) in order to retain their status with minimal effort. But what I haven’t yet covered and will now talk about is the motivations and goals of the Expert Beginner. Obviously, motivation is a complex subject, and motivations will be as varied as individuals. But I believe that Expert Beginner ambition can be roughly categorized into groups and that these groups are a function of their tolerance for cognitive dissonance.

Wikipedia (and other places) defines cognitive dissonance as mental discomfort that arises from simultaneously holding conflicting beliefs. For instance, someone who really likes the taste of steak but believes that it’s unethical to eat meat will experience this form of unsettling stress as he tries to reconcile these ultimately irreconcilable beliefs. Different people have different levels of discomfort that arise from this state of affairs, and this applies to Expert Beginners as much as anyone else. What makes Expert Beginners unique, however, is how inescapable cognitive dissonance is for them.

An Expert Beginner’s entire career is built on a foundation of cognitive dissonance. Specifically, they believe that they are experts while outside observers (or empirical evidence) demonstrate that they are not. So an Expert Beginner is sentenced to a life of believing himself to be an expert while all evidence points to the contrary, punctuated by frequent and extremely unwelcome intrusions of that reality.

So let’s consider three classes of Expert Beginner, distinguished by their tolerance for cognitive dissonance and their paths through an organization.

Xenophobes (Low Tolerance)

An Expert Beginner with a low tolerance for cognitive dissonance is basically in a state of existential crisis, given that he has a low tolerance for the thing that characterizes his career. To put this more concretely, a marginally competent person, inaccurately dubbed “Expert” by his organization, is going to be generally unhappy if he has little ability to reconcile or accept conflicting beliefs. A more robust Expert Beginner has the ability to dismiss evidence against his ‘Expert’ status as wrong or can simply shrug it off, but not Xenophobe. Xenophobe becomes angry, distressed, or otherwise moody when this sort of thing happens.

But Xenophobe’s long term strategy isn’t simply to get worked up whenever something exposes his knowledge gap. Instead, he minimizes his exposure to such situations. This process of minimizing is where the name Xenophobe originates; he shelters himself from cognitive dissonance by sheltering himself from outsiders and interlopers that expose him to it.

If you’ve been to enough rodeos in the field of software development, you’ve encountered Xenophobe. He generally presides over a small group with an iron fist. He’ll have endless reams of coding standards, procedures, policies, rules, and quirky ways of doing things that are non-negotiable and soul-sucking. This is accompanied by an intense dose of micromanagement and insistence on absolute conformity in all matters. Nothing escapes his watchful eye, and his management generally views this as dedication or even, perversely, mentoring.

This practice of micromanagement serves double duty for Xenophobe. Most immediately, it allows him largely to prevent the group from being infected by any foreign ideas. On the occasion that one does sneak in, it allows him to eliminate it swiftly and ruthlessly to prevent the same perpetrator from doing it again. But on a longer timeline, the oppressive micromanagement systematically drives out talented subordinates in favor of malleable, disinterested ones that are fine with brainlessly banging out code from nine to five, asking no questions, and listening to the radio. Xenophobe’s group is the epitome of what Bruce Webster describes in his Dead Sea Effect post.

All that Xenophobe wants out of life is to preserve this state of affairs. Any meaningful change to the status quo is a threat to his iron-fisted rule over his little kingdom. He doesn’t want anyone to leave because that probably means new hires, which are potential sources of contamination. He will similarly resist external pushes to change the group and its mission. New business ventures will be labeled “unfeasible” or “not what we do.”

KingOfSmallKingdom

Most people working in corporate structures want to move up at some point. This is generally because doing so means higher pay, but it’s also because it comes with additional status perks like offices, parking spaces, and the mandate to boss people around. Xenophobe is not interested in any of this (beyond whatever he already has). He simply wants to come in every day and be regarded as the alpha technical expert. Moving up to management would result in whatever goofy architecture and infrastructure he’s set up being systematically dismantled, and his ego couldn’t handle that. So he demurs in the face of any promotion to project management or real management because even these apparently beneficial changes would poke holes in the Expert delusion. You’ll hear Xenophobe say things like, “I’d never want to take my hands off the keyboard, man,” or, “this company would never survive me moving to management.”

Company Men (Moderate Tolerance)

Company Man does not share Xenophobe’s reluctance to move into a line or middle management role. His comfort with this move results from being somewhat more at peace with cognitive dissonance. He isn’t so consumed with preserving the illusion of expertise at all costs that he’ll pass up potential benefits–he’s a more rational and less pathological kind of Expert Beginner.

Generally speaking, the line to a mid-level management position requires some comfort with cognitive dissonance whether or not the manager came into power from the ranks of technical Expert Beginners. Organizations are generally shaped like pyramids, with executives at the top, a larger layer of management in the middle, and line employees at the bottom. It shares more than just shape with a pyramid scheme–it sells to the rank and file the idea that ascension to the top is inevitable, provided they work hard and serve those above them well.

The source of cognitive dissonance in the middle, however, isn’t simply the numerical impossibility that everyone can work their way up. Rather, the dissonance lies in the belief that working your way up has much to do with merit or talent. In other words, only the most completely daft would believe that everyone will inevitably wind up in the CEO’s office (or even in middle management), so the idea bought into by most is this: each step of the pyramid selects its members from the most worthy of the step below it. The ‘best’ line employees become line managers, the ‘best’ line managers become mid-level managers, and so on up the pyramid. This is a pleasant fiction for members of the company that, when believed, inspires company loyalty and often hard work beyond what makes rational sense for a salaried employee.

But the reality is that mid-level positions tend to be occupied not necessarily by the talented but rather by people who have stuck around the company for a long time, people who are friends with or related to movers and shakers in the company, people who put in long hours, people who simply and randomly got lucky, and people who legitimately get work done effectively. So while there’s a myth perpetuated in corporate American that ascending the corporate ‘ladder’ (pyramid) is a matter of achievement, it’s really more of a matter of age and inevitability, at least until you get high enough into the C-level where there simply aren’t enough positions for token promotions. If you don’t believe me, go look at LinkedIn and tell me that there isn’t a direct and intensely strong correlation between age and impressiveness of title.

So, to occupy a middle management position is almost invariably to drastically overestimate how much talent and achievement it took to get to where you are. That may sound harsh, but “I worked hard and put in long hours and eventually worked my way up to an office next to the corner office” is a much more pleasant narrative than “I stuck with this company, shoveled crap, and got older until enough people left to make this office more or less inevitable.” But what does all of this have to do with Expert Beginners?

Well, Expert Beginners that are moderately tolerant of cognitive dissonance have approximately the same level of tolerance for it as middle management, which is to say, a fair amount. Both sets manage to believe that their positions were earned through merit while empirical evidence points to them getting there by default and managing not to fumble it away. Thus it’s a relatively smooth transition, from a cognitive dissonance perspective, for a technical Expert Beginner to become a manager. They simply trade technical mediocrity for managerial mediocrity and the narrative writes itself: “I was so good at being a software architect that I’ve earned a shot and will be good at being a manager.”

The Xenophobe would never get to that point because asking him to mimic competence at a new skill-set is going to draw him way outside of his comfort zone. He views moving into management as a tacit admission that he was in over his head and needed to be promoted out of danger. Company Man has no such compunction. He’s not comfortable or happy when people in his group bring in outside information or threaten to expose his relative incompetence, but he’s not nearly as vicious and reactionary as Xenophobe, as he can tolerate the odd creeping doubt of his total expertise.

In fact, he’ll often alleviate this doubt by crafting an “up after a while” story for himself vis-a-vis management. You’ll hear him say things like, “I’m getting older and can’t keep slinging code forever–sooner or later, I’ll probably just have to go into management.” It seems affable enough, but he’s really planning a face-saving exit strategy. When you start out not quite competent and insulate yourself from actual competence in a fast-changing field like software, failure is inevitable. Company Man knows this on some subconscious level, so he plans and aspires to a victorious retreat. This continues as high as Company Man is able to rise in the organization (though non-strategic thinkers are unlikely to rise much above line manager, generally). He’s comfortable with enough cognitive dissonance at every level that he doesn’t let not being competent stop him from assuming that he is competent.

Master Beginners (High Tolerance)

If Xenophobes want to stay put and Company Men want to advance, you would think that the class of people who have high tolerance for and thus no problem with cognitive dissonance, Master Beginners, would chomp at the bit to advance. But from an organizational perspective, they really don’t. Their desired trajectory from an org chart perspective is somewhere between Xenophobe and Company Man. Specifically, they prefer to stay put in a technical role but to expand their sphere of influence, breadth-wise, to grow the technical group under their tutelage. Perhaps at some point they’d be satisfied to be CTO or VP of Engineering or something, but only as long as they didn’t get too far away from their domain of ‘expertise.’

Master Beginners are utterly fascinating. I’ve only ever encountered a few of these in my career, but it’s truly a memorable experience. Xenophobes are very much Expert Beginners by nurture rather than nature. They’re normal people who backed their way into a position for which they aren’t fit and thus have to either admit defeat (and, worse, that their main accomplishment in life is being in the right place at the right time) or neurotically preserve their delusion by force. Company Men are also Expert Beginners by nurture over nature, though for them it’s less localized than Xenophobes. Company Men buy into the broader lie that advancement in command-and-control bureaucratic organizations is a function of merit. If a hole is poked in that delusion, they may fall, but a lot of others come with them. It’s a more stable fiction.

But Master Beginners are somehow Expert Beginners by nature. They are the meritocratic equivalent of sociopaths in that their incredible tolerance for cognitive dissonance allows them glibly and with an astonishing lack of shame to feign expertise when doing so is preposterous. It appears on the surface to be completely stunning arrogance. A Master Beginner would stand up in front of a room full of Java programmers, never having written a line of Java code in his life, and proceed to explain to them the finer points of Java, literally making things up as he went. But it’s so brazen–so utterly beyond reason–that arrogance is not a sufficient explanation. It’s like the Master Beginner is a pathological liar of some kind (though he’s certainly also arrogant.) He most likely actually believes that he knows more about subjects he has no understanding of than experts in those fields because he’s just that brilliant.

This makes him an excellent candidate for Expert Beginnerism both from an external, non-technical perspective and from a rookie perspective. To put it bluntly, both rookies and outside managers listen to him and think, “wow, that must be true because nobody would have the balls to talk like that unless they were absolutely certain.” This actually tends to make him better at Expert Beginnerism than his cohorts who are more sensitive to cognitive dissonance, roughly following the psychological phenomenon coined by Walter Langer:

People will believe a big lie sooner than a little one. And if you repeat it frequently enough, people will sooner than later believe it.

So the Master Beginner’s ambition isn’t to slither his way out of situations where he might be called out on his lack–he actually embraces them. The Master Beginner is utterly unflappable in his status as not just an expert, but the expert, completely confident that things he just makes up are more right than things others have studied for years. Thus the Master Beginner seeks to expand aggressively. He wants to grow the department and bring more people under his authority. He’ll back down from no challenge to his authority from any angle, glibly inventing things on the spot to win any argument, pivoting, spinning, shouting, threatening–whatever the situation calls for. And he won’t stop until everyone hails him as the resident expert and does everything his way.

Success?

I’ve talked about the ambitions of different kinds of Expert Beginners and what drives them to aspire to these ends. But a worthwhile question to ask is whether or not they tend to succeed and why or why not. I’m going to tackle the fate of Expert Beginners in greater detail in my next post on the subject, but the answer is, of course, that it varies. What tends not to vary, however, is that Expert Beginner success is generally high in the short term and drops to nearly zero on a long enough time line, at least in terms of their ambitions. In other words, success as measured by Expert Beginners themselves tends to be somewhat ephemeral.

It stands to reason that being deluded about one’s own competence isn’t a viable, long-term success strategy. There is a lesson to be learned from the fate of Expert Beginners in general, which is that better outcomes are more likely if you have an honest valuation of your own talents and skills. You can generally have success on your own terms through the right combination of strategy, dedication, and earnest self-improvement, but to improve oneself requires a frank and honest inventory of one’s shortcomings. Anything short of that, and you’re simply advancing via coincidence and living on borrowed time.

Edit: The E-Book is now available. Here is the publisher website which contains links to the different media for which the book is available.

« Older Posts
Newer Posts »