DaedTech

Stories about Software

Oct
11

By

Casting is a Polymorphism Fail

Category: .NET Tags: , | 34 Comments

Have you ever seen code that looked like the snippet here?

public class Menagerie
{
    private List _animals = new List();

    public void AddAnimal(Animal animal)
    {
        _animals.Add(animal);
    }

    public void MakeNoise()
    {
        foreach (var animal in _animals)
        {
            if (animal is Cat)
                ((Cat)animal).Meow();
            else if (animal is Dog)
                ((Dog)animal).Bark();
        }
    }
}

You probably have seen code like this, and I hope that it makes you sad. I know it makes me sad. It makes me sad because it’s clearly the result of a fundamental failure to understand (or at least implement) polymorphism. Code written like this follows an inheritance structure, but it completely misses the point of that structure, which is the ability to do this instead:

public class Menagerie
{
    private List _animals = new List();

    public void AddAnimal(Animal animal)
    {
        _animals.Add(animal);
    }

    public void MakeNoise()
    {
        foreach (var animal in _animals)
            animal.MakeNoise();
    }
}

What’s so great about this? Well, consider what happens if I want to add “Bird” or “Bear” to the mix. In the first example with casting, I have to add a class for my new animal, and then I have to crack open the menagerie class and add code to the MakeNoise() method that figures out how to tell my new animal to make noise. In the second example, I simply have to add the class and override the base class’s MakeNoise() method and Menagerie will ‘magically’ work without any source code changes. This is a powerful step toward the open/closed principle and the real spirit of polymorphism — the ability to add functionality to a system with a minimum amount of upheaval.

But what about more subtle instances of casting? Take the iconic:

public void HandleButtonClicked(object sender, EventArgs e)
{
    var button = (Button)sender;
    button.Content = "I was clicked!";
}

Is this a polymorphism failure? It can’t be, can it? I mean, this is the pattern for event subscription/handling laid out by Microsoft in the C# programming guide. Surely those guys know what they’re doing.

As a matter of fact, I firmly believe that they do know what they’re doing, but I also believe that this pattern was conceived of and created many moons ago, before the language had some of the constructs that it currently does (like generics and various frameworks) and followed some of the patterns that it currently does. I can’t claim with any authority that the designers of this pattern would ask for a mulligan knowing what they do now, but I can say that patterns like this, especially ones that become near-universal conventions, tend to build up quite a head of steam. That is to say, if we suddenly started writing even handlers with strongly typed senders, a lot of event producing code simply wouldn’t work with what we were doing.

So I contend that it is a polymorphism failure and that casting, in general, should be avoided as much as possible. However, I feel odd going against a Microsoft standard in a language designed by Microsoft. Let’s bring in an expert on the matter. Eric Lippert, principal developer on the C# compiler team, had this to say in a stack overflow post:

Both kinds of casts are red flags. The first kind of cast raises the question “why exactly is it that the developer knows something that the compiler doesn’t?” If you are in that situation then the better thing to do is usually to change the program so that the compiler does have a handle on reality. Then you don’t need the cast; the analysis is done at compile time.

The “first kind” of cast he’s referring to is one he defines earlier in his post as one where the developer “[knows] the runtime type of this expression but the compiler does not know it.” That is the kind that I’m discussing here, which is why I chose that specific portion of his post. In our case, the developer knows that “sender” is a button but the compiler does not know that. Eric’s point, and one with which I wholeheartedly agree, is “why doesn’t the compiler know it and why don’t we do our best to make that happen?” It just seems like a bad idea to run a reality deficit between yourself and the compiler as you go. I mean, I know that the sender is a button. You know the sender is a button. The method knows the sender is a button (if we take its name, containing “ButtonClicked” at face value). Maintainers know the sender is a button. Why does everyone know sender is a button except for the compiler, who has to be explicitly and awkwardly informed in spite of being the most knowledgeable and important party in this whole situation?

But I roll all of this into a broader point about a polymorphic approach in general. If we think of types as hierarchical (inheritance) or composed (interface implementation), then there’s some exact type that suits my needs. There may be more than one, but there will be a best one. When writing a method and accepting parameters, I should accept as general a type as possible without needing to cast so that I can be of the most service. When returning something, I should be as specific as possible to give clients the most options. But when I talk about “possible” I’m talking about not casting.

If I start casting, I introduce error possibilities, but I also necessarily introduce a situation where I’m treating an object as two different things in the same scope. This isn’t just jarring from a readability perspective — it’s a maintenance problem. Polymorphism allows me to care only about some public interface specification and not implementation details — as long as the thing I get has the public API I need, I don’t really care about any details. But as soon as I have to understand enough about an object to understand that it’s actually a different object masquerading as the one I want, polymorphism is right out the window and I suddenly depend on knowing the intricate relationship details of the class in question. Now I break not only if my direct collaborators change, but also if some inheritance hierarchy or interface hierarchy I’m not even aware of changes.

The reason I’m posting all of this isn’t to suggest that casting should never happen. Clearly sometimes it’s necessary, particularly if it’s forced on you by some API or framework. My hope though is that you’ll look at it with more suspicion — as a “red flag”, in the words of Eric Lippert. Are you casting because it’s forced on you by external factors, or are you casting to communicate with the compiler? Because if it’s the latter, there are other, better ways to achieve the desired effect that will leave your code more elegant, understandable, and maintainable.

By the way, if you liked this post and you're new here, check out this page as a good place to start for more content that you might enjoy.

Oct
10

By

Static and New Are Like Inline

Category: Anti Patterns Tags: , , | 3 Comments

C++ Inline

Reaching back into my C++ days, a concept exists called “inline”. “Suggesting inline” is a concept where you tell the C++ compiler that you want to dispense with function call overhead and slam the code in question right into the code stream of the caller. (It’s a matter of suggesting because the compiler might ignore this request during its optimization such as if you decide to rip a hole in space-time by suggesting inline on a recursive method). So, for instance:


inline int multiply(int x, int y)
{
     return x*y;
}

int main(int argc, char** argv)
{
     int product = multiply(2,5);
}

is effectively transformed into:

int main(int argc, char** argv)
{
     int product = 2*5;
}

This is conceptually similar to the concept of macros in C/C++. The idea is simple — you may have some block of code that you want to abstract out for reuse or readability, but you’d prefer the performance to mimick what would happen if you just wrote the code right in the method in question. You want to have your cake and to eat it too. Understandable — I mean, who doesn’t and what’s the point of having cake if you can’t eat it?

In the .NET world of managed languages, we don’t have this concept. It wouldn’t make any sense anyway as our builds generate byte code, which is processor-agnostic. The actual object code is going to be a matter between the runtime and the OS. So we don’t see the option for inline in the sense of runtime performance.

Metrics in OOP

One of the metrics in OOP that I think is a generally fair one is lines of code as a liability. If you have a 20-100 line class then life is good, but as you start creeping toward 300 lines it starts to smell. If it’s over 500 lines, kill it with fire (or, break it up into reasonable classes). The reason for this is a nod toward the Single Responsibility Principle and the concept of code smell. There’s nothing inherently wrong with large classes, per se, but they’re almost always an indication that a bunch of responsibilities are all knotted together in one tightly coupled mess. The same goes for methods on a smaller scale and with smaller scope of responsibility.

So, your personal preferences (and mine) about class/method size notwithstanding, it bears mentioning that smaller and more focused is generally considered better. The result of this is that it’s fairly common to evaluate class and method sizes with fellow developers and see people getting antsy as sizes get larger. On the flip side, it’s common to see people feel good when they keep class sizes small and to feel particularly good when they slay some hulking 5000 line beast and see it divided up into 20 more reasonably sized classes.

Another fairly common metric that I like is the number of parameters passed to a method. Like lines of code and large method/class, parameters trend toward code smell. No parameters is great, one is fine, two is a little noisy, and three is pushing it. More than that and you’ve written a method I want nothing to do with. I think a lot of people feel this way in terms of code they write and almost everyone feels this way when they’re a client (if a method has a bunch of parameters, good luck figuring out how to use it). This is true not only because it’s hard to keep all of the parameters straight but also because a method that needs a whole bunch of things to do its job is likely doing too large a job or too many jobs.

We like our methods/classes small and our parameter lists short.

Static and New: Gaming the System

One way to accomplish these goals is to ensure that your scopes are well defined, factored, and focused. But, if you don’t feel like doing that, you can cheat. If, for instance, you come across a constructor that takes 5 dependency parameters, the best thing to do would be to rework the object graph to have a more cohesive class that needed fewer dependencies. But, if time is short, you can just kick the pile of debris under the bed by having the constructor reach out into the static universe and pull its dependencies from the ether. Now your class has the cologne of a blissfully simple constructor hiding its dependnecy smell, thanks to some static or singleton access. The same thing can be applied with the “new” keyword. If you don’t want to rip holes in the fabric of your object graph with static state and functionality, you can always instantiate your own dependencies to keep that constructor looking slim (this would be identical in concept to having a stateless static factory method).

This trick also applies to cutting down lines of code. If you have a gigantic class, you could always port out some of its state and behavior out into a static class with static state or to an instance class spun up by the beast in question. In either case, lines of code goes down and arguments to public APIs stays the same.

Inline Revisited

Consider the following code:

public class Foo
{
    public int GetTotal(int n)
    {
        int total = 0;
        for (int index = 0; index < n; index++)
        {
            total += index;
        }
        return total;
    }
}

Let’s say that we thought that GetTotal method looked way too long and we wanted to shorten it by kicking parts of it under the bed when company came by to look at the class. What about this:

public class Foo
{
    public int GetTotal(int n)
    {
        return Utils.RunningSum(n);   
    }
}

This is fewer lines of code, to be sure. We’ve created a static Utils class that handles the dirty work of getting the running sum of all numbers up to and including a given number and we’ve delegated the work to this class. Not bad — we’ve eliminated 5 lines of code from both Foo and GetTotal(). And RunningSum is stateless, so there’s no worry about the kind of weird behavior and dependencies that you get from static state. This is a good thing, right?

Well, no, not really. I’d argue that this is fool’s gold in terms of our metrics. In a very real conceptual sense, Foo has no fewer lines of code than it initially did. Sure, from the perspective of organizing code it does — I’ve separated the concern of RunningSum from Foo’s GetTotal and we might make an argument that this factoring is a good thing (it’d be more interesting in a less trivial example). But from the perspective of coupling in the object graph, I’ve done exactly nothing.

When you call GetTotal(n), all of the same code is going to be executed in either case. All of the same branching will occur, all of the same logic will guide that branching, and all of the same local variables will be declared. From a dependency perspective as expressed in source code, you might as well inline Utils.RunningSum() into GetTotal(). To put this another way, you might as well conclude that Foo and GetTotal() are just as many lines of code as they ever were.

And that’s my larger point here. When your code invokes a static method or instantiates an object, client code calling your stuff has no choice in the matter. If I’m calling Foo’s GetTotal() method, it doesn’t make any difference to me if you call Utils.RunningSum() or just do the work yourself. It’s not as though I have any say in the matter. It’s not as though I can specify a strategy for computing the sum myself.

Now, by contrast, consider this:

public interface IComputeTotals
{
    int RunningSum(int n);
}

public class Foo
{
    public int GetTotal(int n, IComputeTotals computer)
    {
        return computer.RunningSum(n);
    }
}

Here I have a method that allows me to specify a number and a strategy for totals computing and it returns the computed total. Is this like inline? No, of course not — as the client, I have a lot of control here. Foo isn’t inlining anything because it needs me to specify the strategy.

But what about encapsulation? With the code in the method or abstracted to a hidden collaborator (static or new) the details of computation are hidden from me as the client whereas here they may not be (depending on where/how I got ahold of an instance that implements that interface). To that I say that it’s admittedly a tradeoff. This latter implementation is providing a seam and giving more options to client code whereas the former implementation is hiding more but leaving client code with fewer options. Assuming that any static methods involved are stateless/functional, that’s really the main consideration here – how much to cover up/hide and how much to expose.

I certainly have a preference for inversion of control and an aversion to static implementations both because of my desire for decoupled flexibility and my preference for TDD. But your mileage may vary. All I’d ask of anyone is to make informed decisions with eyes wide open. Metrics/smells like those about parameters and class/method size don’t exist in a vacuum. “Fixing” your 5000 line class by creating a static class and delegating 4800 lines of work to it behind the scenes is not reducing the size of that class in any meaningful sense, and it’s not addressing the bad smell the class creates; you’re just spraying perfume on it and hoping no one notices. The real problem isn’t simply that 5000 lines of code exist in one source file but rather that 5000 lines exist with no way to pry the dependencies apart and swap in alternate functionality.

So when you’re coding and bearing in mind certain heuristics and metrics, think of using static method calls and instantiating collaborators as what they really are — cosmetic ways to move code out of a method. The fact that the code moves to another location doesn’t automatically make it any more flexible, easier to maintain or less smelly. It’s providing additional seams and flexibility that has the effect you’re looking for.

Oct
4

By

Favor Outcomes Over Rules

Category: Anti Patterns Tags: , , , | 3 Comments

Cargo Cults and Angry Monkeys

If you have never heard the term “Cargo Cult Programming“, it’s an it’s an excellent way to describe blindly following processes without understanding the theory behind them. The term originates from a story about aboriginal islanders during World War II. During the war, cargo planes regularly (and from their perspective, magically) arrived with supplies and food that benefited these islanders and then after the war the planes stopped coming. This apparently didn’t stop the aboriginals from building ad-hoc landing strips and the like in hopes of ‘summoning’ more planes with supplies. If that story isn’t to your liking, there is another one of questionable accuracy about monkeys and bananas.

I’ve made a fair number of posts lately that address subjects near and dear to programmers while still having broader reach and this is certainly another such subject. Doing things without understand why is generally the province of the incurious or the busy, but I think it’s worth generally forcing yourself to ask why you’re doing pretty much anything, particularly if you’re in a fairly cerebral line of work. As professionals, it behooves us to realize that we want supplies or that we want not to get sprayed rather than thinking that building landing strips and beating up monkeys are just things that we do and such is life.

In fact, I’d venture to say that we pride ourselves in this sort of thing as we advance in our careers. As programmers, we even help our users understand this concept. “I understand that you normally push the green button three times and hit the backspace key while pressing the mute button, but what are you actually trying to do when you do all that?” How often have you asked a user something like this? In essence you’re saying “forget the process for a minute — what’s the larger goal here?”

Do Unto Others

It then bears asking, if we pride ourselves on critical thinking and we seek to help users toward that end, why do we seem to encourage each other to dance for supplies and spray our team members with cold water? Why do we issue cryptic cargo cult orders to other programmers when we understand our own rationale? Let me give an example.

A few years back I was setting up a new machine for work on a project to which I was new, and the person guiding me through the setup told me that I needed to install KDiff, a spectacularly average comparison tool that even then hadn’t been revved or maintained in a few years. Now, I have nothing, per se, against KDiff, and the price is right, but this struck me as a very… specific… order. What difference did it make what I used for diff? I had used a paid version of Beyond Compare prior to that and always liked it, but when it comes to tooling I do enjoy poking around and so I didn’t mind trying a different tool.

I’m pickier about some things than others, so I just said “Bob’s Your Uncle,” installed KDiff and didn’t think about it again for a long time. It was actually many months later that I was sitting in on a code review for a developer who had just started on the project and found out the reasoning behind this cargo cult practice. The developer whose code was being reviewed casually opened BeyondCompare and one of the more tenured reviewers than me promptly freaked out that he wasn’t using KDiff. Bemused, I tuned in to hear the answer to a question that I’d completely forgotten about. The issue is that the project had a handful of XML files containing application meta-data that were source controlled and that some unnamed developer had, at some point, done something, presumably with some sort of diff tool, that had allegedly hosed one of these files by screwing up the text encoding. Nobody in the room, including the freaker-outer, knew who this was or exactly how or when it had happened, but nonetheless, it had been written in stone that from that day, Thou Shalt Use KDiff if you want to avoid punishment in the form of Biblical plagues.

This certainly wasn’t my only experience with such a thing. More recently, I overheard that the procedure for editing a particular file in source control was to grab the latest, edit it really fast and then check it in immediately. I was a bit perplexed by this until I learned that the goal was to avoid merge conflicts as this was a commonly edited file. I thought, why not just say in the first place that a lot of people edit this file and that merge conflicts are bad, in the same way that I had previously thought “why not just tell people that you’re worried about messed up encodings?”

And that line of questioning really drives to the heart of the matter here. Developers are generally quite skilled and pretty intelligent people and, more importantly, they tend to be be people that like to solve problems. So why not give them the benefit of the doubt when you’re working with them? Instead of giving your peers rote procedures to follow because they happened to work for you once, why not explain the problem and say “this is how I solved it, but if you have a better idea, I’m all ears!”

And you know what? They might. What if instead of forcing people to use KDiff, someone had written a validation script for the offending file that prevented checkins of a badly formed edit? What if instead of having flash edit-checkins of a file, the design of the application were altered to eliminate the contention and potential for error around that file? I suggest these things because I’m a fan of making the bad impossible. Are they the greatest ideas? Are they even practical in those situations? I honestly don’t know, but they might be approaches that have the advantage of placing fewer restrictions on developers or meaning one less thing to remember.

I would encourage you to trust your peers to grasp the bigger picture whenever possible (unless perhaps they’ve demonstrated that this trust isn’t warranted). As I say in the title, it seems like a good idea to favor describing desired outcomes over inventing rules to be memorized. With the former approach you free people you work with to be creative. With the latter approach, you constrain them and possibly even stress them out or frustrate them. And who knows, they may even surprise you with ideas and solutions that make your life easier.

Oct
2

By

How Software Groups Rot: Legacy of the Expert Beginner

Category: The Life of a Programmer Tags: , , , | 31 Comments

Expert Beginner Recap

In my last post I introduced the term “Expert Beginner” to describe someone who has capped out in their learning at some sort of local maximum, convinced that the local is global. Expert Beginners are developers who do not understand enough of the big picture to understand that they aren’t actually experts. What I mean by this is that they have a narrow enough perspective to think that whatever they have been exposed to is the best and only way to do things; examples would include a C# developer who pooh-poohs Java without ever having tried it, or a MySQL DBA who dismisses the NoSQL movement as a passing fad. This isn’t to say that not liking a technology or not having worked with one makes someone an Expert Beginner, but rather the vaguely solipsistic mindset that “if it’s not in my toolchest or something I’ve had experience with, it’s not worth doing.”

Another characteristic of the Expert Beginner, however, is that they have some position of relative authority or influence within a software group. In my last post, I proposed the term Expert Beginner without explaining the rationale, planning to discuss that here. The Advanced Beginner is someone who is in the advanced stage of being a beginner, whereas the Expert Beginner descriptor is both literal and intentionally ironic; it’s literal in that someone with that much practice at being a beginner could be said to be an expert and ironic in that the “expert” title is generally self-applied in earnest or else applied by managers or peers who don’t know any better.

The iconic example might be the “tech guy” at a small, non-technical firm. He “knows computers” so as the company grew and evolved to have some mild IT needs, he became a programming dilettante out of necessity. In going from being a power user to being a developer, his skills exceeded his expectations, so he became confident in his limited, untrained abilities. Absent any other peers in the field, the only people there to evaluate his skills are himself and non-technical users who offer such lofty praises as “it seems to work, kind of, I think.” He is the one-eyed man in the valley of the blind and, in a very real and unfortunate sense, a local expert. This is the iconic example because it has the fewest barriers to Expert Beginnerism–success is simple, standards are low, actual experts are absent, competition is non-existent, and external interaction is not a given.

A Single Point of Rot…

So far I’ve talked a lot about the Expert Beginner–his emergence, his makeup, his mindset, and a relatively sympathetic explanation of the pseudo-rationality, or at least understandability, of his outlook. But how does this translate into support of my original thesis that Expert Beginners cause professional toxicity and degeneration of software groups? To explain that, I’m going to return to my bowling analogy, and please bear with me if the metaphor is a touch strained. For those of you who didn’t read the first post, you might want to give the second section of it a read because this is a lot to rehash here.

Let’s say that bowling alleys make their money by how well their bowlers bowl and that I live in a small town with a little, startup bowling alley. Not being able to find any work as a software developer, I try my hand bowling at the local alley. I don’t really know what I’m doing, and neither do they, but we both see me improving rapidly as I start bowling there, in spite of my goofy style. My average goes up, the bowling alley makes money, and life is good–there’s no arguing with profit and success!

Around about the time my score was topping 150 and the sky seemed the limit, the bowling alley decided that it was time to expand and to hire a couple of entry-level bowlers to work under my tutelage. On the day they arrived, I showed them how to hold the ball just like I hold it and how to walk just like I do. When they ask what the thumb and finger holes are for, I respond by saying, “don’t worry about those–we don’t use them here.” Eager to please, they listen to me and see their averages increase the way mine had, even as I’m starting to top out at around a 160 average.

As time goes by, most of them are content to do things my way. But a couple are ambitious and start to practice during their spare time. They read books and watch shows on bowling technique. One day, these ambitious bowlers come in and say, “the guys on TV put their fingers in the ball, and they get some really high averages–over 200!” They expect me to be as interested as they are in the prospect of improvement and are crestfallen when I respond with, “No, that’s just not how we do things here. I’ve been bowling for longer than you’ve been alive and I know what I’m doing… besides, you can’t believe everything you see on TV.”

And thus, quickly and decisively, I squash innovation for the group by reminding them that I’m in charge by virtue of having been at the bowling alley for longer than they have. This is a broadly accepted yet totally inadequate non sequitur that stops discussion without satisfying. At this point, half of the ambitious developers abandon their “fingers in the ball” approach while the other half meet after bowling at another alley and practice it together in semi-secret. After a while, their averages reach and surpass mine, and they assume that this development–this objective demonstration of the superiority of their approach–will result in a change in the way things are done. When it instead results in anger and lectures and claims that the scores were a fluke and I, too, once bowled a 205 that one time, they evaporate and leave the residue behind. They leave my dead-end, backward bowling alley for a place where people don’t favor demonstrably inferior approaches out of stubbornness.

The bowling alley loses its highest average bowlers not to another alley, but to an Expert Beginner.

…That Poisons the Whole

The bowlers who don’t leave learn two interesting lessons from this. The first lesson they learn is that if they wait their turn, they can wield unquestioned authority regardless of merit. The second lesson they learn is that it’s okay and even preferred to be mediocre at this alley. So when new bowlers are hired, in the interests of toeing the company line and waiting their turn, they participate in the inculcation of bad practices to the newbies the same way as was done to them. The Expect Beginner has, through his actions and example, created additional Expert Beginners and, in fact, created a culture of Expert Beginnerism.

The other interesting development that results comes in the acquisition process. As the Expert-Beginner-in-Chief, I’ve learned a pointed lesson. Since I don’t like being shown up by ambitious young upstarts, I begin to alter my recruitment process to look for mediocre “team players” that won’t threaten my position with their pie-in-the-sky “fingers in the ball” ideas. Now, I know what you’re thinking–doesn’t this level of awareness belie the premise of the Expert Beginner being unaware of the big picture? The answer is no. This hiring decision is more subconscious and rationalized than overt. It isn’t, “I won’t hire people that are better than me,” but, “those people just aren’t a good fit here with my ‘outside the box’ and ‘expert’ way of doing things.” And it may even be that I’m so ensconced in Expert Beginnerism that I confuse Competent/Expert level work with incompetent work because I don’t know any better. (The bowling analogy breaks down a bit here, but it might be on par with a “bowling interview” where I just watched the form of the interviewee’s throw and not the results, and concluded that the form of a 220 bowler was bad because it was different than my own.) And, in doing all this, I’m reinforcing the culture for everyone including my new Expert Beginner lieutenants.

And now the bowling alley is losing all of its potentially high average bowlers to a cabal of Expert Beginners. Also notice that Bruce Webster’s “Dead Sea Effect” is fully mature and realized at this point.

Back in the Real World

That’s all well and good for bowling and bowling alleys, but how is this comparable to real software development practices? Well, it’s relatively simple. Perhaps it’s a lack of automated testing. Giant methods/classes. Lots of copy and paste coding. Use of outdated or poor tooling. Process. It can be any number of things, but the common thread is that you have a person or people in positions of authority that have the culturally lethal combination of not knowing much; not knowing what they don’t know; and assuming that, due to their own expertise, anything they don’t know isn’t worth knowing. This is a toxic professional culture in that it will force talented or ambitious people either to leave or to conform to mediocrity.

You may think that this is largely a function of individual personalities, that departments become this way by having arrogant or pushy incompetents in charge, but I think it’s more subtle than that. These Expert Beginners may not have such personality defects at all. I think it’s a natural conclusion of insular environments, low expectations, and ongoing rewards for mediocre and/or unquantifiable performances. And think about the nature of our industry. How many outfits have you worked at where there is some sort of release party, even (or especially) when the release is over budget, buggy and behind schedule? How many outfits have you worked at that gave up on maintaining some unruly beast of an application in favor of a complete rewrite, only to repeat that cycle later? And the people involved in this receive accolades and promotions, which would be like promoting rocket makers for making rockets that looked functional but simply stopped and fell back to Earth after a few hundred feet. “Well, that didn’t work, Jones, but you learned a lot from it, so we’re promoting you to Principal Rocket Builder and having you lead version two, you rock star, you!” Is it any wonder that Jones starts to think of himself as King Midas?

As an industry, we get away with this because people have a substantially lower expectation of software than they do of rockets. I’m not saying this to complain or to suggest sweeping change but rather to explain how it’s easy for us to think that we’re further along in our skills acquisition than we actually are, based on both our own perception and external feedback.

Create a Culture of Acquisition instead of Stagnation

Having identified a group-(de)forming attitude that could most effectively be described as a form of hubris, I would like to propose some relatively simple steps to limit or prevent this sort of blight.

First of all, to prevent yourself from falling into the Expect Beginner trap, the most important thing to do is not to believe your own hype. Take pride in your own accomplishments as appropriate, but never consider your education complete or your opinion above questioning regardless of your title, your years of experience, your awards and accomplishments, or anything else that isn’t rational argumentation or evidence. Retaining a healthy degree of humility, constantly striving for improvement, and valuing objective metrics above subjective considerations will go a long way to preventing yourself from becoming an Expert Beginner.

In terms of preventing this phenomenon from corrupting a software group, here is a list of things that can help:

  1. Give team members as much creative freedom as possible to let them showcase their approaches (and remember that you learn more from failures than successes).
  2. Provide incentives or rewards for learning a new language, approach, framework, pattern, style, etc.
  3. Avoid ever using number of years in the field or with the company as a justification for favoring or accepting anyone’s argument as superior.
  4. Put policies in place that force external perspectives into the company (lunch-and-learns, monthly training, independent audits, etc).
  5. Whenever possible, resolve disputes/disagreements with objective measures rather than subjective considerations like seniority or democratic vote.
  6. Create a “culture of proof”–opinions don’t matter unless they’re supported with independent accounts, statistics, facts, etc.
  7. Do a periodic poll of employees, junior and senior, and ask them to list a few of their strengths and an equal number of things they know nothing about or would like to know more about. This is to deflate ahead of time an air of “know-it-all-ism” around anyone–especially tenured team members.

This list is more aimed at managers and leaders of teams, but it’s also possible to affect these changes as a simple team member. The only difference is that you may have to solicit help from management or persuade rather than enforce. Lead by example if possible. If none of that is possible, and it seems like a lost cause, I’d say head off for greener pastures. In general, it’s important to create or to have a culture in which “I don’t know” is an acceptable answer, even for the most senior, longest-tenured leader in the group, if you want to avoid Expert Beginner fueled group rot. After all, an earnest “I don’t know” is something Expert Beginners never say, and it is the fundamental difference between a person who is acquiring skill and a person who has decided that they already know enough. If your group isn’t improving, it’s rotting.

Series is continued here: “How Stagnation is Justified: Language of the Expert Beginner”

No Fields Found.

Sep
30

By

How Developers Stop Learning: Rise of the Expert Beginner

Category: The Life of a Programmer Tags: , , | 93 Comments

Beyond the Dead Sea: When Good Software Groups Go Bad

I recently posted what turned out to be a pretty popular post called “How to Keep Your Best Programmers,” in which I described what most skilled programmers tend to want in a job and why they leave if they don’t get it.

Today, I’d like to make a post that works toward a focus on the software group at an organization rather than on the individual journeys of developers as they move within or among organizations. This post became long enough as I was writing it that I felt I had to break it in at least two pieces. This is part one.

In the previous post I mentioned, I linked to Bruce Webster’s “Dead Sea Effect” post, which describes a trend whereby the most talented developers tend to be the most marketable and thus the ones most likely to leave for greener pastures when things go a little sour. On the other hand, the least talented developers are more likely to stay put since they’ll have a hard time convincing other companies to hire them.

This serves as important perspective for understanding why it’s common to find people with titles like “super-duper-senior-principal-fellow-architect-awesome-dude,” who make a lot of money and perhaps even wield a lot of authority but aren’t very good at what they do. But that perspective still focuses on the individual. It explains the group only if one assumes that a bad group is the result of a number of these individuals happening to work in the same place (or possibly that conditions are so bad that they drive everyone except these people away).

Dale will tell you what’s wrong with so-called professional ORMs.

I believe that there is a unique group dynamic that forms and causes the rot of software groups in a way that can’t be explained by bad external decisions causing the talented developers to evaporate. Make no mistake–I believe that Bruce’s Dead Sea Effect is both the catalyst for and the logical outcome of this dynamic, but I believe that some magic has to happen within the group to transmute external stupidities into internal and pervasive software group incompetence.

In the next post in this series, I’m going to describe the mechanism by which some software groups trend toward dysfunction and professional toxicity. In this post, I’m going to set the stage by describing how individuals opt into permanent mediocrity and reap rewards for doing so.

Learning to Bowl

Before I get to any of that, I’d like to treat you to the history of my bowling game. Yes, I’m serious.

I am a fairly athletic person. Growing up, I was always picked at least in the top 1/3rd or so of any people, for any sport or game that was being played, no matter what it was. I was a jack of all trades and master of none. This inspired in me a sort of mildly inappropriate feeling of entitlement to skill without a lot of effort, and so it went when I became a bowler.

Most people who bowl put a thumb and two fingers in the ball and carefully cultivate tossing the bowling ball in a pattern that causes the ball to start wide and hook into the middle. With no patience for learning that, I discovered I could do a pretty good job faking it by putting no fingers and thumbs in the ball and kind of twisting my elbow and chucking the ball down the lane.

It wasn’t pretty, but it worked.

It actually worked pretty well the more I bowled, and, when I started to play in an after work league for fun, my average really started to shoot up. I wasn’t the best in the league by any stretch–there were several bowlers, including a former manager of mine, who averaged between 170 and 200, but I rocketed up past 130, 140, and all the way into the 160 range within a few months of playing in the league.

Not too shabby.

But then a strange thing happened. I stopped improving. Right at about 160, I topped out.

I asked my old manager what I could do to get back on track with improvement, and he said something very interesting to me. Paraphrased, he said something like this:

There’s nothing you can do to improve as long as you keep bowling like that. You’ve maxed out. If you want to get better, you’re going to have to learn to bowl properly.

You need a different ball, a different style of throwing it, and you need to put your fingers in it like a big boy. And the worst part is that you’re going to get way worse before you get better, and it will be a good bit of time before you get back to and surpass your current average.

I resisted this for a while but got bored with my lack of improvement and stagnation (a personal trait of mine–I absolutely need to be working toward mastery or I go insane) and resigned myself to the harder course. I bought a bowling ball, had it custom drilled, and started bowling properly.

Ironically, I left that job almost immediately after doing that and have bowled probably eight times in the years since, but c’est la vie, I suppose. When I do go, I never have to rent bowling shoes or sift through the alley balls for ones that fit my fingers.

Dreyfus, Rapid Returns and Arrested Development

In 1980, a couple of brothers with the last name Dreyfus proposed a model of skill acquisition that has gone on to have a fair bit of influence on discussions about learning, process, and practice. Later they would go on to publish a book based on this paper and, in that book, they would refine the model a bit to its current form, as shown on wikipedia.

The model lists five phases of skill acquisition:

  1. Novice
  2. Advanced Beginner
  3. Competent
  4. Proficient
  5. Expert

There’s obviously a lot to it, since it takes an entire book to describe it, but the gist of it is that skill acquirers move from “dogmatic following of rules and lack of big picture” to “intuitive transcending of rules and complete understanding of big picture.”

All things being equal, one might assume that there is some sort of natural, linear advancement through these phases, like earning belts in karate or money in the corporate world. But in reality, it doesn’t shake out that way, due to both perception and attitude.

At the moment one starts acquiring a skill, one is completely incompetent, which triggers an initial period of frustration and being stymied while waiting for someone, like an instructor, to spoon-feed process steps to the acquirer (or else, as Dreyfus and Dreyfus put it, they “like a baby, pick it up by imitation and floundering”). After a relatively short phase of being a complete initiate, however, one reaches a point where the skill acquisition becomes possible as a solo activity via practice, and the renewed and invigorated acquirer begins to improve quite rapidly as he or she picks “low hanging fruit.”

Once all that fruit is picked, however, the unsustainably rapid pace of improvement levels off somewhat, and further proficiency becomes relatively difficult from there forward. I’ve created a graph depicting this (which actually took me an embarrassingly long time because I messed around with plotting a variant of the logistic 1/(1 + e^-x) function instead of drawing a line in Paint like a normal human being).

This is actually the exact path that my bowling game followed in my path from bowling incompetence to some degree of bowling competence. I rapidly improved to the point of competence and then completely leveled off. In my case, improvement hit a local maximum and then stopped altogether, as I was too busy to continue on my path as-is or to follow through with my retooling.

This is an example of what, for the purposes of this post, I will call “arrested development.” (I understand the overlap with a loaded psychology term, but forget that definition for our purposes here, please.) In the sense of skills acquisition, one generally realizes arrested development and remains at a static skill level due to one of two reasons: maxing out on aptitude or some kind willingness to cease meaningful improvement.

For the remainder of this post and this series, let’s discard the first possibility (since most professional programmers wouldn’t max out at or before bare minimum competence) and consider an interesting, specific instance of the second: voluntarily ceasing to improve because of a belief that expert status has been reached and thus further improvement is not possible..

This opting into indefinite mediocrity is the entry into an oblique phase in skills acquisition that I will call “Expert Beginner.”

The Expert Beginner

The Road to Expert... and Expert Beginner When you consider the Dreyfus model, you’ll notice that there is a trend over time from being heavily rules-oriented and having no understanding of the big picture to being extremely intuitive and fully grasping the big picture. The Advanced Beginner stage is the last one in which the skill acquirer has no understanding of the big picture.

As such, it’s the last phase in which the acquirer might confuse himself with an Expert. A Competent has too much of a handle on the big picture to confuse himself with an Expert: he knows what he doesn’t know. This isn’t true during the Advanced Beginner phase, since Advanced Beginners are on the “unskilled” end of the Dunning Kruger Effect and tend to epitomize the notion that, “if I don’t understand it, it must be easy.”

As such, Advanced Beginners can break one of two ways: they can move to Competent and start to grasp the big picture and their place in it, or they can ‘graduate’ to Expert Beginner by assuming that they’ve graduated to Expert.

This actually isn’t as immediately ridiculous as it sounds. Let’s go back to my erstwhile bowling career and consider what might have happened had I been the only or best bowler in the alley. I would have started out doing poorly and then quickly picked the low hanging fruit of skill acquisition to rapidly advance.

Dunning-Kruger notwithstanding, I might have rationally concluded that I had a pretty good aptitude for bowling as my skill level grew quickly. And I might also have concluded somewhat rationally (if rather arrogantly) that me leveling off indicated that I had reached the pinnacle of bowling skill. After all, I don’t see anyone around me that’s better than me, and there must be some point of mastery, so I guess I’m there.

The real shame of this is that a couple of inferences that aren’t entirely irrational lead me to a false feeling of achievement and then spur me on to opt out of further improvement. I go from my optimistic self-assessment to a logical fallacy as my bowling career continues: “I know that I’m doing it right because, as an expert, I’m pretty much doing everything right by definition.” (For you logical fallacy buffs, this is circular reasoning/begging the question).

Looking at the graphic above, you’ll notice that it depicts a state machine of the Dreyfus model as you would expect it. At each stage, one might either progress to the next one or remain in the current one (with the exception of Novice or Advanced Beginner who I feel can’t really remain at that phase without abandoning the activity). The difference is that I’ve added the Expert Beginner to the chart as well.

The Expert Beginner has nowhere to go because progression requires an understanding that he has a lot of work to do, and that is not a readily available conclusion. You’ll notice that the Expert Beginner is positioned slightly above Advanced Beginner but not on the level of Competence. This is because he is not competent enough to grasp the big picture and recognize the irony of his situation, but he is slightly more competent than the Advanced Beginner due mainly to, well, extensive practice being a Beginner.

If you’ve ever heard the aphorism about “ten years of experience or the same year of experience ten times,” the Expert Beginner is the epitome of the latter. The Expert Beginner has perfected the craft of bowling a 160 out of 300 possible points by doing exactly the same thing week in and week out with no significant deviations from routine or desire to experiment. This is because he believes that 160 is the best possible score by virtue of the fact that he scored it.

Expert Beginners in Software

Software is, unsurprisingly, not like bowling. In bowling, feedback cycles are on the order of minutes, whereas in software, feedback cycles tend to be on the order of months, if not years. And what I’m talking about with software is not the feedback cycle of compile or run or unit tests, which is minutes or seconds, but rather the project.

It’s during the full lifetime of a project that a developer gains experience writing code, source controlling it, modifying it, testing it, and living with previous design and architecture decisions during maintenance phases. With everything I’ve just described, a developer is lucky to have a first try of less than six months, which means that, after five years in the industry, maybe they have ten cracks at application development. (This is on average–some will be stuck on a single one this whole time while others will have dozens.)

What this means is that the rapid acquisition phase of a software developer–Advanced Beginnerism–will last for years rather than weeks. And during these years, the software developers are job-hopping and earning promotions, especially these days. As they breeze through rapid acquisition, so too do they breeze through titles like Software Engineer I and II and then maybe “Associate” and “Senior,” and perhaps eventually on up to “Lead” and “Architect” and “Principal.”

So while in the throes of Dunning-Kruger and Advanced Beginnerism, they’re being given expert-sounding titles and told that they’re “rock stars” and “ninjas” and whatever by recruiters–especially in today’s economy. The only thing stopping them from taking the natural step into the Expert Beginner stage is a combination of peer review and interaction with the development community at large.

But what happens when the Advanced Beginner doesn’t care enough to interact with the broader community and for whatever reason doesn’t have much interaction with peers? The Daily WTF is filled with such examples.

They fail even while convinced that the failure is everyone else’s fault, and the nature of the game is such that blaming others is easy and handy to relieve any cognitive dissonance. They come to the conclusion that they’ve quickly reached Expert status and there’s nowhere left to go. They’ve officially become Expert Beginners, and they’re ready to entrench themselves into some niche in an organization and collect a huge paycheck because no one around them, including them, realizes that they can do a lot better.

Until Next Time

And so we have chronicled the rise of the Expert Beginner: where they come from and why they stop progressing. In the next post in this series, I will explore the mechanics by which one or more Expert Beginners create a degenerative situation in which they actively cause festering and rot in the dynamics of groups that have talented members or could otherwise be healthy.

Next up: How Software Groups Rot: Legacy of the Expert Beginner

No Fields Found.
« Older Posts
Newer Posts »