DaedTech

Stories about Software

Feb
19

By

API Lost Opportunity: A Case Study

Category: Abstractions Tags: , , | 2 Comments

I’m aware that recently there’s been some brouhaha over criticism/trashing of open-source code and the Software Craftsmanship movement, and I’d like to continue my longstanding tradition of learning what I can from things without involving myself in any drama. Toward that end, I’d like to critique without criticizing, or, to put it another way, I’d like to couch some issues I have scan0001with an API as things that could be better rather than going on some kind of rant about what I don’t like. I’m hoping that I can turn the mild frustration I’m having as I use an API into being a better API writer myself. Perhaps you can pick up a few tidbits from this as well.

Recently I’ve been writing some code that consumes the Microsoft CRM API using its SDK. To start off with the good, I’d say that API is fairly powerful and flexible–it gives you access to the guts of what’s in the CRM system without making you write SQL or do any other specific persistence modeling. Another impressive facet is that it can handle the addition of arbitrary schema to the database without any recompiling of client code. This is done, in essence, by using generalized table-like objects in memory to model actual tables. Properties are handled as a collection of key-value string pairs, effectively implementing a dynamic-typing-like scheme. A final piece of niceness is that the API revolves around a service interface which makes it a lot nicer than many things for mocking when you test.

(If you so choose, you can use a code generator to hit your CRM install and spit out strongly-typed objects, but this is an extra feature about which it’s not trivial to find examples or information. It also spits out a file that’s a whopping 100,000+ lines, so caveat emptor.)

But that flexibility comes with a price, and that price is a rather cumbersome API. Here are a couple of places where I believe the API fails to some degree or another and where I think improvement would be possible, even as the flexible scheme and other good points are preserved.

Upside-Down Polymorphism

The API that I’m working with resides around a series of calls with the following basic paradigm:

VeryAbstractBaseRequest request = new SomeActualConcreteRequest();
VeryAbstractBaseResponse response = service.Execute(request);

There is a service that accepts very generic requests as arguments and returns very generic responses as return values. The first component of this is nice (although it makes argument capture during mocking suck, but that’s hardly a serious gripe). You can execute any number of queries and presumably even create your own, provided they supply what the base wants and provided there isn’t some giant switch statement on the other side downcasting these queries (and I think there might be, based on the design).

The second component of the paradigm is, frankly, just awful. The “VeryAbstractBaseResponse” provides absolutely no useful functionality, so you have to downcast it to something actually useful in order to evaluate the response. This is not big-boy polymorphism, and it’s not even adequate basic design. In order to get polymorphism right, the base type would need to be loaded up with methods so that you never need to downcast and can still get all of the child functionality out of it. In order not to foist a bad design on clients, it has to dispense with needing to base your response type on the request type. Here’s what I mean:

VeryAbstractBaseRequest request = new GetSomeCookiesRequest();
VeryAbstractBaseResponse response = service.Execute(request);
var meaningfulResponse = (GetSomeCookiesResponse)response; //Succeeds only because I have inappropriate knowledge of an implied coupling
var secondmeaningfulResponse = (GetSomeBrowniesResponse)response; //Bombs out because apparently brownies and cookies aren't fungible
var thirdMeaningfulResponse = (GetSomeBrownieBytesResponse)response; //Succeeds oddly because apparently BrownieBytes are children of Coookies (or a conversion operator exists)

Think about what this means. In order to get a meaningful response, I need to keep track of the type of request that I created. Then I have to have a knowledge of what responses match what requests on their side, which is perhaps available in some horrible document somewhere or perhaps is only knowable through trial/error/guess. This system is a missed opportunity for polymorphism even as it pretends otherwise. Polymorphism means that I can ask for something and not care about exactly what kind of something I get back. This scheme perverts that–I ask for something, and not only must I care what kind of something it is, but I also have to guess wildly or fail with runtime exceptions.

The lesson?

Do consumers of your API a favor. Be as general as possible when accepting arguments and as specific as possible when returning them. Think of it this way to help remember. If you have a method that takes type “object” and can actually do something meaningful with it, that method is going to be incredibly useful to anyone and everyone. If you have a method that returns type object, it’s going to be pretty much useless to anyone who doesn’t know how the method works internally. Why? Because they’re going to have to have an inappropriate knowledge of your workings in order to know what this thing is they’re getting back, cast it appropriately, and use it.

Hierarchical Data Transfer Objects

With this interface, the return value polymorphism issues might be livable if not for the fact that these request/response objects are also reminiscent of the worst in OOP, reminiscent of late ’90s, nesting-happy Java. What does a response consist of? Well, a response has a collection of entities and a collection of results, so it’s not immediately clear which path to travel, but if you guess right and query for entities, you’re just getting started. You see, entities isn’t an enumeration–it’s something called “EntityCollection” which brings just about nothing to the table except to provide the name of the entities. Ha ha–you probably wanted the actual entities, you fool. Well, you have to call response.EntityCollection.Entities to get that. And that’s an enumeration, but buried under some custom type called DataCollection whose purpose isn’t immediately clear, but it does implemented IEnumerable so now you can get at your entity with Linq. So, this gets you an entity:

RetrieveMultipleResponse response = new RetrieveMultipleResponse();
Entity blah = response.EntityCollection.Entities.First();

But you’re quite naieve if you thought that would get you anything useful. If you want to know about a property on the entity, you need to look in its Attributes collection which, in a bit of deja vu, is an AttributeCollection. Luckily, this one doesn’t make you pick through it because it inherits from this DataCollection thing, meaning that you can actually get a value by doing this:

RetrieveMultipleResponse response = new RetrieveMultipleResponse();
object blah = response.EntityCollection.Entities.First().Attributes["nameOfMyAttribute"];

D’oh! It’s an object. So after getting your value from the end of that train wreck, you still have to cast it.

The lesson?

Don’t force Law of Demeter violations on your clients. Provide us with a way of getting what we want without picking our way through some throwback, soul-crushing object hierarchy. Would you mail someone a package containing a series of Russian nesting dolls that, when you got to the middle, had only a key to a locker somewhere that the person would have to call and ask you about? The answer is, “only if you hated that person.” So please, don’t do this to consumers of your API. We’re human beings!

Unit Tests and Writing APIs

Not to bang the drum incessantly, but I can’t help but speculate that the authors of this API do not write tests, much less practice TDD. A lot of the pain that clients will endure is brought out immediately when you start trying to test against this API. The tests are quickly littered with casts, as operators, and setup ceremony for building objects with a nesting structure four or five references deep. If I’m writing code, that sort of pain in tests is a signal to me that my design is starting to suck and that I should think of my clients.

I’ll close by saying that it’s all well and good to remember the lessons of this post and probably dozens of others besides, but there’s no better way to evaluate your API than to become a client, which is best done with unit tests. On a micro-level, you’re dog-fooding when you do that. It then becomes easy to figure out if you’re inflicting pain on others, since you’ll be the first to feel it.

Feb
18

By

Practical Programmer Math: Bit Arithmetic

Category: Practical Math Tags: | 2 Comments

The “Why This Should Matter to You” Story

Imagine that you’re on a phone interview and things are going pretty well. You’ve fielded some gotcha trivia about whether such and such code would compile and what default parameters are and whatnot. After explaining the ins and outs of the strategy pattern and recursive methods, you’re starting to feel pretty good about yourself when, suddenly, the record skips. “Uh, excuse me?” you stammer.

“Why do integers roll over to negative when you loop all the way up to and past their max values?”

You had always just assumed that this was like an old Nintendo game where, naturally, when you completely leave the right side of the screen you simply appear on the left side. “Uh… they just do…? Pass?”

“No problem–let’s move on. Why do unsigned and signed integers occupy the same number of bits even though signed integers have more information?”

“Oh, they have more information? I know that unsigned integers can’t be negative!”

“Yes, well, okay. Let’s try this: how would you multiply a number by two without using the * operator?”

“Well,” you begin, “I’d–”

“And without adding it to itself.”

“Oh. Huh. I dunno.”

The interviewer sighs a little and says, “okay, so what can you tell me about the way numbers are represented as bits and bytes?”

Remembering the last practical math post, you talk about how you know that bits are 1 or 0 and that 8 of these bits makes up a byte. You talk about how bits can be and-ed and or-ed like booleans, and you offer hopefully that you even know an interesting bit of trivia in that 1K is not actually 1,000 bytes. The interviewer nods absently, and looking back later, you have no trouble pinpointing where things went off the rails. In spite of you knowing a bit about how information is represented at the lowest level, you realize you’re lacking some additional basics as to how this bit and bite stuff works.

You wrap up the interview and when you ask what the next steps are, the interviewer says, “don’t call us–we’ll call you.”

Math Background

First of all, you may want to go back and brush up on some previous posts in this series.

Let’s start with the simplest thing first: bit shifting. To help you understand bit-shifting, I’m going to introduce the concept of “bit significance.” Bit significance is a way of categorizing the “bits” in a binary number from most significant (the “most significant bit” or MSB) to least significant (the “least significant bit” or “LSB”). This is really just a fancy of way of describing the bits that correspond to the largest power of two from greatest to smallest. For example, the number six is written as 0x110, and we would say that the first 1 is the most significant; the middle one the next most; and 0 the least since they correspond to the 4, 2, and 1 values, respectively.

This may seem like a definition so basic as to be pointless, but it’s important to remember that not all architectures necessarily interpret bits from left to right. After all, reading “110” as 6 is rather arbitrary–we could just as easily read it as 3 if we read it the other way. This is reminiscent of how some Middle Eastern written languages are interpreted from right to left. So just to be clear, for the remainder of this post, the MSB will be the leftmost bit and the LSB will be the rightmost.

Bit shifting, often denoted in programming languages by the “>>” and “<<” operators, is the simple procedure of adding a zero to one end or the other of a binary sequence. For example, consider the following code:

int x = 0x0011;  //Set x = 3
int y = x << 1; //Left shift by 1 place

What you would wind up with for y is 0x0110 or 6. Notice how there’s now an extra 0 on the right and this results in the value being multiplied by two. If we had bit-shifted by 2 instead of 1, we’d have 0x1100 or 12, which is 3 x 4. If you bit-shift the other way, you get back to where you started:

int x = 0x1100;  //Set x = 12
int y = x >> 2; // Right shift by 2 places

Here, y winds up being 0x0011 or 3 again. Performing a subsequent shift would result in 0x0001 or 1. When you divide unevenly by two, the algorithm ’rounds’ down. Another and perhaps easier way to understand this shifting concept is to realize it’s actually pretty familiar with the decimal numbers that you’re accustomed to. For example, if you “decimal shifted” the number 14 left, you’d have 140 and if you did it to the right, you’d wind up with 1 (rounding down from 1.4). Decimal shifting, as anyone who has “misplaced a zero” knows, is really just a question of multiplying and dividing by powers of 10. Bit-shifting is the same thing, but with powers of two.

Now that you understand a bit about bit significance and the concept of shifting, let’s tackle the problem of wrap-around and integer overflow. In dealing with binary numbers so far, we’ve dealt exclusively with positive integers and zero, meaning that we haven’t dealt with negative numbers or non-integers like decimals. The latter will be fodder for a future post on floating point arithmetic, but let’s consider how to represent a negative number in binary.

PhoneInterviewIf it were simply a matter of writing it out, that’d be easy. We could simply write -0x11 for -3, the way we do on paper. But when dealing with actual bits on a computer, we don’t have that luxury. We have to signify negativity somehow using the same ones and zeroes that we use for representing the number. Given that negative/positive is an either/or proposition, the common way to do this is with a bit–specifically, the MSB. This lets you represent negativity, but it necessarily cuts in half the count of numbers that you can represent.

Think of a byte, which is 8 bits. Using all 8 bits, you can represent 2^8 = 256 numbers (0 through 255). But using 7, you can only represent 2^7 = 128 (0 through 127). Think of the language construct “unsigned” in most programming languages (with “signed” being the default). When you’re making this distinction, you’re telling the compiler whether or not you want to sacrifice one of the bits in your number type to use as the +/- sign.

Great, but how do we get to the rollover issue? Well, the reason for that is a little more complicated and it involves a concept called “two’s complement.” Let’s say that we represented binary numbers by taking the MSB and making it 0 for positive and 1 for negative. That’s probably what you were picturing in the last paragraph, and it’s called “sign magnitude” representation. It’s workable, but it has the awkward side effect of giving us two ways to represent 0. For instance, consider a signed 3 bit scheme where the first bit was the sign and the next two the number. 101 and 001 would represent negative and positive 1, respectively, but what about 0x100 and 0x000. Positive and negative zero? It also requires different mechanics for arithmetic operations depending on whether the number is positive or negative, which is computationally inefficient. And finally, the fact that there are two versions of zero means that a potential number that could be represented is wasted.

Instead, processors use a scheme called “two’s complement.” In this scheme, the MSB still does wind up representing the sign, but only by side effect. The positive numbers are represented the way they would be in the binary numbers you’re used to seeing. So, in a 4 bit number, the largest normal number would be 7: 0x0111. But when you flip that last bit and go to what would be 8 as a signed number, 0x1000, you’re suddenly representing -8. If you keep counting, you get pairs of (0x1001 or -7), (0x1010 or -6) … (0x1111 or -1). So you walk the scheme up to the most positive number, which you arrive at halfway through, and then you suddenly hit the most negative number and start working your way back to zero. Adding 1 to 0x0111, which corresponds to Integer.Max for a 4 bit integer, would result in 0x1000, which corresponds to Integer.Min.

Negative and positive that you’re used to operate as a line stretching out either way to infinity with the two concepts colliding at zero. But in a two’s complement scheme, they act as a wheel, and you “roll over.” The reason that two’s complement behaves this way is a bit more involved, and I may cover it in a smaller future post, but suffice it to say that it makes bit arithmetic operations easier to reason about algorithmically and more elegant. It also gets rid of that pesky double zero problem.

How it Helps You

In the short term all of this helps you answer the phone interview questions, but I don’t think of that as too terribly important since I consider trivia questions during the interview process to be a hiring anti-pattern, often indicative of a department rife with petty oneupsmanship and knowledge hoarding. You’re better off without that job, and, now that you know the answers to the questions, you have the best of all worlds.

On a longer timeline, you’re learning the fundamentals of how data is represented and works in its most core form in computer science, and that opens some nice doors, in general. Superficially, knowing to bit-shift instead of multiply or divide may allow you to optimize in some environments (though I believe that with most modern compilers they can do pretty well optimizing this on their own). But at a deeper level, you’re gaining an understanding of the internal workings of the data types that you’re using.

When data is sent “over the wire” and you’re examining it with some kind of dump tool, it’s coming in streams of bits. Sure there’s an endpoint somewhere re-assembling it into things like unsigned ints and floats and even higher level abstractions. But if those are the lowest levels at which you understand data, it’s going to make programs hard to debug when you’re using a tool that shows you raw data. It may be tedious to interpret things at that level, but in a tight spot it can be a life saver.

Further Reading

  1. A pretty nice guide to bit shifting
  2. Detailed explanation of Two’s Complement
  3. A nice video on Two’s Complement
  4. A comparison of different schemes for representing negative in bits
By the way, if you liked this post and you're new here, check out this page as a good place to start for more content that you might enjoy.

Feb
15

By

Switch Statements are Like Ants

Category: Abstractions, Language Agnostic Tags: |

Switch statements are often (and rightfully, in my opinion) considered to be a code smell. A code smell, if you’ll recall, is a superficial characteristic of code that is often indicative of deeper problems. It’s similar in concept to the term “red flag” for interpersonal relationships. A code smell is like someone you’ve just met asking you to help them move and then getting really angry when you don’t agree to do it. This behavior is not necessarily indicative of deep-seated psychological problems, but it frequently is.

Consequently, the notion that switch statements are a code smell indicates that if you see switch statements in code, there’s a pretty good chance that design tradeoffs with decidedly negative consequences have been made. The reason I say this is that switch statements are often used to simulate polymorphism for those not comfortable with it:

public void Attack(Animal animal)
{
    switch (animal.Type)
    {
        case AnimalType.Cat: 
            Console.WriteLine("Scratch"); 
            break;
        case AnimalType.Dog: 
            Console.WriteLine("Bite"); 
            break;
        case AnimalType.Wildebeest: 
            Console.WriteLine("Headbutt"); 
            break;
        case AnimalType.Landshark: 
            Console.WriteLine("Running-bite");
            break;
        case AnimalType.Manticore: 
            Console.WriteLine("Eat people");
            break;
    }
}

Clearly a better design from an OOP perspective would be an Animal base class/interface and an Attack() method on that, overridable by children/implementers. This design has the advantage of requiring less code, scaling better, and conforming to the Open/Closed principle–if you want to add some other animal later, you just add a new class to the code base and probably tweak a factory method somewhere and you’re done.

This method isn’t really that bad, though, compared to how bad it could be. The design is decidedly procedural, but its consequences aren’t far reaching. From an execution perspective, the switch statement is hidden as an implementation detail. If you were to isolate the console (or wrap and inject it), you could unit test this method pretty easily. It’s ugly and it’s a smell, but it’s sort of like seeing an ant crawling in your house–a little icky and potentially indicative of an infestation, but sometimes life deals you lemons.

But what about this:

public string Attack(Animal animal)
{
    switch (animal.Type)
    {
        case AnimalType.Cat:
            return GetAttackFromCatModule();
        case AnimalType.Dog:
            return GetAttackFromDogModule();
        case AnimalType.Wildebeest:
            return GetAttackFromWildebeestModule();
        case AnimalType.Landshark:
            return GetAttackFromLandsharkModule();
        case AnimalType.Manticore:
            return GetAttackFromManticoreModule();
    }
}

Taking the code at face value, this method figures out what the animal’s attack is and returns it, but it does so by invoking a different module for each potential case. As a client of this code, your path of execution can dive into any of five different libraries (and more if this ‘pattern’ is followed for future animals). The fanout here is out of control. Imagine trying to unit test SavageAntthis method or isolate it somehow. Imagine if you need to change something about the logic here. The violence to the code base is profound–you’d be changing execution flow at the module level.

If the first switch state was like an ant in your code house, this one is like an ant with telephone poles for legs, carving a swath of destruction. As with that happening in your house, the best short-term strategy is scrambling to avoid this code and the best long-term strategy is to move to a new application that isn’t a disaster zone.

Please be careful with switch statements that you use. Think of them as ants crawling through your code–ants whose legs can be tiny ant legs or giant tree trunk legs. If they have giant tree trunk legs, then you’d better make sure they’re the entirety of your application–that the “ant” is the brains ala an Ioc container–because those massive swinging legs will level anything that isn’t part of the ant. If they swing tiny legs, then the damage only occurs when they come in droves and infest the application. But either way, it’s helpful to think of switch statements as ants (or millipedes, depending on the number of cases) because this forces you to think of their execution paths as tendrils fanning out through your application and creating couplings and dependencies.

By the way, if you liked this post and you're new here, check out this page as a good place to start for more content that you might enjoy.

Feb
13

By

The Council of Elders Anti-Pattern

Category: The Life of a Programmer Tags: , | 2 Comments

Convene the High Council

Today I’d like to talk about something that I’ve observed on some occasions throughout my career. It sort of builds on the concepts that I’ve touched on in previous writings “How to Keep your Best Programmers” and the Expert Beginner posts, including Bruce Webster’s “Dead Sea Effect.” The phenomenon is an anti-pattern that is unique to shops with fairly mature Dead Sea/Expert Beginner presences, and I’ll call it “Council of Elders.”

To understand what I mean by this, let’s consider what happens in an organization with a more vital software group full of engaged, competent, and improving engineers and not bogged down in the politics and mediocrity of seniority-based advancement. In such a group, natural divisions of labor emerge based on relative expertise. For example, you might have an effective team with the “database guy” and the “architecture gal” and “markup guru” and the “domain knowledge expert” or some such thing. Each resident expert handles decision making within his or her sphere of influence and trusts other team members to likewise do the right thing in their own areas of strength. There might be occasional debate about things that blur the lines or occur on the boundary between two experts, but that’s just part of a healthy group dynamic.

And truly, this is a healthy group. There is no dead weight, and all of the group members have autonomy within a subset of the work that is done, meaning that people are happy. And that happiness breeds mutual trust and productivity. I’m oversimplifying and prettying up the counterexample a bit, but it really is something of a virtuous cycle with team members happily playing to their own strengths and areas of interest.

TheCouncilBut what happens in particularly salty Dead Sea shops where authority comes neither from merit nor from expertise, but rather from tenure with the company? What happens when the roost is ruled by Expert Beginners? Well, for one thing, the lords and ladies here tend to guard their fiefdoms more jealously since expertise on the part of others is more threat than benefit. Perhaps more importantly and broadly, however, knowledge and expertise are devalued in favor of personal politics and influence with debates being won on the basis of “who are you and how loud/charming/angry/glib/etc. are you” rather than on the merit of ideas. The currency of Dead Sea departments is everything but ideas–in benevolent ones, it may be “how long have you been here” or “how nice a person are you,” and, in “high pressure culture” ones, it might simply be psychopathy or other cutthroat forms of “might makes right.” And with evaluation of ideas out the window, every council member is freed to hold forth as an expert on every topic, regardless of how much or little he knows about that topic. Nobody is going to dispute anything he says–non-members are cowed into submission and fellow members recognize the importance of putting on a unified public front since they want to be be able to do the same without being questioned.

If you put this political yeast in the oven and let it rise, some of the results are fairly predictable: idea stagnation, increasingly bone-headed solutions to problems, difficulty keeping talent, etc. But an interesting consequence isn’t necessarily intuitive–namely that you’ll wind up with a kind of cabal of long-tenured people that collectively make all decisions, however big or small. I call this the “Council of Elders,” and it’s like one of those Magic Eye paintings that’s hidden but couldn’t be more obvious once you see it.

The Council of Elders is sort of like the Supreme Court of the department, and it’s actually surprisingly democratic as opposed to the more expected ladder system which ranks people by years and months with the company (or if you’re a fan of The Simpsons, the system in the Stonecutters episode where all members of the club are assigned a numeric rank in the order of joining, which determines their authority). The reason that it’s democratic is that actually assigning rank based on years/months of tenure would unceremoniously poke a hole in any illusion of meritocracy. So the council generally makes entrance to the club a matter of tenure, but status within the club a shifting matter of alliances and status games once past the velvet rope.

The Council is fundamentally incapable of delegation or prioritization of decision making. Since entrance is simply a matter of “paying your dues” (i.e. waiting your turn) and largely earned by Expert Beginners, it’s really impossible to divide up decision making based on expertise. The members tend to be very good at (or at least used to) company politics and procedures but not much else. They mostly have the same ‘skill’ set. The lack of prioritization comes from the main currency in the group being status. If a decision, however small, is made by someone not on the Council, it threatens to undermine the very Council itself, so a policy of prevention is adopted and any attempts at circumvention are met with swift and terrible justice (in the form of whatever corporate demerits are in place).

Recognizing Your Elders

What does this council look like in the wild, and how do you know if one is pulling your strings? Here’s a set of symptoms that you’re being governed by a Council of Elders:

  • In any meeting convened to make a decision, the same group of people with minor variants is always present.
  • Members of the software group are frequently given vague instructions to “talk to so and so before you do anything because his cousin’s dog’s trainer once did something like that or something.”
  • There is a roster, emailed out or tacked up, of people who can “clear” you to do something (e.g. they give code reviews, approve time spent, etc.)
  • Sparks fly when something wasn’t “cleared” with someone, even when a few others “approved” it and it has apparently nothing to do with him. “Someone should have told me.” More than one or two people like this and you have a Council.
  • People in the group often feel caught between a rock and a hard place when involved in the political posturing of the Council (e.g. Yoda tells the intern Padiwan to implement the new hours tracking app using SQL Server, but then Mace Windu screams at him that this is a MySQL shop–the Council settles this matter later with no apologies)
  • There is a junta of team members that seem to do nothing but shuffle from meeting to meeting all day like mid-level managers.
  • There are regular meetings viewed by newbies and junior developers as rites of passage.

These are just easy ways to detect Councils in their natural habitat: “where there’s smoke, there’s fire” situations. But lest you think that I’m trying to paint every software department with senior developers that go to meetings with this brush, here is a list of specific criteria — “minimum Council Requirements,” if you will:

  • Lack of clearly defined roles on the group/team or else lack of clear definition for the assigned roles (e.g. there is no ‘architect’ or if there is, no one really knows what that means).
  • A “dues paying” culture and promotions, power and influence largely determined by length of tenure.
  • Lack of objective evaluation criteria of performance, skill and decision-making acumen (i.e. answers to all questions are considered subjective and matters of opinion).
  • Proposed changes from above are met with claims of technical infeasibility and proposed changes from juniors/newbies or other groups are met with vague refusals or simply aren’t acknowledged at all.
  • Actions of any significance are guarded with gatekeeper policies (official code reviews, document approval, sign-off on phases, etc).
  • Line manager tacitly approves of, endorses or is the product of the Council.
  • A moderate to large degree of institutional bureaucracy is in place.
  • New technologies, techniques, approaches, etc are met systematically with culturally entrenched derision and skepticism.

Not a Harmless Curiosity

At this point, you might think to yourself, “So what? It might not be ideal, but rewarding people with a little gratifying power in exchange for company loyalty is common and mostly harmless.” Or perhaps you’re thinking that I’m overly cynical and that the Council generally has good advice to dispense–wisdom won in the trenches. Of course all situations are different, but I would argue that a Council of Elders has a relatively noticeable and negative effect on morale, productivity, and general functioning of a group.

  • The Council is a legitimate bottleneck in its micromanagement of other team members, severely hampering their productivity even with the best assumption of its judiciousness
  • A group of people that spends all its time debating one another over how to rule on every matter doesn’t actually get much work done. The SCOTUS, for example, hasn’t represented a lot of clients lately, but that’s because their job is to be a ruling council–yours probably has titles like, “Senior Software Engineer.”
  • People lacking expertise but put in positions of authority tend to overcompensate by forming strong opinions and sticking to them stubbornly. A room full of people meeting this criterion is going to resemble the US House of Representatives with its gridlock more than a software team.
  • Major problems are likely to languish without solutions because the committee doesn’t do much prioritizing and is likely to be sidetracked for a few days by the contentious issue of what color to make the X for closing the window.
  • Important decisions are made based on interpersonal dynamics among the council rather than merit. Jones might have the best idea, but Jones shorted the check at lunch that day, so the other Council Members freeze him out.
  • Councils make it hard to give meaningful roles or titles to anyone and thus give rise to preposterous situations in which departments have eight ‘architects’ and five developers, or where a project manager decides what database to use while a DBA writes requirements. If everyone on the Council has authority and is an expert at everything, project roles are meaningless anyway.
  • Even under the best circumstances, democratic voting on software design trends toward mediocrity: see Design by Committee Anti-Pattern
  • People toiling away under the rule of a Council, if they don’t leave, will tend to wind up checked out and indifferent. Either that, or they’ll be assimilated into the Council culture.
  • The Council is a natural preservation agent of the Dead Sea problem. Being micromanaged by a team of people whose best qualification is being around for a while and negotiating power will separate the wheat from the chaff, but not in the good way.
  • The only thing that unites the Council is outside threats from above or below. If managers or newer members want change, the Council will lock together like a Praetorian Legion to preserve the status quo, however dysfunctional that may be.
  • Suggestions for improvement are construed with near universality as threats.

If you’re in a position to do so, I’d suggest breaking up the cartel. Figure out what people are good at and give them meaningful roles. Keep responsibilities divided, and not only will autonomy become more egalitarian, but also people tenured and new alike will develop complementary skills and areas of expertise. With people working together rather than forming shifting alliances, you’ll find prioritizing tasks, dividing up work, and getting things done becomes easier. The Council’s reptile brain will take over and it will fight you every step of the way when you try to disband it, but once you do, it will be good for everyone.

Feb
11

By

TDD: Simplest is not Stupidest

Category: Language Agnostic Tags: , | 4 Comments

Where the Message Gets Lost In Teaching TDD

I recently answered a programmers’ stackexchange post about test-driven development. (As an aside, it will be cool when me linking to a SE question drives more traffic their way than them linking to me drives my way 🙂 ). As I’m wont to do, I said a lot in the answer there, but I’d like to expand a facet of my answer into a blog post that hopefully clarifies an aspect of Test-Driven Development (TDD) for people–at least, for people who see the practice the way that I do.

One of the mainstays of showcasing test-driven development is to show some extremely bone-headed ways to get tests to pass. I do this myself when I’m showing people how to follow TDD, and the reason is to drive home the point “do the simplest thing.” For instance, I was recently putting together such a demo and started out with the following code:

[TestMethod, Owner("ebd"), TestCategory("Proven"), TestCategory("Unit")]
public void IsEven_Returns_False_For_1()
{
    var inspector = new NumberInspector();
 
    Assert.IsFalse(inspector.IsEven(1));
}

public class NumberInspector
{
    public bool IsEven(int target)
    {
        return false;
    }
}

This is how the code looked after going from the “red” to the “green” portion of the cycle. When I used CodeRush to define the IsEven method, it defaulted to throwing NotImplementedException, which constituted a failure. To make it pass, I just changed that to “return false.”

The reason that this is such a common way to explain TDD is that the practice is generally being introduced to people who are used to approaching problems monolithically, as described in this post I wrote a while back. For people used to solving problems this way, the question isn’t, “how do I get the right value for one,” but rather, “how do I solve it for all integers and how do I ensure that it runs in constant time and is the modulo operator as efficient as bit shifting and what do I do if the user wants to do it for decimal types should I truncate or round or throw an exception and whoah, I’m freaking out man!” There’s a tendency, often fired in the forge of undergrad CS programs, to believe that the entire algorithm has to be conceived, envisioned, and drawn up in its entirety before the first character of code is written.

So TDD is taught the way it is to provide contrast. I show people an example like this to say, “forget all that other stuff–all you need to do is get this one test passing for this one input and just assume that this will be the only input ever, go, now!” TDD is supposed to be fast, and it’s supposed to help you solve just one problem at a time. The fact that returning false won’t work for two isn’t your problem–it’s the problem of you forty-five seconds from now, so there’s no reason for you to bother with it. Live a little–procrastinate!

You refine your algorithm only as the inputs mandate it, and you pick your inputs so as to get the code doing what you want. For instance, after putting in the “return false” and getting the first test passing, it’s pretty apparent that this won’t work for the input “2”. So now you’ve got your next problem–you write the test for 2 and then you set about getting it to pass, say with “return target == 2”. That’s still not great. But it’s better, it was fast, and now your code solves two cases instead of just the one.

Running off the Rails

But there is a tendency I think, as demonstrated by Kristof’s question, for TDD teachers to give the wrong impression. If you were to write a test for 3, “return target == 2” would pass and you might move on to 4. What do you do at 4? How about “return target == 2 || target == 4;”

So far we’ve been moving in a good direction, but if you take off your “simplest thing” hat for a moment and think about basic programming and math, you can see that we’re starting down a pretty ominous path. After throwing in a 6 and an 8 to the or clause, you might simply decide to use a loop to iterate through all even numbers up to int.MaxValue, or-ing a return value with itself to see if target is any of them.

public bool IsEven(int target)
{
    bool isEven = false;
    for (int index = 0; index < int.MaxValue - 1; index += 2)
        isEven |= target == index;
    return isEven;
}

Yikes! What went wrong? How did we wind up doing something so obtuse following the red-green-refactor principles of TDD? Two considerations, one reason: "simplest" isn't "stupidest."

Simplicity Reconsidered

The first consideration is that simple-complex is not measured on the same scale as stupid-clever. The two have a case-by-case, often interconnected relationship, but simple and stupid aren't the same just as complex and clever aren't the same. So the fact that something is the first thing you think of or the most brainless thing that you think of doesn't mean that it's the simplest thing you could think of. What's the simplest way to get an empty boolean method to return false? "return false;" has no branches and one hardcoded piece of logic. What's the simplest way that you could get a boolean method to return false for 1 and true for 2? "return target == 2" accomplishes the task with a single conditional of incredibly simple math. How about false for 1 and true for 2 and 4? "return target % 2 == 0" accomplishes the task with a single conditional of slightly more involved math. "return target == 2 || target == 4" accomplishes the task with a single conditional containing two clauses (could also be two conditionals). Modulo arithmetic is more elegant/sophisticated, but it is also simpler.

Now, I fully understand the importance in TDD of proceeding methodically and solving problems in cadence. If you can't think of the modulo solution, it's perfectly valid to use the or condition and put in another data point such as testing for IsEven(6). Or perhaps you get all tests passing with the more obtuse solution and then spend the refactor phase refining the algorithm. Certainly nothing wrong with either approach, but at some point you have to make the jump from obtuse to simple, and the real "aha!" moment with TDD comes when you start to recognize the fundamental difference between the two, which is what I'll call the second consideration.

The second consideration is that "simplest" advances an algorithm where "stupidest" does not. To understand what I mean, consider this table:

ConditionalClauseChart

In every case that you add a test, you're adding complexity to the method. This is ultimately not sustainable. You'll never wind up sticking code in production if you need to modify the algorithm every time a new input is sent your way. Well, I shouldn't say never--the Brute Forces are busily cranking these out for you to enjoy on the Daily WTF. But you aren't Brute Force--TDD isn't his style. And because you're not, you need to use either the green or refactor phase to do the simplest possible thing to advance your algorithm.

A great way to do this is to take stock after each cycle, before you write your next failing test and clarify to yourself how you've gotten closer to being done. After the green-refactor, you should be able to note a game-changing refinement. For instance:

MilestoneTddChart

Notice the difference here. In the first two entries, we make real progress. We go from no method to a method and then from a method with one hard-coded value to one that can make the distinction we want for a limited set of values. On the next line, our gains are purely superficial--we grow our limited set from distinguishing between 2 values to 3. That's not good enough, so we can use the refactor cycle to go from our limited set to the complete set.

It might not always be possible to go from limited to complete like that, but you should get somewhere. Maybe you somehow handle all values under 100 or all positive or all negative values. Whatever the case may be, it should cover more ground and be more general. Because really, TDD at its core is a mechanism to help you start with concrete test cases and tease out an algorithm that becomes increasingly generalized.

So please remember that the discipline isn't to do the stupidest or most obtuse thing that works. The discipline is to break a large problem into a series of comparably simple problems that you can solve in sequence without getting ahead of yourself. And this is achieved by simplicity and generalizing--not by brute force.

« Older Posts
Newer Posts »