The Dirty Work for Software Architects

Last week, I spent some time in two Egyptian cities: Alexandria and Cairo. I enjoyed this a lot. The Egyptian people were very friendly and good-natured, and there were some truly fascinating sites to see. Beyond just the obvious trip to the Pyramids, I saw Greco-Roman ruins, an incredible library in Alexandria, forts constructed during the Middle Ages, etc. These cities were very much alive and had distinct and enjoyable personalities.

One thing that really struck me, on the negative side of the ledger, however, was how much garbage there was everywhere. It looked as though it was trash day for the entire city and some kind of low-grade typhoon passed through, leaving the buildings intact and scattering trash everywhere. The problem was so bad that, in the canals feeding off the Nile, the government had actually started to fill them in with dirt, stopping the flow of water to prevent local children from playing in what had essentially become open sewage.

One might chalk this up to some kind of cultural difference or something, but that didn’t seem right to me. As I made my way around, I saw shopkeepers sweeping up their floors, people scrubbing their windows, occupants picking up debris from the sidewalks. Residents clearly cared about the conditions of their surroundings. This struck me as some kind of logistical problem, and a little googling around upon my return seemed to confirm this suspicion. Apparently, since the overthrow of Mubarak, public works suffered and the programs implemented by Morsi haven’t sufficiently addressed the problem. In Egypt, more garbage is currently generated than is hauled away–to the dismay of the locals, I’d imagine.

On the long, long flight home a few days later, my mind wandered between the trip I had just experienced and the work to which I was returning. I thought about the role of architect on a project and how in some ways it’s like a role of governance. In realpolitik context such as discussed in my post about job titles, it’s like government in that the role comes with a lot of perks that people tend to take for granted, at times to the extent of forgetting what they’re really supposed to be doing. This gives rise to the classic jokes about supervision being a role that involves doing nothing.

But in a less cynical context, the role of the architect is one of vision, persuasion, balancing concerns and understanding business needs such as keeping costs down. And this is very similar to government as it’s meant to be undertaken. It’s not all slush funds, lobbyist-bought dinners, and cigars–you need to get things done. Some of that work may leave your stamp on the world (or at least the group for which you’re the architect), allowing you to tout that you introduced continuous integration or forged ahead with a new, distributed architecture. But sometimes, you just have to pick up the garbage, even when you want to build pyramids.

Architect as Trash Collector

I’ve seen a lot of different code bases and approaches over the years. Sometimes you see well curated, organized code bases being maintained by conscientious developers, but this is probably the least common scenario because it’s hard and requires training and practice. The most common scenario tends to be that a rather rotten code base is being maintained by developers whose attitudes range from checked out and indifferent to interested but frustrated. Code bases tend toward entropy, and the longer they’ve existed, often the more run-down they are. The garbage starts to pile up in the streets.

But surprisingly, I’ve found that a scenario you see more than you’d expect is one where the code base is in a sorry state but the developers are conscientious and hard working. And this is what my mind kept coming back to as I reflected on my tour through parts of Egypt–hardworking people in a frustrating and often futile struggle to be clean in an unclean world too big for them to control. The developers do their best with new code and improve legacy code when they can, following the Boy Scout rule, but improvement is slow and painful, if not at a standstill.

How would you fix this as architect? Start over with a complete, grand rewrite? Good luck with that because, if you’ll recall, one of the things that the architect should do, like the government, is understand value tradeoffs and budgets. Rewriting the software is sort of like saying, “Well, retirees, sorry, but paying your social Social Security is really dragging us down…” It’s a non-starter. Do you order some pyramids built–some introduction of an impressive, slick piece of latest and greatest framework, design pattern, or tool? Hmm… I’m reminded of Percy Shelley’s Ozymandias–terrible in your greatness and power, all that will remain of your architecture when you’re five years gone from the company is dust and the occasional, cryptic comment in the code about the great things you were going to do. “I am Ozymandias, Architect of architects–look on my code generation ye Mighty, and despair!”

Governed agencies that succeed are built on good infrastructure and logistics. Non-glamorous things like those are what governance is really all about. Building impressive, shiny things is a luxury and a thing that governments built to last only do when the lifeblood of the government is healthy and flowing and the foundation is strong. When communication fails and rubbish piles up in the streets, that has to be addressed first. So what does a good, governing architect do? Well, he rolls up his sleeves and takes out the trash.

You Have to Get Your Hands Dirty

You do drudgery clean-up. You methodically go through the code base breaking apart 1000 line methods into 200 line ones. You break those into 40 line ones and those into 5-10 line ones. You get rid of copy-paste jobs in the same way, and you add automation to your builds, tests, and process as you go. People laugh at first and think you’re crazy. They tell you that the architect should be meeting with the department heads and executives, drawing massive UML diagrams and entity-relationship models. They’re right–you have to do those things and this too. So you chip away in the hour before you leave but after everyone else has taken off. You remote in on weekends. You make progress and build sweat equity. People stop laughing at you and join you, appreciating that you work as hard as they do when you don’t have to anymore and that you’re rolling your sleeves to take the trash out. It’s infectious, and you start to fix the problems in earnest and make the infrastructure hum as improvements are noticeable.

I think that this is what an architect ought to be. You do all of the planning and technical lead work, making decisions, mentoring developers, strategizing and scheming, but you lead the way in the trenches, too. You work hard making life better for the developers on the team, ensuring their code base stays clean and easy to work with. You keep the potholes filled, the crime down, and the garbage out of the street.

In with the Proto-Geeks, Out with the Loafers

Technology Progressives vs Conservatives Reconsidered

When it comes to our relationship with technology, most people think in terms of the classic struggle between inventors that capture the imagination and laborers that don’t want to be imagination-captured right out of a steady paycheck. I think this is a rather obvious way to consider technology in our lives, but that there’s another dimension that sort of kicks in as an “aha” moment in much the same way that a line between “left” and “right” politically is trumped by the political compass quadrant system that also addresses libertarianism vs authoritarianism.

Towards that end, I propose quadrants related to technology. On one axis, there is the line from Luddite to Technologist. This is the line along which enjoyment of technology ranges when it comes to aesthetics and emotions. Luddites have a visceral dislike for technology, often because it makes them feel insecure or afraid (e.g. laborers in a factory who are about to be replaced by robots). Technologists have an emotional connection with technological advancement, generally because they find change and new possibilities exciting and intriguing (e.g. people that read about scientific advances, hobbyists that build robots, etc.).

On the other axis, there is a range from Laborer to Automater. Along this line, applied and practical interaction with technology varies. The Laborer will have little opportunity and/or motivation to use technology to automate day-to-day tasks (e.g. an artisan or a member of a think tank). Automaters embrace optimizations brought about through the use of technology and will typically seek to be as efficient as possible. People at this end of the spectrum value technology in a pragmatic sort of way, using it as well as process to optimize for time, cost, etc. People who use mobile devices to track appointments and tasks to the point of hyper-organization are good examples.

These two axes yield the following graph, with labeled quadrants.

The most obvious quadrants are the Luddite-Laborer and Technologist-Automater, as those are the roles that we think of as the Curmudgeon and the Proto-Geek, respectively. Curmudgeons hate and regard with suspicion anything that is newfangled and unfamiliar, while their polar opposite, Proto-Geeks, embrace things specifically because they’re new and flashy. All of the elements of the ethos of hard work and humble living (Protestant Work Ethic to the point of absurdity) versus endless, powerful novelty (decadence, convenience, playing God) come into play here, and this represents the linear spectrum that most envision about technology. The less frequently discussed quadrants are the ones that I’m calling, “Fanboy” and, “Loafer.”

“Fanboy” is the set of people that enjoy technology for its aesthetics and have a concept of tech-chic. These are people who will buy something like a cell phone for an accessory more so than for the various practical purposes of automating parts of their lives such as the aforementioned task tracking and organization. Non-technical gamers may also fall in this quadrant. The other less commonly considered quadrant is the Loafer, who prefers to automate things in life but has no love for new technologies and techniques. Contradictory as this may seem, it’s actually relatively common. And worse, it’s toxic to your software development group.

Inside the Mind of the Loafer

To understand the problem, it’s essential to understand what makes the Loafer tick. First of all, the Loafer isn’t a malingerer in the sense that he’ll shirk his responsibilities or do work that is of inferior quality. Rather, the Loafer seeks to optimize efficiency so that the value proposition of doing work is improved. In other words, if you task a Loafer with alphabetizing a list of 50,000 names, he’ll write a program to do it because that gets the job done more quickly. He’ll charge you the same rate regardless of methodology, though, so he’s automating specifically to allow himself more free time (or profit, as it were).

By way of contrast, consider the Proto-Geek’s response to this same task. She’d most likely blurt out, “That’s easy! I could do that in any programming language–you know what? I’ll do it in Ruby because that will give me a chance to play with Ruby. Cost? Oh, geez, I dunno. Maybe ten bucks…? It’s really no big deal.” The Proto-Geek is interested in solving the problem and leveraging cool new techs to do it. Profit/time optimizing is a secondary concern. (It is also interesting to consider the reaction of the other quadrants to this scenario. Curmudgeon would probably extol the virtues of doing it by hand on your mental acuity, and Fanboy would probably do it by hand while tweeting about how there should be something that did this automatically.)

Proto-Geeks view automation as having a high cool factor. This is in contrast to Loafer, who views it as having a high profit factor. But why is this a detriment to your group? After all, it stands to reason that Loafer has the best business sense. It seems as though it’d be important to have people on the team that chose the best tool for the job rather than because they were simply excited to use it. And that is true. The problem is that Automaters further down the chart toward Luddite are operating in a way to maximize their own personal efficiency and profit, rather than the group’s.

Most Loafers in a software group don’t particularly like what they do, but they know that if they do more of it faster, they get more money–or more time to, well, Loaf. So once they find a set of tools, languages, frameworks, etc. and become facile with them, they’re not only disinterested in new technologies–they’re actively resistant. Loafers want nothing more than to continue to solve all problems that come their way as quickly and with as little effort as possible, but with a locally-maximizing, greedy approach; investing a lot of extra effort in the shorter term to learn new ways of doing things more efficiently down the road is not appealing.

If you have one or two Loafers in your group, they can serve the useful purpose as natural conservatives that prevent Proto-Geeks from running buck-wild with new techs and frameworks. But this dynamic must be carefully monitored, like an inoculation, since the proportions can quickly become toxic. Tip the group’s dynamic a little too far into the Loafer quadrant and suddenly you have a bad case of “not invented here” and complete resistance to new ideas. There’s also frequently a strong correlation between Loafers and Expert Beginnerism; established Loafers often have a track record of some success with their approach and thus a powerful, built-in argument for inertia and the status quo. And finally, Loafers tend to kill enthusiasm for learning pretty quickly since presentation of new technologies is met with sarcasm, skepticism, and derision, making it essentially uncool to learn and embrace new things lest one be accused naivete or being some kind of shill.

If you’re in a position of authority within a group and recognize a Loafer dynamic, I’d suggest shaking things up to see if you can re-awaken or inspire a Technologist in some of them. Or perhaps you can infuse the group with some fresh-faced Proto-Geeks. If, on the other hand, you’re a powerless Proto-Geek in a Loafer group, you have a range of options from persuasion to proof-of-concept pitches to management to dusting off your resume. In either case, I think it’s important to recognize your surroundings and to try to seek out the quadrant in which you’re most comfortable.

The Council of Elders Anti-Pattern

Convene the High Council

Today I’d like to talk about something that I’ve observed on some occasions throughout my career. It sort of builds on the concepts that I’ve touched on in previous writings “How to Keep your Best Programmers” and the Expert Beginner posts, including Bruce Webster’s “Dead Sea Effect.” The phenomenon is an anti-pattern that is unique to shops with fairly mature Dead Sea/Expert Beginner presences, and I’ll call it “Council of Elders.”

To understand what I mean by this, let’s consider what happens in an organization with a more vital software group full of engaged, competent, and improving engineers and not bogged down in the politics and mediocrity of seniority-based advancement. In such a group, natural divisions of labor emerge based on relative expertise. For example, you might have an effective team with the “database guy” and the “architecture gal” and “markup guru” and the “domain knowledge expert” or some such thing. Each resident expert handles decision making within his or her sphere of influence and trusts other team members to likewise do the right thing in their own areas of strength. There might be occasional debate about things that blur the lines or occur on the boundary between two experts, but that’s just part of a healthy group dynamic.

And truly, this is a healthy group. There is no dead weight, and all of the group members have autonomy within a subset of the work that is done, meaning that people are happy. And that happiness breeds mutual trust and productivity. I’m oversimplifying and prettying up the counterexample a bit, but it really is something of a virtuous cycle with team members happily playing to their own strengths and areas of interest.

But what happens in particularly salty Dead Sea shops where authority comes neither from merit nor from expertise, but rather from tenure with the company? What happens when the roost is ruled by Expert Beginners? Well, for one thing, the lords and ladies here tend to guard their fiefdoms more jealously since expertise on the part of others is more threat than benefit. Perhaps more importantly and broadly, however, knowledge and expertise are devalued in favor of personal politics and influence with debates being won on the basis of “who are you and how loud/charming/angry/glib/etc. are you” rather than on the merit of ideas. The currency of Dead Sea departments is everything but ideas–in benevolent ones, it may be “how long have you been here” or “how nice a person are you,” and, in “high pressure culture” ones, it might simply be psychopathy or other cutthroat forms of “might makes right.” And with evaluation of ideas out the window, every council member is freed to hold forth as an expert on every topic, regardless of how much or little he knows about that topic. Nobody is going to dispute anything he says–non-members are cowed into submission and fellow members recognize the importance of putting on a unified public front since they want to be be able to do the same without being questioned.

If you put this political yeast in the oven and let it rise, some of the results are fairly predictable: idea stagnation, increasingly bone-headed solutions to problems, difficulty keeping talent, etc. But an interesting consequence isn’t necessarily intuitive–namely that you’ll wind up with a kind of cabal of long-tenured people that collectively make all decisions, however big or small. I call this the “Council of Elders,” and it’s like one of those Magic Eye paintings that’s hidden but couldn’t be more obvious once you see it.

The Council of Elders is sort of like the Supreme Court of the department, and it’s actually surprisingly democratic as opposed to the more expected ladder system which ranks people by years and months with the company (or if you’re a fan of The Simpsons, the system in the Stonecutters episode where all members of the club are assigned a numeric rank in the order of joining, which determines their authority). The reason that it’s democratic is that actually assigning rank based on years/months of tenure would unceremoniously poke a hole in any illusion of meritocracy. So the council generally makes entrance to the club a matter of tenure, but status within the club a shifting matter of alliances and status games once past the velvet rope.

The Council is fundamentally incapable of delegation or prioritization of decision making. Since entrance is simply a matter of “paying your dues” (i.e. waiting your turn) and largely earned by Expert Beginners, it’s really impossible to divide up decision making based on expertise. The members tend to be very good at (or at least used to) company politics and procedures but not much else. They mostly have the same ‘skill’ set. The lack of prioritization comes from the main currency in the group being status. If a decision, however small, is made by someone not on the Council, it threatens to undermine the very Council itself, so a policy of prevention is adopted and any attempts at circumvention are met with swift and terrible justice (in the form of whatever corporate demerits are in place).

Recognizing Your Elders

What does this council look like in the wild, and how do you know if one is pulling your strings? Here’s a set of symptoms that you’re being governed by a Council of Elders:

• In any meeting convened to make a decision, the same group of people with minor variants is always present.
• Members of the software group are frequently given vague instructions to “talk to so and so before you do anything because his cousin’s dog’s trainer once did something like that or something.”
• There is a roster, emailed out or tacked up, of people who can “clear” you to do something (e.g. they give code reviews, approve time spent, etc.)
• Sparks fly when something wasn’t “cleared” with someone, even when a few others “approved” it and it has apparently nothing to do with him. “Someone should have told me.” More than one or two people like this and you have a Council.
• People in the group often feel caught between a rock and a hard place when involved in the political posturing of the Council (e.g. Yoda tells the intern Padiwan to implement the new hours tracking app using SQL Server, but then Mace Windu screams at him that this is a MySQL shop–the Council settles this matter later with no apologies)
• There is a junta of team members that seem to do nothing but shuffle from meeting to meeting all day like mid-level managers.
• There are regular meetings viewed by newbies and junior developers as rites of passage.

These are just easy ways to detect Councils in their natural habitat: “where there’s smoke, there’s fire” situations. But lest you think that I’m trying to paint every software department with senior developers that go to meetings with this brush, here is a list of specific criteria — “minimum Council Requirements,” if you will:

• Lack of clearly defined roles on the group/team or else lack of clear definition for the assigned roles (e.g. there is no ‘architect’ or if there is, no one really knows what that means).
• A “dues paying” culture and promotions, power and influence largely determined by length of tenure.
• Lack of objective evaluation criteria of performance, skill and decision-making acumen (i.e. answers to all questions are considered subjective and matters of opinion).
• Proposed changes from above are met with claims of technical infeasibility and proposed changes from juniors/newbies or other groups are met with vague refusals or simply aren’t acknowledged at all.
• Actions of any significance are guarded with gatekeeper policies (official code reviews, document approval, sign-off on phases, etc).
• Line manager tacitly approves of, endorses or is the product of the Council.
• A moderate to large degree of institutional bureaucracy is in place.
• New technologies, techniques, approaches, etc are met systematically with culturally entrenched derision and skepticism.

Not a Harmless Curiosity

At this point, you might think to yourself, “So what? It might not be ideal, but rewarding people with a little gratifying power in exchange for company loyalty is common and mostly harmless.” Or perhaps you’re thinking that I’m overly cynical and that the Council generally has good advice to dispense–wisdom won in the trenches. Of course all situations are different, but I would argue that a Council of Elders has a relatively noticeable and negative effect on morale, productivity, and general functioning of a group.

• The Council is a legitimate bottleneck in its micromanagement of other team members, severely hampering their productivity even with the best assumption of its judiciousness
• A group of people that spends all its time debating one another over how to rule on every matter doesn’t actually get much work done. The SCOTUS, for example, hasn’t represented a lot of clients lately, but that’s because their job is to be a ruling council–yours probably has titles like, “Senior Software Engineer.”
• People lacking expertise but put in positions of authority tend to overcompensate by forming strong opinions and sticking to them stubbornly. A room full of people meeting this criterion is going to resemble the US House of Representatives with its gridlock more than a software team.
• Major problems are likely to languish without solutions because the committee doesn’t do much prioritizing and is likely to be sidetracked for a few days by the contentious issue of what color to make the X for closing the window.
• Important decisions are made based on interpersonal dynamics among the council rather than merit. Jones might have the best idea, but Jones shorted the check at lunch that day, so the other Council Members freeze him out.
• Councils make it hard to give meaningful roles or titles to anyone and thus give rise to preposterous situations in which departments have eight ‘architects’ and five developers, or where a project manager decides what database to use while a DBA writes requirements. If everyone on the Council has authority and is an expert at everything, project roles are meaningless anyway.
• Even under the best circumstances, democratic voting on software design trends toward mediocrity: see Design by Committee Anti-Pattern
• People toiling away under the rule of a Council, if they don’t leave, will tend to wind up checked out and indifferent. Either that, or they’ll be assimilated into the Council culture.
• The Council is a natural preservation agent of the Dead Sea problem. Being micromanaged by a team of people whose best qualification is being around for a while and negotiating power will separate the wheat from the chaff, but not in the good way.
• The only thing that unites the Council is outside threats from above or below. If managers or newer members want change, the Council will lock together like a Praetorian Legion to preserve the status quo, however dysfunctional that may be.
• Suggestions for improvement are construed with near universality as threats.

If you’re in a position to do so, I’d suggest breaking up the cartel. Figure out what people are good at and give them meaningful roles. Keep responsibilities divided, and not only will autonomy become more egalitarian, but also people tenured and new alike will develop complementary skills and areas of expertise. With people working together rather than forming shifting alliances, you’ll find prioritizing tasks, dividing up work, and getting things done becomes easier. The Council’s reptile brain will take over and it will fight you every step of the way when you try to disband it, but once you do, it will be good for everyone.