Stories about Software


Musing on Agile’s Built In Caveat Emptor

First things first.  I’d like to thank the folks who submitted “Ask Erik” questions (I’m thinking I might need to come up with a title that’s not as lame — iterate all the things!)  I’m pleased with the results so far, and early returns on my hypothesis look good.  Interestingly, I received more than one question related to my take on capital-A Agile as a movement.  I plan on answering these questions directly, but I have opined on this subject a bit in the past:

Reading the questions I received and letting them kick around in my head a bit as I was jogging earlier today, I started to think obliquely about the topic and how it’s approached and regarded in the industry.  What’s going to follow is fairly raw riffing on this topic, and please forgive me if I seem a little loopy.  It’s about 1:30 AM, I just got back from a concert, and, given that I’ll be off the grid for a couple of weeks starting September 5th, I didn’t want to let a Friday lapse sans post.

1 Agilius

So, story time.  What would you think if I laid the following narrative on you?

17 well known, well respected elders from different villages came to realize that the prevailing methods for tending and cultivating crops were not sufficient to feed the growing populations of their villages.  These elders had some ideas, both mystical and practical, for how to solve that problem.  They’d gained much knowledge on their own and needed to spread the word.

They decided to convene a summit, but the odds were stacked against them.  Each village had its own ways of doing things and cultural preferences, and the elders were each accustomed to those around deferring to them.  Initially, they could not even agree on a location for the summit!  How were they ever going to agree on providing food for the known world?

Against all odds, however, they made progress.  They retreated into the mountains and toiled for 2 days and 2 nights, expecting little to come out of the gathering, but when the sun rose on the 3rd morning, not only had they made strides — they’d all agreed, as if by Divine Providence.  What emerged from this gathering were stone tablets containing the 4 Core Values and the 12 Principles of food cultivation.

Over the years, the Word spread far and wide.  What started as 17 quickly became dozens, and eventually hundreds, thousands, and hundreds of thousands.  Great centers of learning and cultural exchange emerged, devotees made annual pilgrimages to important sites, and the new methods for food cultivation spread far and wide.  And, lo, it was good.  There was much rejoicing.

In case this doesn’t ring a bell with you off the cuff, it’s basically the history of the Agile Manifesto.  And, before you bristle, read that history that I linked.  It says epistle in there to describe one of the communications between signatories, for goodness sake — if that doesn’t invite religious comparisons, I don’t know what does.  Try googling that word and finding a reference in the top 10 that doesn’t prominently mention the Bible.


This post isn’t actually about comparing capital-A Agile to religion.  This is well trod ground, and a well placed google search will let you tread it to your heart’s content.  But if you temporarily accept the premise as axiomatic, there’s an interesting plank to the agile canon that’s generally missing from religion, at least in my experience with it.  What I’m talking about is self-reference in a way that isn’t begging the question.   Read More


Agile Methodologies or Agile Software?

Over the last couple of months, I’ve been doing mostly management-y things, so I haven’t had a lot of trade craft driven motivations to pick Pluralsight videos to watch while jogging. In other words, I’m not coming up to speed on any language, framework, or methodology, so I’m engaging in undirected learning and observation. (I’m also shamelessly scouring other authors’ courses for ways to make my own courses better). That led me to watch this course about Agile fundamentals.

As I was watching and jogging, I started thinking about Agile Manifesto and the 14 years that have passed since its conception. “Agile” is undeniably here to stay and probably approaching “industry standard.” It’s become so commonplace, in fact, that it is an industry unto itself, containing training courses, conferences, seminars, certifications — the works. And this cottage industry around “getting Agile” has sparked a good bit of consternation and, frequently, derision. We as an industry, critics might say, got so good at planning poker and daily standups that we forgot about the relatively minor detail of making software. Martin Fowler coined a term, “flaccid Scrum” to describe this phenomenon wherein a team follows all of the mechanics of some Agile methodology (presumably Scrum) to the letter and still produces crap.

It’s no mystery how something like this could happen. You’ve probably seen it. The most common culprit is some “Waterfall” shop that decides it wants to “get Agile.” So the solution is to go out and get the coaches, the certifiers, the process experts, and the whole crew to teach everyone how to do all of the ceremonies. A few weeks or months, some hands on training, some seminars, and now the place is Agile. But, what actually happens is that they just do the same basic thing they’d been doing, more or less, but with an artificially reduced cycle time. Instead of shipping software every other year with a painful integration period, they now ship software quarterly, with the same painful integration period. They’re just doing waterfall on a scale of an eighth of the previous size. But with daily standups and retrospectives.

There may be somewhat more nuance to it in places, but it’s a common theme, this focus on the process instead of the “practices.” In fact, it’s so common that I believe the Software Craftsmanship Manifesto and subsequent movement was mainly a rallying cry to say, “hey, remember that stuff in Agile about TDD and pair programming and whatnot…? Instead of figuring out how to dominate Scrum until you’re its master, let’s do that stuff.” So, the Agile movement is born and essentially says, “let’s adopt short feedback cycles and good development practices, and here are some frameworks for that” and what eventually results is the next generation of software process fetishism (following on the heels of the “Rational Unified Process” and “CMM”).

That all played through my head pretty quickly, and what I really started to contemplate was “why?” Why did this happen? It’s not as if the original signatories of the manifesto were focused on process at the exclusion of practices by a long shot. So how did we get to the point where the practices became a second class citizen? And then, the beginnings of a hypothesis occurred to me, and so exists this post.

The Agile Manifesto starts off with “We are uncovering better
of developing software…” (emphasis mine). The frameworks for this type of development were and are referred to as “Agile Methodologies.” Subtly but very clearly, the thing we’re talking about here — Agile — is a process. Here were a bunch of guys who got together and said, “we’ve dumped a lot of the formalism and had good results and here’s how,” and, perversely, the only key phrase most of the industry heard was “here’s how.” So when the early adopted success became too impressive too ignore, the big boys with their big, IBM-ish processes brought in Agile Process People to say, “here’s a 600 page slide deck on exactly how to replace your formal, buttoned-up waterfall process with this new, somehow-eerily-similar, buttoned-up Agile process.” After all, companies that have historically tended to favor waterfall approaches tend to view software development as a mashup of building construction and assembly line pipelining, so their failure could only, possibly be caused by a poorly engineered process. They needed the software equivalent of an industrial engineer (a process coach) to come in and show them where to place the various machines and mindless drones in their employ responsible for the software. Clearly, the problem was doing design documents instead of writing story cards and putting Fibonacci numbers on them.

The Software Craftsmanship movement, I believe, stands as evidence to support what I’m saying here. It removes the emphasis altogether from process and places it, in very opinionated fashion, on the characteristics of the actual software: “not only working software, but also well-crafted software.” (emphasis theirs) I can’t speak exactly to what drove the creation of this document, but I suspect it was at least partially driven by the obsession with process instead of with actually writing software.


All of this leads me to wonder about something very idly. What if the Agile Manifesto, instead of talking about “uncovering better ways,” had spoken to the idea of “let’s create agile software?” In other words, forget about the process of doing this altogether, and let’s simply focus on the properties of the software… namely, that it’s agile. What if it had established a definition that agile software is software that should be able to be deployed within, say, a day? It’s software that anyone on the team can change without fear. It’s software that’s so readable that new team members can understand it almost immediately. And so on.

I think there’s a deep appeal to this concept. After all, one of the most annoying things to me and probably to a lot of you is having someone tell me how to solve a problem instead of what their problem is, when asking for help. And, really, software development methodologies/processes are perhaps the ultimate example of this. Do a requirements phase first, then a design phase, then an implementation phase, etc. Or, these days, write what the users want on story cards, have a grooming session with the product owner, convene the team for planning poker, etc. In both cases, what the person giving the direction is really saying is, “hey, I want you to produce software that caters to my needs,” but instead of saying that and specifying those needs, they’re telling you exactly how to operate. What if they just said, “it should be possible to issue changes to the software with the press of a button, it needs to be easy for new team members to come on board, I need to be able to have new features implemented without major architectural upheaval, etc?” In other words, what if they said, “I need agile software, and you figure out how to do that?”

I can’t possibly criticize the message that came out of that meeting of the minds and gave birth to the Agile Manifesto. These were people that bucked entrenched industry trends, gained traction, and pulled it off with incredible success. They changed how we conceive of software development and clearly for the better. And it’s entirely possible that any different phrasing would have made the message either too radical or too banal for widespread adoption. But I can’t help but wonder… what if the call had been for agile software rather than agile methods.


Agile: False Hope and Real Promise

I went shopping for jeans last weekend, which was, as always, somewhat of an aggravating experience. It’s not just jeans, but, really, any wearable thing: shoes, work shirts, belts, even socks. The source of my irritation is the fact that I can’t simply replace items I have that are worn out because the thing you bought last time is always decommissioned. Strangely, major things like cars have better staying power than clothes. There’s a very simple use case missing: As a shoe wearer, given that I love a pair of shoes and thus wear them out in 2 years, I want to go back to the same store and buy the exact same pair of friggin’ shoes so that my life is not interrupted. And the software of life fails to perform.

The reason for this is the nebulous concept of fashion. Naturally, wearable manufacturers and retailers don’t want to be ‘stale’ so they churn things at a rate that prevents me from being pragmatic with my shopping and turns my trips into maddening treasure hunts instead of simple errands. So I contemplate fashion a bit with a natural bias toward annoyance, and this has informed a pretty cynical take on the subject. I have an ongoing hypothesis that the thing that drives fashion isn’t some guy sitting in Paris, saying “I’m going to make those idiots wear really tight jeans this year and then laugh at them.” Instead, its the shifting tectonic plates of poor people trying to look like rich people and rich people trying not to look like poor people.

By way of narrative, guy in Paris releases skinny jeans and charges $700 for them. They’re new and expensive, so rich people buy them so that they can be seen wearing new and expensive things — signaling their wealth and cosmopolitan tastes. Bulk clothing manufacturers start knocking off Paris designer and selling it to the masses for $39 a pair and they sell out like crazy as everyone wants to ape the A list celebrities and influencers wearing these things. Rich people get annoyed that they’re no longer distinguishable on sight and this demand drives Paris guy to dream up some new, weird thing that they can buy to look rich. Rinse, repeat.


This narrative isn’t limited to the arena of rich and poor people with clothes, however. It also extends to things like industry thought leadership and buzzwords. Crowd of cool kids starts doing something and succeeding/making money. They give it a name and everyone rushes to mimic their success, adopting this thing. Imitators adopt it en masse, bring mediocrity and misunderstanding to it, and the cool kids lament that everyone is getting it wrong and that it’s time for another industry shake-up. Queue the “Is {Insert Tech} Dead” link bait and search for the next great thing.

“Agile” is probably the poster child for this sort of thing in the development world. Back around the turn of the millennium, various industry thinkers were experimenting with ways to move software development away from software development practices that mimicked construction and physical engineering since the results produced by this traditional, “waterfall” approach were spotty at best. A group of representatives of some of these experiments came together to find common ground and the result was called, “The Agile Manifesto.” In the interceding 13 years, the industry has come to accept that the Agile approach is “the right thing” while agreeing that, by and large, no one does it right.

Off the top, I’ve heard the following expressed recently (paraphrased from my recollection):

  • Agile is dead.
  • Scrum is a scam that makes grunts stand while they deliver their status reports.
  • If XP doesn’t work it’s because you’re doing it wrong.
  • Agile is stupid because you really can’t work sanely without comprehensive documentation and specs.
  • Agile is a marketing gimmick to create phony demand for meaningless certifications.

And yet, you’d be hard-pressed to find any non-agile shop that didn’t look down shamefacedly and mumble “yeah, we do waterfall.”  So, the reasonable conclusions from this are:

  • Waterfall (or pseudo-waterfall things like RUP) are really out of style.
  • Agile is mainstream, but the real fashionistas are so over it because everyone’s messing it up and ruining it.
  • Cool kids are looking for what’s next: a post-Agile world.

Agile, Distilled

I’m not certified in Scrum or XP or anything else. I have no project management credentials and no letters after my name. I certainly have experience in these methodologies and understand the ceremonies and the roles, but I’d hardly count myself an encyclopedia of how everything should operate. It’s always bemused me that arguments emerge over the finer points of the exact approaches to these methodologies and that people can actually be certified in exact adherence when the focus of the Agile Manifesto seems to me to be best summarized by the idea of bringing pragmatism and empirical experimentation to software development. But, opportunities for snark notwithstanding, it’s easy to see why people become disillusioned. Anytime there is some sort of process and any failures are blamed on a lack of true, absolute adherence, you have a recipe for quackery.

But at it’s core, I think Agile methodologies can be distilled to a single compound principle: tighten the feedback loop and improve the feedback. Think about the different things that are done in various flavors of agile: automated testing/TDD, short sprints, pair programming, retrospectives, big/visible charts, etc. All of it is about delivering feedback faster, delivering feedback more obviously, and acting quickly on the feedback. This is really, really non-controversial. The sooner you get feedback on what you’re working on, the sooner you can improve. So, where does everything go off the rails?

Is it that things don’t go well because people aren’t following the processes right? Is it that the processes themselves are broken? Is it that the processes are good, but a bunch of hucksters are peddling fools’ gold? Is it that people are just too entrenched in their ways to be compatible with these principles? Personally, I don’t think it’s any of this. I think that the fundamental problem is one of expectations (though some of the hucksters do nothing to mitigate this). Specifically, I think the main source of woe is the assumption that “going Agile” will cure what ails you because the only thing standing between the broken, over-budget pile of crap you’re about to ship and software nirvana is having a different set of meetings and adopting a different desk arrangement. The Agile promise isn’t realized by tweaking process; it’s realized by getting a scheme in place for receiving meaningful feedback and using that feedback, with a lot of practice, to get better at making software.

10,000 Hours

Wait, what? Practice? That doesn’t have anything to do with Kanban boards, swim lanes, stand-ups, and all of the other Agile stuff that gets so much focus. In the words of the esteemed Alan Iverson, “What are we talking about? Practice? We’re talking about practice, man.” The wide world is talking about the magic dust you can sprinkle on a team to make it crank out better software, faster, and I’m in here talking about practice.

During my semi-weekly drive to Detroit, I’ve been listening to audio books, including, most recently, Malcom Gladwell’s “Outliers.” Without offering up too much of a spoiler, one of the things that he talks about is that people like Bill Gates don’t reach the incredible levels of success that they do by being dominant in a vacuum, but rather they combine incredible fortune and opportunity with cleverness, ingenuity and a whole lot of really hard work. Bill Gates was born at a time where him being a teenager coincided with the very first machines that allowed for desktop based computing, rather than punch cards. He had the incredible good fortune of gaining access to such a computer in the late 60’s, as a teenager, when almost no adults or professionals on Earth had such access. Addicted to the fun of programming, he worked nights and weekends during his teenage years, programming, at a time when few others in the world had that opportunity. By the time he was an adult, ready to start Microsoft, he was one of the few people on Earth with 10,000 hours of experience programming this way.

The hypothesis for which the book is best known is that 10,000 hours of deliberate practice is the general average amount of time put in before a person acquires mastery of a field. As I understand it, there has been some corroboration of this hypothesis as well as some counter-evidence, but the broad point remains that achieving mastery requires a whole lot of deliberate work and concerted, sustained effort to improve. You know how you get really good at delivering software? Do it for years and years, constantly learning from your mistakes. You know how you don’t get really good at delivering software? By having slightly different meetings, shipping crap more frequently, and hoping for different results.

The way I see it, big A Agile, at its core, is not about fixing you or your software. It’s about making your practice more meaningful and efficient. To return to the basketball metaphor, Agile isn’t a weekend retreat that turns you into Michael Jordan. It’s a weekend retreat that shows you how to do a bunch of layup and dribbling drills that, after 10,000 hours of practice, will make you pretty good, as long as you keep challenging yourself. The great shame of Agile or any other kind of movement is that of false hope in easy fixes. Done well, it will expose your pain points more quickly, provide you with tools for removing obstacles, illuminate a path to improvement, and show you how to self-evaluate regularly. But it won’t make you awesome. Only practice and continuous learning will do that.


A Metaphor to Help You Suck at Writing Software

“No plan survives contact with the enemy” –Helmuth von Moltke the Elder

Bureaucracy 101

Let’s set the scene for a moment. You’re a workaday developer in a workman kind of shop. A “waterfall” shop. (For back story on why I put quotes around waterfall, see this post). There is a great show of force when it comes to building software. Grand plans are constructed. Requirements gathering happens in a sliding sort of way where there is one document for vague requirements, another document for more specific requirements, a third document for even more specific requirements than that, and repeat for a few more documents. Then, there is the spec, the functional spec, the design spec, and the design document. In fact, there are probably several design documents.

There aren’t just the typical “waterfall” phases of requirements->design->code->test->toss over the wall, but sub-phases and, when the organism grows large enough, sub-sub-phases. There are project managers and business managers and many other kinds of managers. There are things called change requests and those have their own phases and documents. Requirements gathering is different from requirements elaboration. Design sub-phases include high-level, mid-level and low-level. If you carefully follow the process, most likely published somewhere as a mural-sized state machine or possibly a Gantt chart unsurpassed in its perfect hierarchical beauty, you will achieve the BUFD nirvana of having the actual writing of the code require absolutely no brain power. Everything will be so perfectly planned that a trained ape could write your software. That trained ape is you, workaday developer. Brilliant business stakeholder minds are hard at work perfecting the process of planning software in such fine grained detail that you need not trouble yourself with much thinking or problem solving.

Dude, wait a minute. Wat?!? That doesn’t sound desirable at all! You wake up in the middle of the night one night, sit bolt upright and are suddenly fundamentally unsure that this is really the best approach to building a thing with software. Concerned, you approach some kind of senior business project program manager and ask him about the meaning of developer life in your organization. He nods knowingly, understandingly and puts one arm on your shoulders, extending the other out in broad, professorial arc to help you share his vision. “You see my friend,” he says, “writing software is like building a skyscraper…” And the ‘wisdom’ starts to flow. Well, something starts to flow, at any rate.

Let’s Build a Software Skyscraper

Like a skyscraper, you can’t just start building software without planning and a lot of upfront legwork. A skyscraper can’t simply be assembled by building floors, rooms, walls, etc independently and then slapping them altogether, perhaps interchangeably. Everything is necessarily interdependent and tightly coupled. Just like your software. In the skyscraper, you simply can’t build the 20th floor before the 19th floor is built and you certainly can’t interchange those ‘parts’ much like in your software you can’t have a GUI without a database and you can’t just go swapping persistence models once you have a GUI. In both cases every decision at every point ripples throughout the project and necessarily affects every future decision. Rooms and floors are set in stone in both location and order of construction just as your classes and modules in a software project have to be built in a certain order and can never be swapped out from then on.Jenga

But the similarities don’t end with the fact that both endeavors involve an inseparable web of complete interdependence. It extends to holistic approaches and cost as well. Since software, like a skyscraper, is so lumbering in nature and so permanent once built, the concept of prototyping it is prima facie absurd. Furthermore, in software and skyscrapers, you can’t have a stripped-down but fully functional version to start with — it’s all or nothing, baby. Because of this it’s important to make all decisions up-front and immediately even when you might later have more information that would lead to a better-informed decision. There’s no deferral of decisions that can be made — you need to lock your architecture up right from the get-go and live with the consequences forever, whatever and however horrible they might turn out to be.

And once your software is constructed, your customers better be happy with it because boy-oh-boy is it expensive, cumbersome and painful to change anything about it. Like replacing the fortieth floor on a skyscraper, refactoring your software requires months of business stoppage and a Herculean effort to get the new stuff in place. It soars over the budget set forth and slams through and past the target date, showering passerby with falling debris all the while.

To put it succinctly in list form:

  1. There is only one sequence in which to build software and very little opportunity for deviation and working in parallel.
  2. Software is not supposed to be modular or swappable — a place for everything and everything in its place
  3. The concept of prototyping is nonsensical — you get one shot and one shot only.
  4. It is impossible to defer important decisions until more information is available. Pick things like database or markup language early and live with them forever.
  5. Changing anything after construction is exorbitantly expensive and quite possibly dangerous

Or, to condense even further, this metaphor helps you build software that is brittle and utterly cross-coupled beyond repair. This metaphor is the perfect guide for anyone who wants to write crappy software.

Let’s Build an Agile Building

Once you take the building construction metaphor to its logical conclusion, it seems fairly silly (as a lot of metaphors will if you lean too heavily on them in their weak spots). What’s the source of the disconnect here? To clarify a bit, let’s work backward into the building metaphor starting with good software instead of using it to build bad software.

AgileBuildingA year or so ago, I went to a talk given by “Uncle” Bob Martin on software professionalism. If I could find a link to the text of what he said, I would offer it (and please comment if you have one) but lacking that, I’ll paraphrase. Bob invited the audience to consider a proposition where they were contracting to have a house built and maintained with a particular contractor. The way this worked was you would give the contractor $100 and he would build you anything you wanted in a day. So, you could say “I want a two bedroom ranch house with a deck and a hot-tub and 1.5 bathrooms,” plop down your $100 and come back tomorrow to find the house built to your specification. If it turned out that you didn’t like something about it or your needs changed, same deal applied. Want another wing? Want to turn the half bath into a full bath? Want a patio instead of a deck? Make your checklist, call the contractor, give him $100 and the next day your wish would be your house.

From there, Bob invited audience members to weigh two different approaches to house-planning: try-it-and-see versus waterfall’s “big design up front.” In this world, would you hire expert architects to form plans and carpenters to flesh them out? Would you spend weeks or months in a “planning phase”? Or would you plop down $100 and say, “well, screw it — I’ll just try it and change it if I don’t like it?” This was a rather dramatic moment in the talk as the listener realized just before Bob brought it home that given a choice between agile, “try it and see” and waterfall “design everything up front” nobody sane would choose the latter. The “waterfall” approach to houses (and skyscrapers) is used because a better approach isn’t possible and not because it’s a good approach when there are alternatives.

Wither the Software-Construction Canard?

Given the push toward Agile software development in recent years and the questionable parallels of the metaphor in the first place, why does it persist? There is no shortage of people who think this metaphor is absurd, or at least misguided:

  1. Jason Haley, “It’s not like Building a House”
  2. Terence Parr, “Why writing software is not like engineereing”
  3. James Shore, “That Damned Construction Analogy”
  4. A whole series of people on stackoverlow
  5. Nathaniel T. Schutta, Why Software Development IS Like Building a House (Don’t let the title fool you – give this one a detailed read)
  6. Thomas Guest, “Why Software Development isn’t Like Construction”

If you google things like “software construction analogy” you will find literally dozens of posts like these.

So why the persistence? Well, if you read the last article, by Thomas Guest, you’ll notice a reference to Steve McConnell’s iconic book “Code Complete.” This book has an early chapter that explores a variety of metaphors for software development and offers this one up. In my first daedtech post I endorsed the metaphor but thought we could do better. I stand by that endorsement not because it’s a good metaphor for how software should be developed but because it’s a good metaphor for how it is developed. As in our hypothetical shop from the first section of the post, many places do use this approach to write (often bad) software. But the presence of the metaphor in McConnell’s book and for years and years before that highlights one of the main reasons for persistence: interia. It’s been around a long time.

But I think there’s another, more subtle reason it sticks around. Hard as it was to find pro posts about the software-construction pairing, the ones I did find share an interesting trait. Take a look at this post, for instance. As “PikeWake” gets down to explaining the metaphor, the first thing that he does is talk about project managers and architects (well, the first thing is the software itself, but right after that come the movers and shakers). Somewhere below that the low-skill grunts who actually write the software get a nod as well. Think about that for a moment. In this analogy, the most important people to the software process are the ones with corner offices, direct reports and spreadsheets, and the people who actually write the software are fungible drones paid to perform repetitive action, rather than work. Is it any wonder that ‘supervisors’ and other vestiges of the pre-Agile, command and control era love this metaphor? It might not make for good software, but it sure makes for good justification of roles. It’s comfortable in a world where companies like github are canning the traditional, hierarchical model, valuing the producers over the supervisors, and succeeding.

Perhaps that’s a bit cynical, but I definitely think there’s more than a little truth there. If you stripped out all of the word documents, Gantt charts, status meetings and other typical corporate overhead and embraced a world where developers could self-organize, prioritize and adapt, what would people with a lot of tenure but not a lot of desire or skill at programming do? If there were no actual need for supervision, what would happen? These can be unsettling, game changing questions, so it’s easier to cast developers as low-skill drones that would be adrift without clever supervisors planning everything for them than to dispense with the illusion and realize that developers are highly skilled, generally very intelligent knowledge workers quite capable of optimizing processes in which they participate.

In the end, it’s simple. If you want comfort food for the mid-level management set and mediocrity, then invite someone in to sweep his arm professorially and spin a feel-good tale about how building software is like building houses and managing people is like a father nurturing his children. If you want to be effective, leave the construction metaphor in the 1980s where it belongs.