Designs Don’t Emerge
I read a blog post recently from Gene Hughson that made me feel a little like ranting. It wasn’t anything he said — I really like his post. It reminded me of some discussion that had followed in my post about trying too hard to please with your code. Gene had taken a nuanced stand against the canned wisdom of “YAGNI.” I vowed in the comments to make a post about YAGNI as an aphorism, and that’s still in the works, but here is something tangentially related. Now, as then, I agree with Gene that you ignore situational nuance at your peril.
But let’s talk some seriously divisive politics and philosophy first. I’m talking about the idea of creationism/intelligent design versus evolutionary theory and natural selection. The former conceives of the life in our world as the deliberate work of an intelligent being. The latter conceives of it as an ongoing process of change governed by chance and coincidence. In the context of this debate, there is either some intelligent force guiding things or there isn’t, and the debate is often framed as one of omnipotent, centralized planning versus incremental, steady improvement via dumb process and chance. The reason I bring this up isn’t to weigh in on this or turn the blog into a political soapbox. Rather, I want to point out a dichotomy that’s ingrained in our collective conversation in the USA and perhaps beyond that (though I think that the creationist side of the debate is almost exclusively an American one). There is either some kind of central master planner, or there is simply the vagaries of chance.
I think this idea works its way into a lot of discussions that talk about “emergent design” and “big up front design,” which in the same way puts forth a pretty serious false dichotomy. This is most likely due, in no small part, to the key words “design,” “emergent” and especially “evolution” — words that frame the coding discussion. It turns into a blueprint for silly strawman arguments: “Big design” proponents scoff and say things like, “oh yeah, your architecture will just figure itself out magically” while misguided practitioners of agile methodologies (perhaps “no design” proponents) accuse their opponents of living in a coding universe lacking free will — one in which every decision, however small, must be pre-made.
But I think the word “emergent,” rather than “evolution” or “design,” is the most insidious in terms of skewing the discussion. It’s insidious because detractors are going to think that agile shops view design as something that just kind of winks into existence like some kind of friendly guardian angel, and that’s the wrong idea about agile development. But it’s also insidious because of how its practitioners view it: “Don’t worry, a good design will emerge from this work-in-progress at some point because we’re SOLID and DRY and we practice YAGNI.”
Now, I’m not going for a “both extremes are wrong and the middle is the way to go” kind of argument (absent any other reasoning, this is called middle ground fallacy). The word “emergent” itself is all wrong. Good design doesn’t ’emerge’ like a welcome ray of sunshine on a cloudy day. It comes coughing, sputtering, screaming and grunting from the mud, like a drowning man being pulled from quicksand, and the effort of dragging it laboriously out leaves you exhausted.
The big-design-up-front (BDUF) types are wrong because of the essential fallacy that all contingencies can be accounted for. It works out alright for God in the evolution-creation debate context because of the whole omniscient thing. But, unfortunately, it turns out that omniscience and divinity are not core competencies for most software architects. The no-design-up-front (NDUF) people get it wrong because they fail to understand how messy and laborious an activity design really is. In a way, they both get it wrong for the same basic reason. To continue with the Judeo-Christian theme of this post, both of these types fail to understand that software projects are born with original sin.
They don’t start out beautifully and fall from grace, as the BDUF folks would have you believe, and they don’t start out beautifully and just continue that way, emerging as needed, as the NDUF folks would have you believe. They start out badly (after all, “non-functional” and “non-existent” aren’t words which describe great software) and have to be wrangled to acceptability through careful, intelligent and practiced maintenance. Good design is hard. But continuously knowing the next, feasible, incremental step toward a better design at absolutely any point in a piece of software’s life — that’s really hard. That takes deliberate practice, debate, foresight, adaptability, diligence, and a lot of reading and research. It doesn’t just kinda ’emerge.’
If you’re waiting on me to come to a conclusion where I give you a score from one through ten on the NDUF to BDUF scale (and it’s obviously five, right?), you’re going to be disappointed with this post. How much design should you do up front? Dude, I have no idea. Are you building a lunar rover? Probably a lot, then, because the Sea of Tranquility is a pretty unresponsive product owner. Are you cobbling together a minimum viable product and your hardware and business requirements may pivot at any moment? Well, probably not much. I can’t settle your design decisions and timing for you with acronyms or aphorisms. But what I can tell you is that to be a successful architect, you need to have a powerful grasp on how to take any design and make it slightly better this week, slightly better than that next week, and so on, ad infinitum. You have to do all of that while not catastrophically breaking things, keeping developers productive, and keeping stakeholders happy. And you don’t do that “up-front” or “ex post facto” — you do it always.
Bravo, especially the ending. I think going forward what I’ve been referring to as Evolutionary Design (to distinguish it from both emergent design’s NDUF and the fool’s errand of BDUF) will now be called Continuous Evolutionary Design. Excellent post!
I like the addition of continuous, definitely. It seems to convey direction and effort.
Excellent post Erik.
From the perspective of evolutionary design, keep in mind the percentage of things that are successfully living vs. the number of things that ever lived.
I’m pretty sure that purely emergent design has about the same success rate.
lol… yeah, that’s an excellent point. There are an awful lot of mutations that never made it for each one that’s beneficial, and I definitely wouldn’t want those odds when coding.
Well said, and well explained. I favour “just enough” design at every stage: just enough to support and enable the hard work of real-world design that you describe so well. Thanks.