In Devs We Trust

The idea of getting certified in “Scrum” or “Agile” is philosophically unsettling to me.

It’s not so much that I think learning how experienced people execute these processes is a bad thing (I’ve even signed off on such a course for an employee’s career development aspirations), but rather the overtones of such a certification. Certifications are formalized recognition of program completion and endorsements of those that have done the completing. It’s like saying, “congratulations — you have rigorously followed the formalized, rigid process necessary to be agile!”

Why Certs Unsettle Me

Against the backdrop of the original agilistas rejecting button-up formalism for pragmatism and efficiency, this is sort of like a high school principal commissioning a “Responsible Pot Smoking and 3rd Period Ditching” course as part of the core curriculum.

The reason this is unsettling to me rather than comically ironic is that it supplies surprisingly fertile ground for re-growing all of the mistakes of the waterfall era by masking the true nature of those mistakes. The idea that it’s possible to nail down every last detail of a complex software system prior to writing it is cringe-worthy in its naivete (at least this side of a space shuttle), but it’s not really the core problem.

The core problem is the idea that a software project can’t succeed without planning every detail prior to implementation.

A Lack of Trust

This attitude belies a fundamental lack of trust in the people doing the implementation and, further, a suspicion that these people are incompetent malingerers. As an industry, we need to set aside a few high-wage “architects” to create a specification that will allow the drooling imbeciles, or “programmers,” to perform the mindless, labor-intensive task of writing software.

It evokes classic imagery of pyramid construction. Pharaoh and his great builders have the vision, and they just need whips, the wheel, and the backs of a few tens of thousands of fungible, expendable drones to make it a reality.

The fundamental problem with the waterfall approach isn’t that we want to plan out the pyramid but that we want to treat software developers like stone-haulers, as if implementing an efficient self-balancing tree were akin to pushing rocks.

Everything about this “classic” approach to software smacks of this fundamental lack of trust.

• Why plan everything out ahead of time in excruciating detail?
• Why come up with elaborate, painfully detailed processes and mind-numbingly detailed “design documents?”
• Why lay everything out in such a way that there is absolutely no room for creative license or spontaneous thinking?
• Why come up with CMMI level 5?

You do this to prevent people from “getting any ideas” and thinking for themselves — you do it because you don’t trust them (and you probably don’t trust them because you approached hiring them the way you’d approach finding cut-rate car insurance).

The Agile Manifesto, Revisited

If you go read the Agile Manifesto, you’ll see something that’s relatively vague, but underpinned by a consistent theme: “we work best when you trust us, collaborate with us and talk to us.”

It begs producers and consumers to back away from heavy documentation, plan, and process in favor of adaptability, collaboration and thinking on the fly. It implicitly asks that consumers of software trust producers of software and rely on their discretion.

Certs Miss the Point

So fast forward to present time and a world where you can depend on a thriving cottage industry to come in and bless your organization by making you “Agile Certified.” Why would you do this?

Why bring people in and have them tell your developers in specific, painful detail exactly how to conduct meetings and follow the processes that will allow them to be certified agile practitioners? Why spell out exactly how they can go about being agile? You do it because you still don’t trust them.

Scrum and XP are approaches that have merit because they generally consist of individual, specific practices and ideas that may help a team.

Pair programming/peer review, test driven development, making problems visible, etc are potential tools in the tool belt of a solid software developer. Agile practices/processes have value not because they’re the secret sauce that’s going to allow you to deliver version 6 on time where versions 1 through 5 have been lawsuit-inviting nightmares, but rather they have value when and because they force you to trust the software developers to get things done, to try hard, to provide honest answers and, generally, to be professionals.

There is No Secret Sauce

There is no up-up-down-down-left-right-left-right-b-a-b-a-start (I honestly did that without looking it up). There is no achievement to unlock, no box to check, and no certification to buy.

Organizations don’t get off that easily because what they need to do is much harder to quantify as a line item, objective, or KPI. Fortunately, hard as it is to package up and roll into an annual report, it’s really easy to explain.

Organizations have to hire software developers that they trust, clear nonsense and obstacles out of their way, keep them happy, and then trust them to develop software. That’s it.

I’m not saying this is actually easy to execute. Refining a candidate selection process to the point where you bring in competent, hard-working candidates with a high degree of certainty is quite a difficult problem.

Keeping them happy is no simpler. Setting them up for success just keeps increasing the difficulty level. And paying for all of this is really going to make you wince since it’s not possible to short-cut the system by bringing folks in cheap and then, real-quick, getting them certified in “you can trust this person to make good decisions about writing software.” Perhaps hardest of all, though, given our historical culture of micro-management and death-by-process, is actually doing the trusting.

Trusting the Dev Team is the Way Forward

But it has to be done. It’s the only sane way forward.

If you want to get a piece of software written, find some software developers you respect and trust and task them with developing software for you. If they know what they’re doing, they’ll figure out how to validate and verify their work, how to evaluate and improve their process as they go, how to limit distractions and work in progress, how to know when they’ve delivered on what they promised, etc.

They might do this by Scrum or by XP or by something that they invent on the fly, mixing and matching and innovating, but they’ll do it. They don’t need elaborate process or ceremony instruction and they certainly don’t need to be ‘certified’; they need you to trust them and let them work as they see fit.

Imagine a Process Regression

I was recently out to dinner with some friends that are fellow techies, and we were talking about software development methodologies. One of them, who works in a shop that practices pretty true-to-form Scrum, was talking about how management via the product owner was starting to put pressure on them to produce certain, lower point estimates during the planning poker game. He said that it struck the younger team members who’d only worked there as extremely strange and that he personally felt that the powers that be were sort of trying to steer them to a more “waterfall-ish” style. I had a random thought and, rudely, burst out laughing during his anecdote which was, in no way, intended to be funny.

The thing that I had thought of, I explained to my confused friends, was how incredibly weird and counter-intuitive moving to a waterfall development methodology would seem to his teammates who had come into the software development world knowing only Scrum. It’d be like coming home one day and telling your family that you were through buying soap because from now on all you needed was some lye, some elbow grease and the river on the other side of town. Of course, there would (and has been) culture shock going from Waterfall to Agile, but that’s mostly born of a kind of “that’s an impossible pipe dream” mentality. Going the other way has more of a “you want to do what…. and why?” vibe.

Think of this. You’ve never known anything but a disciplined agile approach to software. You have an automated build and deployment setup. Every time you check in code, the build executes, unit tests are run, and the code is pushed to a test environment for exploratory QA and validation of user stories. Checkins are frequent, collaboration is ubiquitous and refactoring and cleaning the code is something you do on a regular basis. Once every other week, you ship the code and it’s no big deal. The (business) day after shipping, you and the team collaboratively plan what you’re going to do for the next two weeks, trusting that someone is prioritizing the work you wind up doing toward a larger technical product/vision. Your users see regular progress and give you regular feedback, creating a tight communication loop and eliminating surprises. And, if the users change their mind about the features they want or don’t like what you’ve done over the last two weeks, you simply spend the next two weeks steering methodically back in the right direction. But don’t worry, that’s all about to change!

Forget about all of that bi-weekly shipping nonsense. We’re going to ship once, as far as we’re concerned. A second shipment is by no means a given because frankly, the first one is rather likely to fail and anger the users. So, let’s not concern ourselves with anything but the first shipment. For the first month or so, we’re going to gather the requirements for the project. We’ll skip through a meadow, plucking them off of the beautiful flora and placing them in our baskets. That’ll be a good month. At the end of that month, we’ll tell our users that they’d better be sure that our endless array of diagrams and thousands of statements about what the software “shall” do matches exactly what they want because once we move onto the next phase, there’s no going back or changing their minds without either expensive change requests or lawyers.

Next up comes the design phase. This is the part where we spend a month figuring out how to write all of the code that we’re going to write over the next several months, but without actually writing that code. While this may sound nuts, the reason that we have to do this is because the labor is going to be divided up in such a way that everyone will kind of go off into their own silo and write code for months prior to an extended “integration phase” where our architect, Dr. Frankenstein, assembles the various individually created pieces into a reasonable attempt at a piece of software and then attempts to breath life into it. In order for that task to be as minimally gruesome as possible, a lot of planning is needed.

The next phase is implementation. Implementation phase is where we start coding and the previous two months of ‘work’ become completely irrelevant. Everyone codes in their silos until integration time. Some attempts are made at early integrations, but those are abandoned when it’s discovered that they are time consuming and the project managers are worried about the fact that they’re already behind schedule. Everyone will then pray that somehow we make up the time that we’ve gotten behind at integration time, but, who are we kidding? That’s not going to happen. The next phase will ideally be the testing phase, but we won’t have time for that. Instead, we’ll start the death march phase, where everyone works 14 hour days, 7 days a week up to the deadline.

Then, my friend, we ship. Tada! Ooh, that wasn’t supposed to happen. Heh, heh, weird. I’m sure the rest of it will be fine if I just — ooh, yikes. What do you mean this screen doesn’t look at all like what you were picturing? Heh, heh, maybe we can reschedule this demo. This is the phase where the thing we chucked over the wall, behind schedule and with a minimum of testing, blows up in our faces. After another month or so of death march following the release, we manage to wrangle the software into a state where, while the source code looks hideous, it’s at least stable and no one is screaming at us anymore.

There, doesn’t that sound better than the one where you’re shipping software every two weeks and not working 90 hour weeks to make up for the fact that you’re not actually very good at predicting the future? But of course, I kid. It was just an interesting exercise for me to think of how the world would seem going from agile to waterfall. Agile methodologies are inherently developer-centric when done correctly. Developers like quick feedback, routine shipments, and process automation because this makes for sustainable, incremental progress. The main complaining I hear from waterfall developers isn’t that agile doesn’t sound good — it’s that it sounds too good to be true. Usually it’s things like “we couldn’t do agile because we’re a shop that does _______” where the blank is some excuse that isn’t actually reasonable unless the company is shipping Mars Rovers. So if you’re already doing agile — you’ve already seen how well it works — going back would seem certifiably crazy. And, you know what? I like that there’s a generation of developers out there that think Waterfall is just a bed time story that the old timers use to scare the newbies into behaving.