Automation and the Art of Software Maintenance
Editorial Note: I originally wrote this post for the SubMain blog. You can check out the original here, at their site. While you’re there, check out CodeIt.Right for automating your code review process.
I have long since cast my lot with the software industry. But, if I were going to make a commercial to convince others to follow suit, I can imagine what it would look like. I’d probably feature cool-looking, clear whiteboards, engaged people, and frenetic design of the future. And a robot or two. Come help us build the technology of tomorrow.
Of course, you might later accuse me of bait and switch. You entered a bootcamp, ready to build the technology of tomorrow. Three years later, you found yourself on safari in a legacy code jungle, trying to wrangle some Sharepoint plugin. Erik, you lied to me.
So, let me inoculate myself against that particular accusation. With a career in software, you will certainly get to work on some cool things. But you will also find yourself doing the decidedly less glamorous task of software maintenance. You may as well prepare yourself for that now.
The Conceptual Difference: Build vs Maintain
From the software developer’s perspective, this distinction might evoke various contrasts. Fun versus boring. Satisfying versus annoying. New problem versus solved problem. My stuff versus that of some guy named Steve that apparently worked here 8 years ago. You get the idea.
But let’s zoom out a bit. For a broader perspective, consider the difference as it pertains to a business.
Build mode (green field) means a push toward new capability. Usually, the business will regard construction of this capability as a project with a calculated return on investment (ROI). To put it more plainly, “we’re going to spend $500,000 building this thing that we expect to make/save us $1.5 million by next year.”
Maintenance mode, on the other hand, presents the business with a cost center. They’ve now made their investment and (at least partially) realized return on it. The maintenance team just hangs around to prevent backslides. For instance, should maintenance problems crop up, you may lose customers or efficiency.
Plan of a Attack: Build vs Maintain
Because the business regards these activities differently, it will attack them differently. And, while I can’t speak to every conceivable situation, my consulting work has shown me wide variety. So I can speak to general trends.
In green field mode, the business tends to regard the work as an investment. So, while management might dislike overruns and unexpected costs, they will tend to tolerate them more. Commonly, you see a “this will pay off later” mentality.
On the maintenance side of things, you tend to see far less forgiveness. Certainly, all parties forgive unexpected problems a lot less easily. They view all of it as a burden.
This difference in attitude translates to the planning as well. Green field projects justifiably command full time people for the duration of the project. Maintenance mode tends to get you familiar with the curious term “half of a person.” By this, I mean you hear things like “we’re done with the Sigma project, but someone needs to keep the lights on. That’ll be half of Alice.” The business grudgingly allocates part time duty to maintenance tasks.
Why? Well maintenance tends to arise out of reactive scenarios.
Reactive Mode and the Value of Automation
Maintenance mode in software will have some planned activities, particularly if it needs scheduled maintenance. But most maintenance programmers find themselves in a reactive, “wait and see” kind of situation. They have little to do on the project in question until an outage happens, someone discovers a bug, or a customer requests a new feature. Then, they spring into action.
Business folks tend to hate this sort of situation. After all, you need to plan for this stuff, but you might have someone sitting around doing nothing. It is from this fundamental conundrum that “half people” and “quarter people” arise. Maintenance programmers usually have other stuff to juggle along with maintaining “Sigma.”
Because of this double duty, the business doubles down on pressure to minimize maintenance. After all, not only does it create cost, but it takes the people away from other, profit-driven things that they could otherwise do.
So how do we, as programmers, and we, as software shops, best deal with this? We make maintenance as turnkey as possible by automating as much as possible. Oh, and you should automate this stuff during green field time, when management is willing to invest. If you tell them it means less maintenance cost, they’ll probably bite.
Automate the Test Suite
First up for automation candidates, think of the codebase’s test suite. Hopefully, you’ve followed my advice and built this during green field mode. But, if not, it’s never too late to start.
Think of how time consuming a job QA has. If manually running the software and conducting experiments constitutes the entirety of your test strategy, you’ll find yourself hosed at maintenance time. With “half a person” allocated, no one has time for that. Without an automated suite, then, testing falls by the wayside, making your changes to a production system even more risky.
You need to automate a robust test suite that lets you know if you have broken anything. This becomes even more critical when you consider that most maintenance programmers haven’t touched the code they modify in a long time, if ever.
Automate Code Reviews
If I were to pick a one-two punch for code quality, that would involve unit tests and code review. Therefore, just as you should automate your test suite, you should automate your code review as well.
If you think testing goes by the wayside in an under-staffed, cost-center model, you can forget about peer review altogether. During the course of my travels, I’ve rarely seen code review continue into maintenance mode, except in regulated industries.
Automated code review tools exist, and they don’t require even “half a person.” An automated code review tool serves its role without consuming bandwidth. And, it provides maintenance programmers operating in high risk scenarios with a modicum of comfort and safety net.
Automate Production Monitoring
For my last maintenance mode automation tip of the post, I’ll suggest that you automate production monitoring capabilities. This covers a fair bit of ground, so I’ll generalize by saying these include anything that keeps your finger on the pulse of your system’s production behavior.
You have logging, no doubt, but do you monitor the logs? Do you keep track of system outages and system load? If you roll software to production, do you have a system of checks in place to know if something smells fishy?
You want to make the answer to these questions, “yes.” And you want to make the answer “yes” without you needing to go in and manually check. Automate various means of monitoring your production software and providing yourself with alerts. This will reduced maintenance costs across the board.
Automate Anything You Can
I’ve listed some automation examples that come to mind as the most critical, based on my experience. But, really, you should automate anything around the maintenance process that you can.
Now, you might think to yourself, “we’re programmers, we should automate everything.” Well, that subject could make for a whole post in and of itself, but I’ll speak briefly to the distinction. Build mode usually involves creating something from nothing on a large scale. While you can automate the scaffolding around this activity, you’ll struggle to automate a significant amount of the process.
But that ratio gets much better during maintenance time. So the cost center nature of maintenance, combined with the higher possible automation percentage, makes it a rich target. Indeed, I would argue that strategic automation defines the art of maintenance.
Well put, and I whole-heartedly agree with all of it. I have been on an Automation team (actually automating tests for our products via hardware and software) for the better part of a year, recently taking over management of it when my predecessor moved out of state to start a family in somewhere other than Texas (long story). As the new manager, one of the things I have had my guys more heavily focus on is automating anything and everything. Automated builds and unit tests are already a given (sadly, our automation code is better unit tested than the production… Read more »
Sounds like a great group to be part of. It always amazes me how many software groups/shops out there seem to take a weird pleasure in doing things by hand in a “work harder, not smarter” kind of way. They automate everything for everyone else but have a blind spot for themselves.
You sound like the kind of manager that’d be fun to work for!
Aww, blush. I’ve only been doing it for 2 months, so we’ll have to ask the team after it’s been a year (although they are the ones who nominated me for the job, or “threw me under the bus” as my predecessor calls it).
We definitely approach everything with a mild irreverence, which probably helps. And we definitely try and channel 2 out of the 3 programming virtues (laziness and impatience. Not much room for hubris in automation, tho).
Could you expand upon what you mean by “automating” a code review? Like checkstyle and other static code analysis tools?
Static analyzers and/or linting tools would probably be the main thrust of this, either out of the box or customized. But I keep it general to say that anything people find themselves covering in review that is repetitive and relatively easily automated should be automated.