Software Craftsmanship and the Art of Software

Context

I’m a member of the LinkedIn group “Software Craftsmanship.” I’m not an active member, but I do like perusing the discussion topics. Recently, I read through this discussion on whether software is more of an “art” or a “craft.” This set me to musing a bit, so I thought I would post about it here.

Is Software a Craft?

A lot of discussion in the group seemed to involve meta-discussion about semantics. “What is a craft?” “What is art?” While this may seem like an academic exercise in splitting hairs, I feel that metaphors for process, software design, and pretty much everything that you might want to do are quite important–so much so that I made an earlier blog post about metaphors for software design in which I discussed this subject at length.

On the idea of whether or not software design and development is a craft, I think there are two operating definitions that potentially cloud the issue. One seems to be the dictionary definition that one might expect, where “craft” is something along the lines of a trade or some occupation that requires skill at something. Using this definition, software design is trivially a craft. There’s not a lot of room for argument there. I think the notion of software as a craft becomes more interesting in the context in which it’s adopted by the Software Craftsmanship movement–the idea of medieval craftsmanship in which there was some notion of apprenticeship being required before the aspiring tradesman could venture out into the world on his own. Perhaps naming the movement “Software Guildsmanship” would be more clear, if a little weird.

Software Guilds

At any rate, the guild metaphor is very good for expressing the feeling that there ought to be some standard met by anyone who wishes to develop software. At first blush, we might think that an undergraduate degree in Computer Science is the standard, but there are still plenty of self-taught programmers and people who attend trade school or else somehow transition into software development. And, furthermore, a computer science degree often gives more of a theoretical background as opposed to teaching would-be programmers the ropes of actual software development. In some very real sense, the CS degree is akin to taking courses in E&M physics to become an electrician. When you get done, you can give an excellent treatise on magnetic flux, but you may have no idea whether it’s really necessary to use conduit when connecting a switch to a junction box.

So, at present, there really is no modern-guild equivalent for software development. They do exist in other professions, though. The aforementioned electricians, lawyers, doctors, and certain types of engineers all belong to the modern equivalent of a guild, whether it’s a union with its political clout or some kind of licensing body (generally the functional equivalent of a union, particularly in the case of lawyers). The principles persist–some required standard for admittance (medical board exams), some notion of standardized pricing of services (union labor), repercussions for not cooperating with the explicit or implicit bylaws of the group (being “disbarred”), etc. The question is whether or not this should apply to software development, and I think that most members of the Software Craftsmanship school of thinking believe that it should.

Me, I’m not so sure. I think the intentions are sound, but the ramifications are potentially odd. Anyone who has looked at a particularly poorly designed piece of software probably, in that moment, would love some standardization of what is acceptable and to have the author of the offending code stripped of some kind of software development credentials. But that sort of overlooks all of the modes in which software is developed.

Lawyers, doctors, electricians, etc. are all sanctioned–“guilded,” if you will–because they provide some kind of external service — they act as agents for individuals or organizations. Software development has a wider range of use. Often we write software and sell it as a product, but sometimes we write software for fun, for friends, as a quick and dirty way to automate something, for posterity (FOSS), or internally for an organization who is both “purchaser” and “vendor” in one swoop. Contrast this with the guilded positions that I’ve been mentioning. People don’t practice law at home to make reading spreadsheets easier. People don’t practice medicine for fun. And some of these guilders could do what they do for free or internally for a company, but even if they did, they would act as an agent for someone who can be legitimately harmed by malpractice. If a pro-bono lawyer messes up, his client can still land in jail. If an incompetent electrician wires up his brother’s apartment, the apartment building can burn down.

With software, it’s a lot murkier. There are many scenarios in which software cannot have any negative ramifications for anyone involved. There is no implied responsibility simply because someone has created some piece of software, whereas there is implied responsibility in the guilded professions.

So Is the Metaphor Valid?

All that said, I think the idea of Software Craftsmanship is a valid one. But I think it’s important to remember that this is an imperfect metaphor. I like the idea of an opt-in software guild (i.e. potentially the Software Craftsmanship movement) in which members agree to practice due diligence with regard to producing good quality software. I even like the idea of apprenticeship, and I think that members of the movement ought to take on ‘apprentices’ (in a strictly voluntary interaction). This group of people ought to be software professionals and aspiring software professionals with the common goal of standardizing good practices and raising the bar of software design in general. But I think a slightly different metaphor is in order. After all, what I’m describing is less like a guild and more like a professional association.

Is Software Development an Art?

To this question, I have to give an opinion of a definite “no.” People like to have their work appreciated. If you spend a long enough time pouring your heart and soul into something, you will want to view it in an artistic context because you’ll want others to appreciate what you’ve done and see beauty in your work. But most forms of art, in the commonly accepted sense, are created for pure aesthetic value. Your software is only art if you never compile it, opting instead to print out the source code, frame it, and hang it over your mantle.

Make no mistake–at its core, software is a tool and a means to an end. You may be particularly efficient in creating these tools, but that doesn’t mean that you’re creating art. You’re automating processes and doing a good job of it. But nobody buys software for aesthetics. The only exception I can think of is software whose purpose is to look pretty or aesthetically appealing, such as video games or fractals. But the art there is in the rendering and not the software plumbing.

To refer to software development as an art, in my opinion, undercuts the value of the activity by implying that the goal is aesthetics rather than utility. We’re not selling people software because they want to own something that has the loosest component coupling they’ve ever seen or because it has no global state–we’re selling people software because they want to do their taxes, check their email, or look for reruns of sitcoms on the internet.

Software development is an art only inasmuch as everything is an art. If I hear developers talking about being good at their art, this to me is akin to mechanics talking about the ‘art’ of fixing cars or engineers talking about the ‘art’ of building planes. It’s only art in the vernacular sense of “This person is so much better at this activity than other people that I’m going to use the term ‘art’ to convey my appreciation.” I would argue that there is a certain danger in taking it any further than that. Specifically, the software designer who believes that his work is art is likely to lose sight of the fact that, without a need, there is no use for his work, and the aesthetics of filling that need do not trump the actual filling of the need. Nestcape might have been destined to be the most beautifully crafted piece of software in the history of mankind, but Internet Explorer still commands the market because its developers were interested in making money, not selling tickets to the grand unveiling.

Tribal Knowledge

In my last post, I alluded briefly to the concept of “tribal knowledge” when developing software. I’ve heard this term defined in various contexts, but for the sake of discussion here, I’m going to define this as knowledge about how to accomplish a task that is not self-evident or necessarily intuitive. So, for instance, let’s say that you have a front door with a deadbolt lock. If you hand someone else the key and they are able to unlock your door, the process of unlocking the door would be general knowledge. If, however, your door is “tricky” and they have to jiggle the key twice and then turn it to the left before turning it back right, these extra steps are “tribal knowledge.” There is no reasonable way to know to do this except by someone telling you to do it.

Today, I’m going to argue that eliminating as much tribal knowledge as possible from a software development process is not just desirable, but critically important.

Background

Often times when you join a new team, you’ll quickly become aware of who the “heroes” or “go-to people” are. They might well have interviewed you, and, if not, you’ll quickly be referred to or trained by them. This creates a kind of professor/pupil feeling, conferring a certain status on our department hero. As such, in the natural realpolitik of group dynamics, being the keeper of the tribal knowledge is a good thing.

As a new team member, the opposite tends to be true and your lack of tribal knowledge is a bad thing. Perhaps you go to check in your first bit of code and it reads something like this:

class SomeClass
{
    Foo _foo;
    Bar _bar;
    Baz _baz;
    ReBar _reBar;
    public SomeClass()
    {
        var _foo = GodClass.Instance.MakeAFoo();
        var _bar = SecondGodClass.Instance.MakeABar(myFoo);
        var _baz = SecondGodClass.Instance.MakeABaz(myFoo);
        var _reBar = GodClass.Instance.ReDoBar(myFoo);
    }
}

You ask for a code review, and a department hero takes one look at your code and freaks out at you. “You can never call MakeABar before MakeABaz!!! If you do that, the application will crash, the computer will probably turn off, and you might just flood the Eastern Seaboard!”

Dully alarmed, you make a note of this and underline it, vowing never to create Bars before Bazes. You’ve been humbled, and you don’t know why. Thank goodness the Keeper of the Tribal Knowledge was there to prevent a disaster. Maybe someday you’ll be the Keeper of the Tribal Knowledge.

The Problem

Forgetting any notions of politics, seniority, or the tendency for newer or entry level people to make mistakes, the thing that should jump out at you is, “How was the new guy supposed to know that?” With the information available, the answer is that he wasn’t. Of course, it’s possible that all of those methods are heavily commented or that some special documentation exists, but nevertheless, there is nothing intuitive about this interface or the temporal coupling that apparently comes with it.

In situations like the one I’ve described here, the learning curve is high. And, what’s more, the pride of having the tribal knowledge reinforces the steep learning curve as a good thing. After all, that tribal knowledge is what separates the wheat from the chaff, and it forces tenure and seniority to matter as much as technical acumen and problem-solving ability. What’s more, it ties knowledge of the problem domain to knowledge of the specific solution, meaning that knowing all of the weird quirks of the software is often conflated with understanding the business domain on which the software operates.

This is a problem for several reasons, by my way of thinking:

1. A steep learning curve is not a good thing. Adding people to the project becomes difficult and the additions are more likely to break things.
2. The fact that only a few chosen Keepers of the Tribal Knowledge understand how things work means that their absence would be devastating, should they opt to leave or be out of the office.
3. The need to know an endless slew of tips and tricks in order to work on a code base means that the code base is unintuitive and difficult to maintain.  Things will degenerate as time goes on, even with heroes and tribal knowledge.
4. When any question or idea raised by somebody newer to the project can be countered with “you just don’t know about all the factors,” new ideas tend to get short shrift, and the cycle of special tribal knowledge is reinforced that much further.
5. Essentially, people are being rewarded for creating code that is difficult to understand and maintain.
6. More philosophically, this tends to create a group dynamic where knowledge hoarding is encouraged and cooperation is minimized.

What’s the Alternative

What I’m proposing instead of the tribal knowledge system and its keepers is a software development group dynamic where you add a customer or stakeholder. Naturally, you have your user base and any assorted marketing or PM types, but the user you add is the “newbie.” You develop classes, controls, methods, routines, etc, all the while telling yourself that someone with limited knowledge of the technologies involved should be able to piece together how to use it. If someone like that wouldn’t be able to use it, you’re doing something wrong.

After all, for the most part, we don’t demand that our end users go through a convoluted sequence of steps to do something. It is our job and our life’s work to automate and simplify for our user base. So, why should your fellow programmers–also your users in a very real sense–have to deal with convoluted processes? Design your APIs and class interfaces with the same critical eye for ease of use that you do the end product for your end user.

One good way to do this is to use people new to the group as ‘testers.’ They haven’t had time to learn all of the quirks and warts of the software that you’ve lived with for months or years. So ask them to code up a simple addition to it and see where they get snagged and/or ask for help. Then, treat this like QC. When you identify something unintuitive to them, refactor until it is intuitive. Mind you, I’m not suggesting that you necessarily take design suggestions from them at that point any more than you would take them from a customer who has encountered sluggishness in the software and has ‘ideas’ for improvement. But you view their struggles as very real feedback and seek to improve.

Other helpful ways to combat the Keepers of the Tribal Knowledge Culture are as follows:

1. Have a specific part of code reviews or even separate code reviews that focus on the usability of the public interfaces of code that is turned in.
2. Avoid global state. Global state is probably the leading cause of weird work-arounds, temporal coupling, and general situations where you have to memorize a series of unintuitive rules to get things done.
3. Unit tests! Forcing developers to use their own API/interface is a great way to prevent convoluted APIs and interfaces.
4. Have a consistent design philosophy and some (unobtrusive) programming conventions.
5. Enforce a pattern of using declarative and descriptive names for things, even if they are long or verbose. Glancing at a method, GetCustomerInvoicesFromDatabase() is a lot more informative than GtCInvDb(). Saving a few bytes with short member names hasn’t been helpful for twenty years.

Divide And Conquer

What Programmers Want

In my career, I’ve participated in projects that have run the gamut of degrees of collaboration. That is to say, I’ve written plenty of software on which I served as architect, designer, implementor, tester, and maintainer and I’ve also worked on projects where I was a cog in a much larger effort. I have been a lead, and I have been a junior developer. But, throughout all of these experiences, I have been an observer, thinking through what worked and what didn’t, what made people happy and what made them frustrated, and what could be done to improve matters.

I have found that I tend to be at my happiest when working by myself or as a lead, and I tend to be least happy when I’m working as a cog. When I first came to this realization, I chalked it up to me not being a “team player” and endeavored to fix this about myself, seeking out opportunities to put myself in the situation and find enjoyment in it. However, I came to realize that I had been subtly incorrect in my self-assessment. It wasn’t that I had an issue with not calling the shots on a project, but rather that I had an issue when there was no decision, however small, that was left up to my discretion–that is to say, if I was working in an environment where a manager, technical lead, or someone else wanted to sign off on anything I did, whether it was as large as submitting a rewrite proposal or as small as what I named local variables inside of my methods or what format I used for code comments.

This was subsequently born out by the experience of working in a collaborative environment where I was not in charge of major decisions, but I was in charge of and responsible for my particular module. I didn’t get to decide what my inputs or outputs would be, but I did get to decide internally how things would work and be designed. I was happy with this arrangement.

Speaking philosophically, I believe that this is important for anyone with a creative spirit and a sense of pride in the work that they do. It doesn’t matter whether the person doing the work is a seasoned professional or an intern. Being able to make decisions, if only small ones, creates a sense of ownership and pride and promotes creative expression. Being denied those decisions creates a sense of apathy about one’s work. In spite of their best intentions, people in the latter situation are going to be inclined to think “what do I care if this works — it wasn’t my idea.” I have experienced firsthand being asked to do something in ‘my’ code and thinking “this is going to fail, but hey, you’re in charge.” This sentiment only begins to occur when you’ve learned by experience that taking the initiative to fix or improve things will result in getting chewed out or being told not to do anything without asking. Someone in this situation will be motivated only by a desire not to get scolded by the micromanagers of the world.

Man Of The People: How to Satisfy Your Programmers

Over time, thinking on this subject has led me to some conclusions about optimal strategies for structuring teams or, at least, dividing up work. I firmly believe in giving each team member a sphere of responsibility or an area of ownership and letting them retain creative control over it until such time as it is proved detrimental. Within that person’s sphere of control, decisions are up to him. It’s certainly reasonable to make critical assessments of what he’s doing or request that he demonstrate the suitability of his approach, but I believe the best format here would be for reviewers or peers to present a perceived better way of doing things and allow the decision to rest with him.

To combat situations where failures may occur, it’s important to create a process where failures are exposed early on. I think the key here is a good division of labor on the development project which, not coincidentally, coincides with good software design practice. Break the effort into modules and appoint each person in charge of his or her own module. One of the first tasks is to define and prioritize the interfaces between the modules, the latter being done based on what is a prerequisite for what. So, if the group is building an expense reporting system, the person in charge of the data access layer should initially provide a small but functional interface so that the person writing the service layer can stub out mock objects and use them for development.

With interfaces defined up front, the project can adopt a practice of frequent integration and thus failures can be detected early. If it is known from the beginning that failure to live up to one’s commitments to others is the only vehicle by which creative control might be stripped via intervention, people will be motivated early by a sense of pride. If they aren’t, then they probably don’t take much pride in their work and would likely not be bothered by the cession of creative control that will follow. Either that, or they aren’t yet ready and will have to wait until the next project to try again. But, in any event, failure to live up to deadlines happens early and will not adversely affect the project. The point is that people are trusted to make good decisions in their spheres of control until they demonstrate that they can’t be trusted. They are free to be creative and prove the validity of their ideas.

If the project is sufficiently partitioned and decoupled with a good architecture, there won’t be those late-in-the-game integration moments where it is realized that module A is completely unworkable with module B. And, as such, there is no need for the role of “cynical veteran that rejects everything proposed by others as unworkable or potentially disastrous.” I think everyone reading can probably recall working with this individual, as there is no shortage of them in the commercial programming world.

I believe there are excellent long term benefits to be had with this strategy for departments or groups. Implemented successfully, it lends itself well to good software design process, though I wouldn’t say it causes it, since a good architecture and integration process is actually a requirement for this to work. Over the long haul, it will allow people hired for their credentials to provide the organization with the benefits of their creativity and ingenuity. And, part and parcel with this is that it is likely to create productive and happy team members who feel a sense of responsibility for the work that they’ve created.

When All Doesn’t Go According to Plan

Of course, there is a potential down side to all of this — it allows for another role with which we’re all probably familiar: “anti-social guy who thinks he’s valuable because he’s a genius but in reality writes such incomprehensible code that no one will so much as glance at it for fear of getting a headache.” Or, perhaps less cynically, you might have the master of the oddball solution. Given creative control of their own projects, these archetypes might create a sustainability mess that they can (or maybe can’t) maintain on their own, but if someone else needs to maintain it, it’s all over.

The solution I propose to this is not to say, “Well, we’ll give people creative control unless we really don’t like the looks of what they’re doing.” That idea is a slippery slope toward micromanaging. After all, one person’s mess may be another’s readable perfection. I think the solution is (1) to make things as objective as possible; and (2) to allow retention of a choice no matter what. I believe this can be accomplished with a scheme wherein some (respected) third party is brought in to examine the code of different people on the team. The amount of time it takes for this person to understand the gist of what the code is doing is recorded. If this takes longer than a certain benchmark, then the author of the difficult-to-understand code is offered a choice – refactor toward easier understanding or spend extra time exhaustively documenting the design.

In this manner, we’re enforcing readability and maintainability while still offering everyone creative control. If you want to write weird code, that’s fine as long as it doesn’t affect anyone else and you go the extra mile to make it understandable. It’s your choice.

Being Flexible

We can, at this point, dream up all manner of different scenarios to try to poke holes in what’s here so far. However, I argue that we’ve established a good heuristic for how to handle them – objective arbitration and choice. Whatever we do, we find a way to allow people to retain creative control over their work and have this justified through objective standards. I hypothesize that the dividends on productivity and team buy-in will counteract, in spades, any difficulties that arise from occasional lack of homogeneity and learning curve in maintenance. And I think these difficulties will be minor anyway, since the nature of the process mandates a decoupled interface with clearly defined specs coming first.