DaedTech

Stories about Software

By

Go On, Live a Little. Denormalize Your Data

I have years of professional experience with and completed several academic courses about relational databases. This is a pretty familiar topic for me, and I’ve worked with a bunch of them: SQL Server, SQLLite, MySQL, PostgreSQL, Oracle, and even MS Access. I’ve always found them, paradoxically, to be both comfortable and awkward. They’re comfortable because of my familiarity with them and awkward because I’ve used them almost exclusively with OO languages and incurred the impedance mismatch.

While RDBMS are in my comfort zone, NoSQL alternatives are comparably alien but also intriguing. Ayende has made some pretty convincing arguments as to why one should think outside the RDBMS, including a .NET Rocks episode in which he talks about constraints, such as expensive disk space, that applied decades ago when RDBMS were designed and are no longer of concern. I’m receptive, my interest is piqued, and I’ve played a bit with MongoDB To understand how these things work. But I think it really clicked for me the other day when I was staring at a schema that had some definite complexity issues. I think I finally groked the use case for document databases.

I was working with a pre-existing schema recently in which (mildly obfuscated) you have a “customer” who can be part of one or more “programs.” The entities “customer” and “program” each have their own properties and, as any diligent normalizer will tell you, that information should not be repeated. So they get their own tables. Since a customer can participate in multiple programs and multiple customers can be in the same program, this is an “M to N” relationship — represented, predictably, by a linking table. What’s more, customers can participate repeatedly in programs so the linking table will sometimes have multiple participation records per customer-program pair, differing by a participation timestamp. I’m sure those of us who have done any significant database development have played out similar scenarios countless times.

As I wrangled Entity Framework to deal with this and other similar relationships that were making it relatively complicated to pull certain information, I started thinking that this seemed harder than it needed to be. There was no use case for handling “what if the program name or another attribute changes in the middle of its offering,” and there really are no good answers to that question. Right now, the implementation is “you can’t do that,” but this is hardly satisfying. To administrative users, it seems arbitrary and like a system shortcoming. But the alternative is unpleasant, too — additional complexity to allow “partial programs” in the schema or the code.

I started thinking about normalization and whether it really mattered here. Allowing duplication is a cardinal RDBMS sin, but it sure would make life easier for a lot of use cases. I mean, imagine a scheme where there was just a table of customers, and participation was recorded as you went, with the program information being duplicated in each participation record. The downside is the information duplication, but the upside is that changing programs midstream is trivial and there is less table overhead and complexity to maintain. No M to N relationships and worrying about whether a program is current or replaced by a new one.

And that’s when it hit me. RDBMS are great for maintaining, well, relational information. For instance, if I work at a company, I have a boss, and my boss has many reports. Modeling the manager-line employee relationship is a great fit for a relational database because it’s about modeling relationships, like “manager to reports.” Tree structures of composition tend to be a good fit as well, such as assemblies like cars and other pieces of machinery. Parts consist of sub-parts and so on. These are relationships in the sense that operations like “delete” and “replace” make sense.

But what about the one-and-done concept of a customer participating in a program one day? That isn’t a relationship any more than I form a relationship with a waiter that I tip and then never see again. That’s something that happened once — not an ongoing arrangement. And that’s where things really snapped into focus for me. RDBMS and the normalized model is great for managing relationships but is not necessarily the be-all-end-all for managing extremely transactional data in the sense of event recording.

And while I’m probably not in a position for this project to quickly introduce document databases and stir up the works too much, I am considering modeling what I might do storage-wise after a document database there in SQL Server. Perhaps it’s time to throw caution to the winds and start allowing some information repetition in there for historical modeling. If the experiment fails, I can always normalize it. But if it succeeds, adding document databases to the mix will be a lot easier since I’ll just be learning the syntax of the API as I go rather than the fundamental underlying concepts and use cases.

By

Laughing All The Way to The Bankruptcy

I was in sort of a smart-ass mood the other day, and I found myself thinking of something incredibly random. Do you remember that show or special called “To Catch a Predator” in which Chris Hansen and a crew of people would set up sting operations for sickos that would try to meet up with young girls? I found myself wondering, “if that crew has LinkedIn profiles, what are they getting endorsed for?” Pederast Snaring? Child Impersonation? What awful skills to have, public service notwithstanding. How might one get much better skills there?

Well, one way is simply to make them up. That would probably be effective. I’ve been endorsed for Ruby and Objective C lately, two languages in which I’ve never written a line of code. If my skills for that can get endorsed, presumably my skills for anything can get endorsed, whether or not they exist. If I wanted to be the most interesting man in the world, what skills would I have? I started a list of things I’d like to see in my profile, in no particular order, to make people say “whoa — I want to know that dude’s story!”

  • Extreme Ice Fishing
  • Dexterity
  • Octopus Farming
  • Dark Arts
  • Phrenology
  • Bear Whisperer
  • Inverted Breathing
  • Street Pharmacy
  • Boomerang
  • NFL Quarterbacking
  • 19th Century Russian Classics Author
  • Gonzo Meditation

One could create a pretty bizarre and interesting composite and probably get plenty of endorsements for it. In fact, I might get more endorsements for these things than programming languages I don’t know because amused contacts would probably go out of their way to endorse me for Octopus Farming. And as I contemplated this, it occurred to me. LinkedIn is a joke.

I don’t mean that in the sense of “oh, it’s really gone downhill in quality,” but I mean that it is actually, seriously now a joke. I can’t remember the last time I heard the site mentioned in casual conversation where the conversation wasn’t about how stupid and funny the skill endorsement system is. And that’s really strange, given its history.

As far as social networks go, each one seems to kind of have its niche and feel. Twitter is sort of like a bar, where there’s rapid-fire, disjointed conversations and things often escalate quickly due to out of context remarks. Google+ is like an abstract art museum. There’s a lot to see, but it’s generally from the same few people, and the place is really weird and quiet. Facebook is like going to a children’s playground. People you don’t know very well bore you with pictures and stories of their children, and the whole world is watching, so nothing happens that isn’t completely vanilla and boring. LinkedIn’s niche has historically been to serve as office space that hosts professional conferences or associations. People meet up there to exchange contact info and have professional conversations.

But that’s all changed more recently with the asinine endorsements and the uptick in punishing recruiter spam. Now it’s like a professional conference brought to you by Amway and Satan where reality shows like “America’s Dumbest Celebrities marry America’s Fattest Ex-Ball Players” stream constantly. If you want to show up and talk about your craft, you have to hear about a super business idea for cleaning products and watch a few short episodes of utter crap before you get down to business. It feels like you’re at the midway point on a journey from professional conference to brothel, where there are just enough booth babes to make everyone uncomfortable.

Amway

So the joke must be on LinkedIn, right? They haven’t figured out just how stupid the world thinks the endorsement system is, and they’ll probably be horrified when they do? Nope. They know it, and they’re laughing all the way to the bank, because here’s how it works. Recruiters can sign up for LinkedIn to recruit people for jobs, and they pay LinkedIn for this privilege in terms of the amount of people they contact and the amount of responses they receive to their overtures. So LinkedIn is incentivized to ensure the highest volume of contact between recruiters and users of their site, and what better way to do that than a system of ‘endorsement’ that is completely trumped up and phony? LinkedIn has all the motivation in the world to create a system where every person is endorsed for every skill and every recruiter emails everyone.

So while we laugh at their joke of an endorsement system and bemoan the fact that we get endless contact requests and junk mail through their site from recruiters, they’re raking it in. And people tend to give in to the sunk cost mentality of not quitting something that they’ve spent a good bit of time over the years building up. People aren’t likely to abandon all of their contacts. But here’s the trouble, strategically. People aren’t going to rage-quit LinkedIn, but their perception of it will alter as it careens toward internet brothel. They’ll stop viewing it as the professional social network and start viewing it as a spam conduit, and they’ll just stay away. Every time they get an email that Judy has endorsed them for “Time Management” and “Multi-Tasking,” they’ll just say, “ugh” and inch ever closer to creating a filter rule in their inbox to send LinkedIn emails to their junk folders. They’ll just quietly stop coming back.

And over the course of time, Linked-In will cease to be a social network. It will instead become a stream of information from LinkedIn’s recruiters (their customers) to the rest of us non-premium users (the product). And then they’ll be no different than Career Builder, Monster, and Dice — with the difference being that those sites never pretended to be anything other than what they are. Perhaps I’m overreacting, but I just get the sense that LinkedIn has jumped the shark. It’s a silly place, and I don’t much care for going there anymore.

By

Prove It

I’m thinking this is going to wind up being a shorter post, and so might some others going forward. Fact of the matter is that I’m burning the candle at both ends a bit lately. I’m under contract for my next Pluralsight course, and I’m also mired in the bureaucratic morass of marching toward a real estate closing, so between that and 55-60-hour work weeks, I’m a little pressed for time. But, that said, we’ll see how it goes.

I was sitting in a meeting today where participants were discussing a recent initiative. (I was, at this point, a relatively passive observer with no horse in the race.) The gist of it was that the initiative was something that would be mildly pleasing to the user base but had some cost associated with it. It was accepted that this would be a bit of a financial loss on the balance sheet but that the ROI would be an eventual bolstering of the brand via goodwill and reputation points, so to speak. The leader of the meeting put a halt to this talk of generalities saying something to the effect of, “okay, so how much is it costing us, how many users are aware of it, and how positive was their response?” The people who had been discussing this got sort of deer-in-headlights. They seemed nonplussed that they were being asked to quantify something like this. I loved the questions, personally. I thought, “yeah, get those metrics, and then we can at least begin modeling scenarios for evaluating if this is worth doing.”

After that meeting, I retreated to my desk, and worked into the evening at the kind of tidying up that I can only seem to get done after business hours when I close the office door. One of the miscellaneous tasks was to come up with material to present tomorrow at the inaugural team lunch and learn — an initiative that I recently started. I don’t have a talk in my back pocket at the moment, so I was weighing options for it and settled on showing a video of a Michael Feathers talk about the synergy between testability and good design. I’ve recently introduced unit testing to the team, and it seems like a good perspective about unit testing beyond some of the more standard rationales. In the early part of the talk, Michael says (paraphrased from memory) that he often annoys developers by telling them that if their design isn’t testable, it’s not a good design. It’s quite a concept — if it can’t be easily verified, it isn’t very good.

Earlier this evening, I was relaxing and working my way through The Clean Coder by Uncle Bob Martin. In it, Bob cites something that Kent Beck told him about arguing — if an argument goes on for more than five minutes or so, then it’s really a religious war. If arguments are being based on facts, data, and empirical evidence, they’re settled pretty quickly. Rather than continuing to argue and get angry, both sides should regroup, do some research, and revisit the issue later. You can’t make a persuasive case for something without raw, hard data.

This struck me as interesting. Within the span of half a day, I was exposed to three separate events with a very fundamental and powerful underlying theme: “prove it.” If it can’t be proved, what good is it? As the characters in the Game of Thrones books seem to like to say, “words are wind.” Don’t tell me that users like it so it’s worth doing; show me evidence of that. Don’t tell me that your design is great; show me that it works for the required inputs. Don’t argue with bluster; show me that you’re right in such excruciating detail that there is simply no argument.

I don’t think anybody would find what I’m saying here controversial — proving and demonstrating are obviously better than simply claiming things without support. But I think the challenge for all of us is to prove (or at least to attempt to prove) things that we wouldn’t necessarily think of proving. How do you prove that user goodwill justifies spending X dollars? I don’t know, but it’s an interesting challenge. And beyond being an interesting challenge, it’s the kind of question that being able to answer will set you apart from the people you work with and make you in high demand. Do everything that you do in your professional life operating under the assumption that someone will audit your thought process and demand, on the spot, to know why you decided to do what you did. Why did you spend a day changing all methods in the code base from Pascal to camel case? Can you justify what that cost your group in terms of labor? How would you make the case for that?

I’m not suggesting that we should all operate constantly as if there were some insane, whip-cracking micromanager monitoring our every movement. What I am saying is that you can stand out by constantly and easily demonstrating the value and sense in the decisions that you make and the actions that you decide to take. Be that guy or gal. It’s less common than you think.

By

Notes on Job Hopping: Fear, Loathing, and Paying Your Dues

I was in a hotel the other night, and I had forgotten my Kindle at home, so I laid in bed for some time, catching up on my RSS feeds. I came across this post by John Sonmez, which I read with interest all the way through before I clicked over to read the post to which he was responding. John and I have very similar work ethics, so the part of his post about soldiering on during times when the ‘fun’ thing you started has lost its new car smell really resonated with me. Almost identically to him, I started blogging purely because I thought it’d be fun, but now there are nights where I have no queued posts and I really don’t feel like writing one. But I usually do anyway. A lot of things in my life follow that pattern, even beyond just programming — home improvement projects, setting up an ALM and ticketing system at work, gardening, etc.

And the point-counterpoint of these posts was interesting. The original poster, Loren, basically said that he’d been passionate about joining a startup, but fast forward two years and the job became soul sucking and unfulfilling even as it was cushy and relatively easy — startup Hotel California. So, in a celebratory announcement post, he said that he’d decided to quit his job and see where the wind blew him and that he’d already felt the passion start to reignite for things he likes to do.

I read John’s counterpoint to this as essentially saying, “don’t be that guy.” The guy I mean is the one you knew as a kid who had a closet full of hobbies and interests he’d started and abandoned after a few weeks. There was a violin he’d played for a little while, that old yellow belt from a brief karate stint, a rock tumbler, a football helmet, etc. We associate that with childhood behavior because, as a child, there really aren’t any stakes to just kind of quitting things and picking up new ones as you please. But as an adult, being a dilettante like this tends to have real life consequences. Most people outgrow it as obligations like spouses, mortgages and children enter the picture. But you still occasionally find Uncle Bill with the grown-up closet full of failed dreams of the moment: menus from the restaurant he tried to start, a Microsoft Programming Certificate, his tools from briefly trying to flip houses, etc. I think John was saying, “Don’t Be Uncle Bill. You’re never going to love something the way you did when you fell in love with the idea, and you can’t chase that dragon forever or it will bankrupt you.”

Simple enough: “I want to go chasing rainbows and I think that’s great” versus “chasing rainbows isn’t a career strategy.” Except, in the comments section on Loren’s post, things took a turn for the weird. At first I thought some of the hostility there was really bizarre for a post that just amounted to “I quit my job and I feel a lot better.” Then I thought that it was probably just a byproduct of coming from Hacker News. (If it is principally a “content and comment” engine like HN, YouTube, Reddit or Slashdot, I find that reading the comments tends to be a vaguely depressing experience, like riding a train through a crowded city with lots of debris and graffiti and broken windows everywhere.) But in the comments I saw a trend where the phrase “pay your dues” was repeated a number of times, and it made more sense to me.

Thank You, Sir, May I Have Another

I had intended this series to be a trilogy, rounded out by a post in which I prognosticated about the future of job-hopping (and that’s still coming), but I decided to add a chapter after reading these posts the other day. In the last post, I talked about the dues-paying culture in a corporate setting as one in which a general standard of mediocrity is forced upon the employees in an attempt to make everyone equally (dis)satisfied without playing favorites — you’ll wait three years for your promotion and pay your dues just like everyone else, mister.

At this point, I’d like to point out that I agree with the sentiments in John’s post from a rational self-interest perspective. That is, I think that quitting a job to spend some time with your hobbies is not a smart career move, and telling yourself otherwise is just a rationalization (though I have no quarrel with someone choosing to do this — I’m very small “L” libertarian when it comes to matters of “live and let live.”) But I do differ from John on a small point, which is really his invocation of the concept of dues paying. That is, he makes a reference to paying your dues in a sense that I think of as simple time investment. You start a project and it’s fun, you keep at it when it stops being fun, and eventually you deliver and reap the benefits. I don’t really think of this as paying dues but as investing time into an effort, pure and simple.

Distinguishing my idea of dues paying from John’s may seem like semantics, but I consider it important for two reasons. Firstly, the rest of this post is frankly going to be an attack on the mentality of “people should pay their dues,” and I don’t want to include John’s message about following through in the attack. Secondly, I think that dues paying is actually a perversion of investment. (For a lengthier dive into this topic, feel free to check out the conclusion of the Expert Beginner ebook when it comes out). Investing in one’s future is working hard now for greater opportunity later, whereas dues paying is some combination of Ponzi scheme and working hard now so you can be lazy later.

First of all, dues paying has a literal meaning and an idiomatic meaning. The literal meaning is that you exchange something of value (generally money, but not necessarily) for membership in some sort of club or organization. Dues-requiring organizations that come to mind, off the top, are unions, professional organizations, country clubs, condo associations, fraternities and sororities, and gangs (often weird barter dues involving violence). By and large, you trade something of value for a personal identifier — I pay club dues because I want to be a member of the club. Dues paying, literally, is a way to purchase identity status and its attendant privileges.

Frat Guy

The idiomatic meaning has evolved to be this: you suffer in the here and now for rewards to be reaped at a later date. It has culturally-ingrained, protestant-work-ethic overtones as well. It isn’t just that you sacrifice now to be rewarded later, but that the sacrifice — the pain, suffering and indignation — is a character-building, humbling benefit tied up with a mastery and maturity bow in the end. It’s a nice narrative, like that of the Karate Kid.

But how do you get from point A, a payment in order to recieve a certificate of identity and belonging, to point B, a humble prostration before a pack of the worthy? I mean, shelling out 20K a year for the right to play golf hardly evokes images of Daniel-San waxing his way on and off to the karate championship. So what gives? Well, I’d say what gave was the increasing status disparity between early and later members. To put it more bluntly: the more people that get into a club, the more ridiculous the things they make new members do. And, thanks to the cognitive biases, the more the new members do those ridiculous things, the more they start to manufacture value for the experience. Before you know it, victims of horrific and bizarre hazing rituals actually reflect fondly back on them as character-building events that make the club they’re in that much better. They were just payin’ their dues, proving they wanted to be in the club badly enough to get in.

Dues Paying in the Workplace

Doesn’t it strike you as fundamentally weird to juxtapose club membership and salaried employment? That is, when people around the office talk about “paying your dues,” stop for a minute and think how odd that is (unless it’s their union rep talking to them). Organizations that collect dues are selling identity whereas corporations are purchasing labor from their employees. So if I have to “pay my dues” in a company, that means that it’s purchasing labor from me and I’m purchasing identity from it. I’m selling my labor to belong to some cabal of tenured people there that get to mete out the good assignments as they see fit, like fraternity pledge masters handing out the chores to the pledges or gang leaders handing out drugs or whatever to their soldiers. Within the organization, a club forms. It acts orthogonally to the organization’s best interests as it bolsters its own social currency on the backs of captive, hapless newbies who, unlike college freshmen, can’t simply say, “leave me alone — I’m not interested in your stupid club.” At least, they can’t say that without job hopping.

It’s in this culture that the corporate notion of dues paying really thrives. It’s the people that have been with an organization the longest that get to form the club and define its rules, currency (real or social), hazing rituals and customs. It’s cultures like this that marry dues paying to time invested rather than quality. And it’s the hapless job hoppers that bounce around until they find somewhere that this self-important silliness is muted to a minimum.

Of course, almost nothing infuriates a member of the club like someone not wanting to play — it’s a direct rejection of a carefully crafted culture and a whole lot of manufactured value. “Well, Jim, you’re new, but the guy with the most years here gets the best donuts, so — what, you don’t like donuts? Oh, sure, whatever. Heh, you’re clearly jealous.” I say almost nothing infuriates like opting out, because the one thing that really redlines club members is someone who scoots in with some kind of favored status, effectively bypassing the hazing, the rituals and the traditions. That’s a dogfight.

People who talk to you about dues paying in a corporate setting (and mean it in the sense I’m talking about here) have one of three main motivations: jealousy, a sense of being threatened, or external rationalization. Jealousy is the easiest to understand and probably the most common, especially from a veteran of dues-paying cultures in the abstract. This was on display in the wailing and gnashing of teeth from the HN set in Loren’s post. It doesn’t take Freud to figure out that a person getting disproportionately angry about someone else quitting his job to have fun is jealous.

A feeling of threat to one’s validity is also closely related. If you get promoted to architect at age 30 whereas a crowd of “wait your turn” dues payers around you didn’t get there until 40, it triggers thoughts of “what’s wrong with me that I didn’t get there at 30?” That’s a rather existential threat that’s hard to swallow, and it goes beyond simple jealousy. There’s an up-and-comer on the way, and he’s clearly gunning for this position. It’s this second motivation that tends to trigger the most action in a corporate setting, with the Council of Elders appealing to some AVP somewhere that these kids don’t know the value of respect. The Council will bring the full weight of their clique’s political influence with the company down on the newbies, refusing to rest until the youngsters are scrubbing PHP code off the toilets and doing pushups in the server room. It’s good for them. Kids these days have no respect and need to pay their dues.

The final motivation is rationalization. It’s the most interpersonally benign and depressingly insidious. This motivation occurs after the club has dominated and stamped out meritocracy altogether and some manager is left to explain a soul-crushing HR promotion matrix to a talented recent hire. This is when a boss says to his report something like “we think you have a bright future here, so pay your dues and you’ll be rewarded down the line,” when what he really means is “I’d promote you, but there’s a rule that I can’t for three years and I think that happened because there’s all of these lazy senior people here that have already paid their dues and will throw a fearsome temper tantrum if we give you tasks that are challenging and fulfilling before the three-year mark.”

My message to Loren, if you’re reading, and to anyone in general is don’t bother paying dues. That’s for MacLeod losers, golfers and unions members. If you can skip dues paying and enjoy success, by all means do it. Define yourself by what you produce and what you accomplish, not by how much you suffer for some kind of token acceptance from some group primarily interested in you succeeding no more rapidly than they did. Don’t stick around waiting for them to let you in — leave and start your own club where you set the rules and regulations. Create a place where you’re free to abide by the rules of meritocracy and ambition.

Both your money and your time are finite resources. You have to choose between putting them toward corporate/industry identity membership and investing them in your own career, yourself, and your life. I think it’s a no-brainer, personally. Paying your dues gets you comfortably in the door if you just hang around long enough and absorb the hazing, but blazing your own trail lets you buy the property that the clubhouse is built on.

By

Defining Done for Your Deployment Process

A Tale of Debugging

The other day, someone came to me and told me that a component of one of the web applications that my team maintains seemed to have a problem. I sat with her, and she showed me what was going on. Sure enough, I saw the issue too. It was the kind of integration thing for which we were able to muster up some historical record and see approximately when the problem had started. Apparently, the problem first occurred around the last time we pushed a version of the website into production. Ruh roh.

scooby

Given that this was a pretty critical issue, I got right down to debugging. Pretty quickly, I found the place in the code where the integration callout was happening, and I stepped through it in the debugger. (As an aside, I realize I’ve often made the case against much debugger use. But when legacy code has no unit, component or integration tests, you really don’t have a lot of options.) No exceptions were thrown, and no obvious problems were occurring. It just hit a third party API, went through it uneventfully, but quietly failed.

At this point, it was time for some tinkering and reverse engineering. When I looked at the method call that seemed to be the meat of the issue, I noticed that it returned an integer that the code I was debugging just ignored. Hovering over it, XML doc comment engine told me that it was returning an error code. I would have preferred an exception, but whatever — this was progress. Unfortunately, that was all the help I got, and there was no indication what any returned error code meant. I ran the code and saw that I was getting a “2,” so presumably there was an error occurring.

Maddeningly, there was no online documentation of this API. The only way I was able to proceed was to install a trial version of this utility locally, on my desktop, and read the CHM file. They offered one through their website, but it was broken. After digging, I found that this error code meant “call failure or license file missing.” Hmm… license file, eh? I started to get that tiny adrenaline rush that you get when a solution seems like it might be just around the corner. I had just replaced the previous deployment process of “copy over the files that are different to the server” with a slightly less icky “wipe the server’s directory and put our stuff there.” It had taken some time to iron out failures and bring all of the dependencies under source control, but I viewed this as antiseptic on a festering sore. And, apparently, I had missed one. Upon diving further into the documentation, I saw that it required some weirdly-named license file with some kind of key in it to be in the web application root’s “bin” folder on the server, or it would just quietly fail. Awesome.

This was confirmed by going back to a historical archive of the site, finding that weird file, putting it into production and observing that the problem was resolved. So time to call it a day, right?

Fixing the Deeper Issue

Well, if you call it a day now, there’s a good chance this will happen again later. After all, the only thing that will prevent this after the next deployment is someone remembering, “oh, yeah, we have to copy over that weird license file thing into that directory from the previous deploy.” I don’t know about you, but I don’t really want important system functionality hinging on “oh, yeah, that thing!”

What about a big, fat comment in the code? Something like “this method call will fail if license file xyz isn’t in abc directory?” Well, in a year when everyone has forgotten this and there’s a new architect in town, that’ll at least save a headache next time this issue occurs. But this is reactionary. It has the advantage of not being purely tribal knowledge, but it doesn’t preemptively solve the problem. Another idea might be to trap error codes and throw an exception with a descriptive message, but this is just another step in making the gap between failure and resolution a shorter one. I think we should try avoid failing at all, though having comments and better error trapping is certainly a good idea in addition to whatever better solution comes next.

What about checking the license file into source control and designing the build to copy it to that directory? Win, right? Well, right — it solves the problem. With the next deploy, the license file will be on the server, and that means this particular issue won’t occur in the future. So now it must be time to call it day, right?

Still no, I’d argue. There’s work to be done, and it’s not easy or quick work. Because what needs to happen now is a move from a “delete the contents of the server directory and unzip the new deliverable” deployment to an automated build and deployment. What also needs to happen is a series of automated acceptance tests in a staging environment and possibly regression tests in a production environment. In this situation, not only are developers reading the code aware of the dependency on that license file, not only do failures happen quickly and obviously, and not only is the license file deployed to where it needs to go, but if anything ever goes wrong, automated notifications will occur immediately and allow the situation to be corrected.

It may seem pretty intense to set all that up, but it’s the responsible thing to do. Along the spectrum of some “deployment maturity model,” tweaking things on the server and forgetting about it is whatever score is “least mature.” What I’m talking about is “most mature.” Will it take some doing to get there and probably some time? Absolutely. Does that mean that anything less can be good enough? Nope. Not in my opinion, anyway.