DaedTech

Stories about Software

By

The Best Way to Hire Developers

Editorial Note: I originally wrote this post for the Infragistics blog.  You can find the original here, at their site.  There’s a lot of good stuff over there, so go give it a look.

The other night, I was remembering what might have been my most impressive performance in the interview process.  What makes this performance particularly interesting, however, is not how well I did, but rather how I did well.  And the how left me feeling unsatisfied with myself and with the process.

I was interviewing for a software development position, and this particular organization’s interview process was (1) phone interview, (2) programming exercise, (3) in-person interview. The phone interview went pretty well, and the recruiter had told me that the company was excited about me – a mildly good sign, for whatever it was worth.

However accurate the recruiter’s assessment may or may not have been, the company’s feelings were positive enough to give me the programming exercise.  This all occurred back when I was in grad school, and, at the time of this particular interview, I was in a class called “Advanced Database Design,” in which we explored persistence options beyond the traditional, relational database.  This was a bit of an avant garde class, at the time, because the NoSQL movement had yet to gain a ton of steam.

When they handed me the programming exercise, I had just, in this very class, wrapped up a chapter in which we’d studied using R-Trees to store geographical information.  This unit of study included what they were, how they were used, and a bit of pseudocode to really drive the point home.  As fate would have it, the R-Tree happened to be an extremely elegant solution to the programming exercise for this interview.

BigPileOfMoney

The Anatomy of Winning an Interview

This exercise felt like, as they say, shooting fish in a barrel.  I recognized that the thing I had just learned was applicable to my situation, and I, well, applied it.  I ran some automated tests on it for verification purposes and then turned the exercise around pretty quickly.

The company was ecstatic, and I didn’t need to try to read their reaction through the recruiter.  They reached out directly about flying me out for an interview and told me, point blank, that this was by far the best solution they’d seen submitted to the exercise.  I felt as though I were the hero in some really lame Indiana Jones movie, where the ancient gatekeeper had been waiting millennia for someone to come along and solve the programming exercise with an R-Tree.

I wound up passing on the interview, but only because something better came along as I negotiated the details of flying out to meet them.  They were clearly disappointed, and they encouraged me to let them know if I changed my mind.  I liked the company, and this overture made me feel good.

And yet, I came away from the whole experience feeling strangely empty.  As good as it made me feel to have impressed a company so profoundly, there was no denying that this was largely a matter of luck.  Had I done the coding exercise a few weeks earlier, I wouldn’t have known about R-Trees.  Had I done the exercise a few months later, I might have forgotten about R-Trees altogether as an option.  In neither case would I have blown away my interviewers, meaning my ”unprecedented” performance was a happy coincidence rather than a reflection of me being some kind of uber-programmer.

Programmers, Don’t Sweat “Failure”

Have you ever sat for an interview or programming exercise that demanded knowledge you simply didn’t have?  Famously, if you ever have occasion to interview for Google, you’ll discover that their initial phone interview is basically like an oral midterm from your undergrad CS program’s senior-year algorithms class.  O notation. Sorting algorithms. All that fun stuff.  They aren’t alone in that by a long shot, but their interview process is iconic.

To combat being caught off guard, you probably study up before you hit the job market, making sure you can field a wide array of technical questions.  After all, you never know what they’ll throw at you.  But the thing is, you never know what they’ll ask—or what knowledge you or any of the other candidates have recently acquired that will be perfectly timed for this interview.

You may be up against someone like me who just happens to have read a white paper detailing a solution to the exact problem of which the interviewers are so proud.  If that’s what happens, well, you’re just out of luck.

And that’s why you shouldn’t feel bad.  The interview process is actually depressingly random and non-scientific.  “Failing” an interview doesn’t mean that you’ve actually “failed” at anything. It just means that you haven’t had the good fortune to read the material that would have let you pass.

Companies, Focus on the Right Things

Do you have an interview process like this one?  Do you have a process where, all else being equal, you’ll wind up hiring a candidate that just happened to read up on R-Trees?  If you do, I’d suggest some introspection to evaluate whether or not that might be a hiring process smell.

In my experience, interviews aim to understand one or more of the following three things: what does this candidate know, what has this candidate done, and what can this candidate do?  These are ordered both in usefulness (least to most) and difficulty to evaluate (least to most).

What a candidate knows is probably the least useful, since human beings spend their lives learning and what they know can change easily.  But it’s easy to evaluate. Include trivia quizzes and the like as part of your interview process, and you can figure out what they know.

What a candidate has done is a little trickier, but it’s more useful.  I mean, sure, they have resumes and anecdotes, but dissembling is always a possibility.  You can be more sure that they know what an R-Tree is by quizzing them than you can be sure that they’ve implemented R Trees by hearing them make that assertion.  But if they’ve successfully implemented it before, that’s more valuable than them knowing what one is.

But the importance of what they know or what they’ve done pales in comparison to this: “Will this person, confronted with a situation that needs an R-Tree, do the requisite research and translate that into a solution?”  That’s what you really want to know. Those last two things are just proxies for trying to know it.

I certainly don’t have any unprecedented insight into the best way to conduct interviews. Evaluating candidates is hard.  But I can tell you that the closer you get to creating evaluations for what candidates can do, the more likely you are to have successful hires.

Want more content like this?

Sign up for my mailing list, and get about 10 posts' worth of content, excerpted from my latest book as a PDF. (Don't worry about signing up again -- the list is smart enough to keep each email only once.)

49 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Minnesota Steve
Minnesota Steve
8 years ago

I really don’t like the programming questions. Or even questions about things I can look up in a book if I need it. I’ve found the best interviews involved a discussion. Like me drawing out a rough architecture of some major change I worked on, and then talking about the benefits or disadvantages we uncovered.

I’ve dealt with a lot of people who knew all the right buzzwords, but when you got right down to it you could tell that they had no clue why. That’s something a coding challenge won’t uncover, but a good discussion will.

Peter Comaish
8 years ago

Brilliant! I hope ALL recruiters READ this and LEARN from it. Failing that, at least some of the good ones would be brilliant and then they would become the best!

Randy A MacDonald
8 years ago
Reply to  Peter Comaish

Forwarding this article, as I have, to recruiters one has encountered, is a good strategy.

doug_b
doug_b
8 years ago
Reply to  Peter Comaish

Recruiters = mindless bottom feeders.

Paul Pettersen
Paul Pettersen
8 years ago
Reply to  doug_b

Good ones do exist. But mostly those who have no skills to do any real job, recruits people who can!

Flemming Bonde Kentved
Flemming Bonde Kentved
8 years ago

I partly agree. Ofc the buzzwords should give no real points before they have been put to a test. Let’s say I need a programmer to develop c# business applications. Then a buzzword would be Entity Framework, ofc I then want that person to explain what it’s good for and also ask some questions on how to do different things. They might not need to sit and program, with me ready to judge, but I might show them some code and ask them to explain what it does behind the scenes.

Erik Dietrich
8 years ago

Over the last few years, I’ve found myself conducting a lot of interviews and designing the format of the interview process. I always feel like what I’m doing (to paraphrase Churchill) is the worst process I’m aware of, except for all the other processes I’m aware of. What I’ve roughly settled on is giving candidates a brief programming exercise, which serves mainly as a discussion starter. After they complete it, the interview portion is a discussion centered around them explaining their rationale for design decisions and approach (surfaces plagiarized solutions and allows them to demonstrate conceptual grasp of principles). Then… Read more »

Francis Kim
8 years ago

I too hate programming questions with a passion.

Robert-T
Robert-T
8 years ago

Excellent article. I seldom read articles thoroughly and almost always lose concentration, resulting in skimming through and skipping most of it. But your article here is very good. I’m not in a position to hire anyone but if I were I would utilise what you have suggested. I also think throwing a candidate into the deep end when they start work is important. If they can do what they claimed during the hiring process, then that should be apparent from the outset when they start work. If they cannot do the work then they should be dismissed. This is probably… Read more »

Mr. Peter
Mr. Peter
8 years ago
Reply to  Robert-T

Try the Soviet Union. I believe their model would work for you.

LionessLover
8 years ago
Reply to  Mr. Peter

It’s just a very obvious trolling bored teenager. No use responding.

Erik Dietrich
8 years ago
Reply to  Robert-T

I’m not sure about the consolidation you mention, as it seems to require a significant bit of legal/governmental reform to be enacted. Notwithstanding my personal politics, I tend to favor approaches that can be achieved with action rather than lobbying. That said, the first part of your response resonates with me. In my ideal world, I would ‘interview’ candidates by paying them to work on a small, relatively non-critical assignment. The interview process as it exists today has a peak predictive efficiency of something like 30% (per Lazlo Bock’s studies), and I attribute this to it trying to predict the… Read more »

Randy A MacDonald
8 years ago

As someone who sees the ‘situation that needs to be done in X manner’ as patently false,I question the value of the most important question as it is stated.

Erik Dietrich
8 years ago

Do you mean, “what can the candidate do?” If so, I don’t mean this in the sense of order-following, but in the sense that the person can be trusted to operate autonomously. Frankly, if I’m hiring software developers, I have little use for people that need prescriptive direction, unless the hire is someone that’s pretty early in their journey.

Randy A MacDonald
8 years ago
Reply to  Erik Dietrich

That begs the question: how far along in their journey should they be? and in what journey?

rockaUniverse
rockaUniverse
8 years ago

The interview process and how it is conducted should depend on the position being interviewed for. The process will be different for a beginner programmer vs a senior programmer. For a beginner programmer, the interviewer must be realistic and knows that the beginner programmer will not have the experience nor a solid portfolio of projects to boast about. Instead, like Minnesota Steve said, he should be tested in his/her approach to put forward a solution, step by step, to a particular problem and even putting together a design in tackling a problem. Answering programming questions do not prove anything as… Read more »

Erik Dietrich
8 years ago
Reply to  rockaUniverse

Out of curiosity, are you familiar with the shu-ha-ri concept? It seems like there are some parallels to what you’re saying and that approach.

doug_b
doug_b
8 years ago

Rarely do I read anything to do with programming. I’m the ‘old guy’ on the block. Got my MS in comp sci 1971. At that time it was a career. Now, I’m not even sure it’s a job. I would rather my daughter come home with a plumber or carpenter than a programmer. Big business and nano-management have killed the profession (unless you like to kiss a**). It’s not a profession for thinking adults, more like guy’s who never really want to grow up. Next time you get your car repaired, dentist, doctor, lawyer, be sure to give them a… Read more »

Erik Dietrich
8 years ago
Reply to  doug_b

I’m not sure I agree with how dire your assessment of programming is, but I do agree with your main premise. A frequent topic of mine on this blog, actually, is how I think programmers should stop accepting a status quo where they’re subordinate to large management hierarchies and career project managers. The comparison to doctors/dentists/lawyers especially resonates with me, as I think that programming, as a professional, should look similar to those professions. Lawyers don’t bring in “lawyer managers” to give them orders and tell them what to do — they hand planning and logistical work out to subordinates… Read more »

Ergulon
8 years ago
Reply to  Erik Dietrich

Good point, Erik, but I would throw in that a lot of these professionals can get stale, especially hidebound in the case of Doctors. Also, lawyers (as I’ve gleaned from binge watching 4 seasons of the Good Wife) can be terribly prone to mismanaging their careers.

pmarfleet
pmarfleet
8 years ago
Reply to  Ergulon

And politicians (based on all series of The Good Wife)

rockaUniverse
rockaUniverse
8 years ago
Reply to  doug_b

I can appreciate what doug_b is saying and what the current profession of programming has become. In today’s society, the profession has virtually eliminated the purpose for having formal training as a programmer and would rather create a jungle where anything goes. A computer science degree bachelor or master is almost completely ignored when looking for a job. It shouldn’t be so because this is what should be preparing you to follow standards, build professionalism, be innovative, be creative and be apart of solving problems at the highest level. It’s been documented that only 30% of software are delivered on… Read more »

Dan Sutton
8 years ago

I’ve got a three-question test I give – one SQL question, one algorithms one, and just for the sake of it, a Boolean logic one, to see if they’ve ever used anything lower-level than JavaScript, and all the questions require a certain degree of lateral thinking if you want to find their ridiculously simple solutions. But I’ve found that the most successful hires I’ve made in the past have nothing to do with that: my experience is that real programmers recognize each other like vampires. One of my most successful hires required a five-second interview on the phone before I… Read more »

Erik Dietrich
8 years ago
Reply to  Dan Sutton

I think the idea that top-notch folks can sense their own is spot on. I wish I could make it algorithmic, somehow, though. I feel the same as you — that I’ve developed a pretty good sense for the kind of people I’d want to work with and that I’d trust. But “I have a sense” is a lot less empirical than I’d like. (Seems like a hard problem to solve)

Jmoore
Jmoore
8 years ago
Reply to  Erik Dietrich

Hi Erik,

I recently posted something about this exact thing … I’m still pretty confident there’s something in watching a fellow coder work that puts an empirical spin on it. If only you could figure out a way to eliminate the ‘stressed’ nature of over-the-shoulder observation…

http://humanumbrella.com/2015/10/a-proposed-solution-to-the-software-engineering-interview-problem/

Great article!

Cheers

Erik Dietrich
8 years ago
Reply to  Jmoore

Glad you liked! I think the idea of a screen-cast/recording of someone approaching a problem is an intriguing one, and one that had never occurred to me. I imagine that there’d be some logistics to sort out, but it seems like it could provide some interesting data points. (This also appeals to me, since I do a lot of screencasting)

Dan Sutton
8 years ago
Reply to  Erik Dietrich

I know. It’s hard to define, let alone to try to formalize. But somehow, it seems to work for me…

MarcelDevG
MarcelDevG
8 years ago
Reply to  Dan Sutton

“real programmers recognize each other like vampires”
Ha, that’s a very good one!

Andrei Dragotoniu
7 years ago
Reply to  MarcelDevG

completely true though ..

Dave Bush
8 years ago
Reply to  Dan Sutton

This made me laugh! The problem with interviewing (in any field) is that the decision is always made in nanoseconds. The rest of the time interviewing is trying to find reasons for the decision we’ve already made. The gig I currently have was all technical c#, but 90% of what I’m doing is JavaScript. I passed the technical interview easily, but really, I think I was hired because I sounded like I could do what they needed to have done. Prior to this gig, the interview was “I know nothing about JavaScript, start talking…” Really, just a way to find… Read more »

Dan Sutton
8 years ago
Reply to  Dave Bush

It’s funny: I hired one guy once who appeared to be the best of a bunch of not-really-satisfactory options, but there was a time crunch. And he could talk about things very intelligently. The problem was that he was very, very good at talking – he couldn’t actually do anything. He lasted a short time, then we fired him… and the next thing you know, we got a call from another company asking if it was true he’d left us because we switched to PHP (it wasn’t) and what did we think of him – so we told them something… Read more »

Erik Dietrich
8 years ago
Reply to  Dan Sutton

A long time ago, I made an off-handed remark to my wife (not in the industry) about how a non-trivial chunk of people with the title “programmer” didn’t know how to program. She was absolutely aghast and amazed at this, and I realized how telling it was that I wasn’t, after being in the industry for a long time.

Dan Sutton
8 years ago
Reply to  Erik Dietrich

Oh, it’s stunning. It’s frustrating beyond hell when I’m interviewing people… but then I realize it means I’ll never be unemployed!

Andrei Dragotoniu
8 years ago
Reply to  Dan Sutton

haha, that’s an amazing way to look at things!

Jerome
Jerome
8 years ago
Reply to  Dan Sutton

I interviewed couple of guys from a country in Asia (living in the USA). They had 6-8 pages of resume! all of them had ‘unit test’ written all over the resume. So…I asked about unit test. they did not know about using Mocks. You want to tell me that you do unit test in 8 different places, non of them use Mocks? yeah…..right…. Then we went to the white board. easy questions. they did not know any of the questions. After that, their recruiter talked to us and said “they felt it went well, but it was too technical”…. too… Read more »

Dan Sutton
8 years ago
Reply to  Jerome

My favorite was an old Russian guy who came in – he was at least 65, but could’ve been far older… For the SQL question, he said, ‘I kyennot do dis”, so I asked why and he said, “Is YesQL. I kyennot.” For the algorithms one (produce an array containing the numbers 1 to 20 in random order) he said, “I kyennot do dis,” so I asked why and he said, “Is not byizniss quyestion!” But for the Boolean logic question (“if a=23 and b=36, what are the values afer: a^=b; b^=a; a^=b;”) he wrote down “a=36; b=23” with no… Read more »

Paul Pettersen
Paul Pettersen
8 years ago

The worst part of the interview process is the trivial technical section. You know 101 useless terms you barely use when developing but they seem to think you need to know them? I’m coming from the dark side, non-object oriented programming. I exclusively code OOP C#.Net now, but I’d prefer to show them I have these skills. I don’t care how much I looked at the terminology, every interview offers a strange new question. Probably the thing that drives me nuts the most are HR people who think they know IT and have no clue. I just came off a… Read more »

Erik Dietrich
8 years ago
Reply to  Paul Pettersen

Wow, yeah, having HR people do technical screens is definitely an anti-pattern, in my book. I think you see this because a recruiter/HR screener is a cheaper hourly investment than a developer, but, then again, so is a landscaper or a hair dresser, and companies don’t ask those people to screen developers.

Stefano Borini
8 years ago

Honestly, I think coding exercises during an interview are a waste of time. No, I won’t code for you an arbitrary problem that has already been solved to perfection in a library somewhere. You want to see if I can code? Check my github. There’s plenty of code written by me that solves a problem I had and took the time to solve properly. There, you will see how clean I code, how I design, how I document and test. Asking me for the n-th time to do a tree traversal is a waste of time. I don’t care about… Read more »

Erik Dietrich
8 years ago
Reply to  Stefano Borini

Personally, when interviewing people, I’d rather look at a portfolio. But you have to bear in mind that 90+ percent of people interviewing don’t have one to offer.

Dave Bush
8 years ago
Reply to  Erik Dietrich

It is hard to have a portfolio when the code you write can’t walk out the door.

Jerome
Jerome
8 years ago
Reply to  Erik Dietrich

or they lie.
Honestly I see guys with blogs that are copy paste from other guys.
now they walk around like code guros that have a nice blog…

JohnLWebb
JohnLWebb
8 years ago
Reply to  Stefano Borini

Stefano I do understand your point (I was the same), but the problem is that most of the people will not have any code that they can show. This includes even the code that is made in private time (maybe he’s freelancing or has a commercial product, site, etc.). Also here are few situations that made me realize that coding tests are unavoidable even with a portfolio: – A person with a very clean and intuitive code in its repositories on a Bitbucket, it turns out he plagiarised a work from one of his acquaintances. – A person with a… Read more »

Kevin Mote
8 years ago

Great post. The next time I am interviewing a candidate, I will ask this question: “Tell me about the last time you DIDNT know how to solve a programming question, and what you did about it.”

Erik Dietrich
8 years ago
Reply to  Kevin Mote

I once heard an episode of Freakonomics where they talked about the tendency of some people to be completely unable/unwilling to admit a lack of knowledge about something. They proposed asking a question that was obviously unknowable just to see if candidates were the sort of people who would bluster/bluff or who were comfortable saying, “I don’t know.”

David S
David S
8 years ago

Another insightful post! Thank you! I have an unusual perspective on this one: I think I’m one of the few people in the world who genuinely enjoys job interviews (As the interviewee, not the interviewer!) I guess I’ve been to enough of them that I’ve gotten pretty good at them. It’s become an exciting challenge to present myself well, think fast for the trivia questions, and gain perspective into a new company. However…. I remain not entirely convinced that the typical software company’s job interview process adds any value over taking all the candidates they’ve decided to interview, and picking… Read more »

Erik Dietrich
8 years ago
Reply to  David S

Thanks for the kind words. While I can’t claim to enjoy job interviews, I do sort of understand what you mean about enjoying sort of a mastery of the game. I’ve written more extensively about the interview process in the book I’m working on, but the tl;dr version of that is to agree with you that interviewing is probably not significantly better than random selection (after some screening and resume fact-checking). Frankly, I think that the most effective ‘interview’ results from people on a team putting forth a person that one or more of them knows. The ‘interview’ is then… Read more »

123
123
8 years ago

what does ”fly someone out” mean?

Erik Dietrich
8 years ago
Reply to  123

I meant that the company was going to pay for my travel to interview with them. I was living near Chicago, and the job was in Boston.