At the moment, the IT industry is considered a pretty prestigious profession. It attracts lots and lots of people because the pay is good, the working conditions are good and the whole society is in awe of the masterful things programmers produce.

Potential workers are trying join the IT train in droves. So many have been joining us that a third of working developers only have <= 4 years of work experience (according to Stack Overflow surveys).

This could have been a golden opportunity for IT companies to test all kinds of interview approaches on a vast number of people and develop a data-backed super approach. One that could finally tell us what makes a good (co-)worker.

Instead, we are willingly standing still, voluntarily propping up this broken and deformed interview process. Why do we do this?

If we mostly agree that the interview process does not produce dreamy IT teams, then why are we sticking with this process?

The answer is simple: it produces “proper” IT teams. It does not create great teams that deliver great products or services or are pleasant to work with or are innovative or are prolific. But it does create teams that look as if they are good IT teams.

We are mostly concerned with copy-pasting what we consciously or subconsciously believe is a good IT team: young, white, male, dressed in hoodies, socially awkward, having gone to the same universities we went to, having the same beer preferences we have, have had the same life experiences that we had, … .

Why do many claim the interview process is broken?

The short answer: people, who consider themselves completely decent devs, feel left out.

Now, are they right or are they not?

That’s hard to judge when what we have are mostly just internet anecdotes. But, my bet is placed squarely on “yes, they are right”.

And my strongest argument is the statistical analysis of the IT population. Given the billions of people on earth, how can it be there is sooo little variance in the IT population? How is it possible that a group of devs can be spotted on the street by their looks alone? This shouldn’t be possible, if we really were a merit-based profession.

The perfectly competent devs feel they were rejected not because they had too little skill or talent, but because they didn’t know answers to questions, they felt weren’t truly relevant to the job posting. And I tend to agree with them.

From what I’ve seen and learned over the years, I have 0 trust in most questions that wind up being asked in interviews.

Obviously, companies are very different from each other and some do have adequate and successful interview processes, but those are the exception, not the rule.

Over the years, I have been on all sides of the interview process, I’ve been the interviewer, the interviewee, the observer, the interviewer-adviser, the interviewee-adviser, … . This does not make me an expert on interviews in any way, but I also can’t just ignore the patterns I’m seeing.

What’s the point of the questions

The worst failing I have noticed is that almost no interviewer I spoke to concerned themselves with the value of all the possible answers to their questions.

Things like: What am I even trying to learn about the interviewee with Question 43? Why am I asking this specific question and not some other? Does this question tell me anything useful about the potential hire, what is the big picture? What are all the possible “good” answers to this question 43, what are really “bad” ones and what does it mean if the interviewee answers something in between, does it mean anything at all?

Too many interviewers treat questions like a test: you get 10 questions, each is worth 10 points, you get a score according to how close you are to what I deem the most correct answer and you need 80 points to get to the next level.

I don’t want to say I never met anybody, who cared whether the questions they were asking actually gave them any useful information, it’s just that there are so few of them.

The obvious flaw in this approach is that all human knowledge is absolutely vast, even if you limit it to IT knowledge. Everybody will have gaping gaps in their knowledge.

You can always prove that somebody doesn’t know enough of something: just pick any niche topic, read about it for an hour and then ask about the details you’ve read. Works almost all the time. To make it work all the time, just pick 2 different niche topics and by the power of probability, you should be covered.

The interviewer will (and should) always know more about a question, because she had to look it up before asking and because she heard a lot of different answers from different people. She will always have the upper hand in this one niche topic. Which should prompt every interviewer to be wary of this bias. But most aren’t.

Even worse, many believe the topics have been chosen objectively.

“I’m searching for somebody, who is just like me”

In reality, the topics are mostly chosen poorly and randomly. More often than not they are chosen from the knowledge that the first interviewer happened to have. And what these topics are good at is finding other people, who are just like that first interviewer.

We are not searching for the best developer, we are searching for those, who feel and look and smell like that first person with hiring power in the company.

If you are not like that first interviewer, then the only other way to get hired is luck or networking. The “correct” answers to all interview topics can usually be obtained by having friends, who work there or are otherwise close to the company.

But last time I checked, to be friends with somebody, you have to have even more in common than just the knowledge of some IT topics.

Rejecting everybody, who is not deeply similar to the existing employees or does not have informal connections to your employees, helps to make the interview process feel broken.

Everybody can be an interviewer

Most companies further damage the process by treating it as something anybody can do, no training required.

Why is this belief so prevalent among us? We don’t believe that everybody can be a team lead or that everybody can give a good presentation, but we do believe that everybody should decide who gets a job. Heck, even Google recommends it:

Make interviewing everyone’s job

… Google wants interviewing to be part of every Googler’s job.

- Google Guide: Train your interviewers

To be honest, Google does add that they train their interviewers before they put them in charge of inspecting applicants.

…, interviewers learn about desired attributes for candidates and review the interview process as a whole. The more interviewers understand what they’re looking for and how it fits into the larger picture, the more effective they can be.

- Google Guide: Train your interviewers

But, I do wonder, how well they really train them. Everybody does not make a good interviewer. Being good at interviewing is a skill and like every other skill, you get better as you practice. But you will suck at it for a long long time, before becoming good at it. Imagine how good a piano student is after 1 month of practice or even after the 1st year of practice. So, how long should you actively practice interviewing, before you can be trusted with a job candidate?

Can you beat bias with bias?

I understand the appeal of making everybody participate in hiring. We are all biased to some degree. People like to hire other people if they “click” with them, which usually means that they have some deep-rooted similarities. A solution to achieve greater diversity of employees might then simply be to make the group of decision-makers very diverse.

If a group of people is diverse, then their biases should be just as diverse and the group together will make a better hiring apparatus.

Diversity is a goal worth pursuing because it gives you greater flexibility and stability. Just as a diverse microbiome in your gut gives you better health or a diverse portfolio of stocks gives your better wealth or a diverse group of friends gives you better social support.

But what gives us the confidence that a diverse group of interviewers has diverse biases? They might be different, but if they are very well assimilated into a cohesive community, then their biases will reflect the average bias of the community.

Just because people seem varied, doesn’t mean their biases are varied. A bias is something that we form from experience and from what we believe is expected of us. It is not as trivial to identify bias as most believe.

GapJumper is a company that tries to perform blind recruitment processes in the name of other companies. They give all applicants the same task and send those with the best results to the company with no identifying information.

In 2019 a company asked them to figure out why they had low promotion levels of female employees in some departments. It turned out that the problem wasn’t bias of those, who reviewed the work of women, but the bias of those, who assigned tasks. The women employees were reviewed favourably, the issue was simply that the tasks they got assigned were less important than the tasks assigned to a few men.

Our analysis of performance reviews found no bias during performance review. Issue was disproportionate assigning of high impact project to a few (male) employees.

- Field Report: Making Talent Decisions Less Biased II

Bias comes in many ways. If our interview process is consistently producing teams of workers that are unquestionably less diverse than the overall population, then we absolutely are excluding some people from our jobs.

Interviewing as a skill

Not everybody can be an interviewer. I very strongly agree with this statement by Aline Lerner, the founder of interviewing.io:

Being a good interviewer takes time and effort and a fundamental willingness to get out of autopilot and engage meaningfully with the other person.

- What do the best interviewers have in common? We looked at thousands of real interviews to find out.

Interviewing is a skill. If you want to be good at it, you have to learn, you have to grow and you have to practice.

But if you have to practice a lot, then you have to enjoy doing it. You have to enjoy helping strangers show off their skills, their personality, while under pressure, even if you don’t see yourself getting a beer with them afterward.

How many people do you know, who interview like that?

We don’t understand our profession, but we do believe the interview process is infallible

What does a programmer do?

Some of us see ourselves as artists, creating beautiful things from emptiness. Others say we are mathematicians, taking formulas and putting them together into new formulas. Still others say we are handypeople, similar to plumbers or carpenters, we go from house to house, doing essentially the same job in each one. Yet others say we are surgeons, writing instructions for aeroplanes and medical equipment, using our precision to make sure people live and don’t die.

Given that our field is highly respected, many are suffering from an inflated ego. And so, we like to compare ourselves to life-saving surgeons or misunderstood genius artists. And the interview setup is a pretty good stage for us to prove we are better than the young ones that are following us.

This can cause 2 problems:

  • We misjudge when picking interview questions.

    Our ideal programmer knows everything about sorting algorithms and bit manipulation, so we ask those questions. Forgetting that neither we nor any of our co-workers used this knowledge in the last 10 years.

  • We are suspicious of the candidate, erring rather on the side of reject.

    After all, the candidate has to prove herself worthy of this job opportunity. What we do is only for the anointed.

The research, however, shows that performance on technical interviews is highly volatile for almost all candidates.

interviewing.io provides mock interviews to their users, they can use the platform to practice interviewing or to practice being interviewed. Each person does multiple of these sessions, which allows the company to compare the results for each person.

They found that just 20% of their users consistently scored the same. The others did well on some interviews and poorly on others. In real life, you usually can’t afford to not be at your best all the time… always. Usually, if you flunk 1 interview step, you are out.

Their scores go from 1 to 4:

Many people who scored at least one 4 also scored at least one 2.

…a good amount of people who scored at least one 4 also scored at least one 1.

If we look at high performers (mean of 3.3 or higher), we still see a fair amount of variation.

- After a lot more data, technical interview performance really is kind of arbitrary. (Hover over the people icons to see their individual scores.)

We penalize those, who don’t know how to play the game

All of this builds up huge pressure on interviewees. Especially on 3 groups:

  • complete beginners, who are still outsiders to our community and don’t yet know how the game is played
  • underrepresented groups, who are more likely to be sidelined by the community and thus also don’t know how to play the game
  • senior devs, because they have to prove themselves with college knowledge as well as detailed accounts of real-life problem-solving experiences

Playing the game is hard work. But that is not the real problem, the real problem is withholding information.

The IT interviewing process is a game when it shouldn’t be.

This game has exact rules of behaviour and exact, unambiguous rules for winning. Proof for this are all the “Cracking the Coding Interview.”-books and courses, which will successfullly teach you how to win in interviews.

If you want to be a master interviewee, you have to look up certain topics, that are considered “basic IT knowledge”, approach problems in a constructive, “I fear you not”-way and you have to have an air around you that says “I have a wide range of experience”, “I have smart things to say about your product”, and of course you need a healthy amount of self-confidence, … . It’s like Tinder, but for jobs.

Nobody actually is all those things, but that’s not a problem, because the interview process is a lot about the looks.

As a complete beginner, I had no idea, there was a game, let alone what rules there were or what was expected of me. I didn’t know the common questions nor the “correct” answers.

And as I belonged to an under-represented group, my friends weren’t in IT and when we met, we would not talk about the latest Linux distro or an awesome dev blog post. It took time for me to figure all this out. But now, that I’m “in”, I can see that most of my colleagues are unaware of this advantage. Even the most welcoming, supportive people, including me sometimes.

It so happened that I once had an opportunity to work with somebody, who I rejected on the interview. And you know what? This person proved to be a good dev, who could do his job successfully. I misjudged him, even though I was actively trying not to 🙂.

Interviews pin your compensation options

I’ve also seen a very good dev, go to an interview, totally unprepared. The result was a job offer for a low position with small pay. interviewing.io also found a correlation between how good the interview goes and your eventual compensation (their data is for San Francisco):

Interview performance doesn’t just get you in the door or not: it can have a demonstrable connection to your eventual compensation. For instance, doing just a point better in your technical interview could be worth 10k or more, and with bonus, it could add 20k to your annual comp.

- There is a real connection between technical interview performance and salary.

The interview doesn’t just determine whether or not you get the job, it also determines what you are going to be paid for the foreseeable future.

I don’t know how it is where you live, but where I live, upwards mobility inside the same company is much more difficult than switching jobs and getting a better position and compensation at a different company.

We are all more sympathetic to people, who have had the same struggles in life as we did and less sympathetic to those who can’t overcome something foreign to us.

Recently an old colleague lamented how devs have become too expensive, but in the same breath said that he doesn’t really want to hire a dev, who doesn’t know what the current rate for a dev is. If you don’t know how much the top 10% of devs in our area is paid, then you probably also don’t know the latest programming approaches.

In my eyes, this principle just makes sure that people, who are on the outside, stay on the outside.

People like forgetting how they came to where they are. We all experience amnesia when it comes to remembering our past incompetence. And still, we are tasked with assessing a job applicant and deciding how much money they are worth.

Rather presumptuous of us. But it does explain why everybody wants to hire seniors because nobody remembers ever being a junior.

Malevolent forces of the “culture fit”

So, what’s the deal? If the interview process is broken, then why do we stick with it? Is it apathy, just plain laziness or is it an intentional conspiracy?

In my eyes, the answer is pretty simple: each system must be more beneficial than disadvantageous to still exist.

Lots of us are asking interview questions that are ineffective and are tormenting job applicants for months. But nobody is forcing us to do it. So, … we must be doing it because we believe this is the best way to figure out who to hire.

The sad truth we don’t realize is, however, that we are perpetuating old wrongs, old biases by making sure the system only allows very limited types of people to pass.

If we were filtering by job-relevant skills, the approach would work. But we don’t.

We filter by what a programmer should feel and look and smell like and most often than not, they should feel and look and smell like our boss and our colleagues and us.

How to objectively score job applicants?

You might think: let’s eliminate the bias by creating an objective scoring system. But what is an objective scoring system, if we have no objective units of measure?

We can’t express a candidate’s Python knowledge in meters or seconds or pascals. Those were examples of objective units. Whereas measuring a candidate’s communication skill on a scale of 1 - 4 is neither accurate nor objective.

Neither the process nor the score can be standardized. The process of measuring the communication skills will differ from candidate to candidate. Just as the scoring will be highly subjective and tied to the current mood of the interviewer.

Triplebyte is an IT job search platform that also tests, measures and recommends devs to companies. They too started out believing that comparing devs amongst themselves is going to be straightforward: just pick the right parameters and score devs on those parameters. But they soon found that there is little objectivity in job interviews:

One of the first people to pass our process really impressed us. He was a superb, intelligent programmer. He solved hard algorithm problems like they were nothing, and understood JavaScript deeply. We introduced him to a company he was excited about, and sat back to watch him get a job. We were startled when he failed his first interview. The company told us they valued process more than raw ability, and he’d not written tests during the interview. He went on to get a bunch of offers from other companies, and one founder told us he was among the best programmers they had ever interviewed.

This lack of agreement is the rule, not the exception. Almost no one passes all their programming interviews.

- Who Y Combinator companies want

The underlying problem is that no objective parameters for dev scoring exist.

It doesn’t matter if we rely on a point system or interviewers’ guts, both approaches originate in the same thing: the interviewer’s impression of the applicant. How close the applicant was to what the interviewer thinks the applicant should be.

The “Princess on the pea”-test

Sadly, the heterogeneity among software engineers is so small that you can cover the vast majority with this description: white, male, aged 25-35, wearing a hoody.

Add to this the cultural stereotypes, we kinda, truly still believe: that the best devs are socially awkward, have no children, know all the trivia of the IT world, are obsessed with their coding to the point of coding all the time and most importantly are born this way.

If somebody does not exhibit at least some of these characteristics, then they are just not a true developer.

We are doing the “Princess on the pea”-test on job applicants. Here’s how the fairytale starts, I’ve replaced “princess” with “developer”. What do you think? Does it fit our reality?

Once upon a time there was a company that wanted to find a developer; but he would have to be a real developer. They looked all over the world to find one, but nowhere could they get what they wanted. There were developers enough, but it was difficult to find out whether they were real ones. There was always something about them that was not as it should be. So they were sad, for they would have liked very much to have a real developer.

- Hans Christian Andersen: The Princess and the Pea

Triplebyte also found that what you feel and look and smell like counts a lot. If there ever was an honest description of the hiring favouritism in IT I found it here:

But company recruiter decisions are driven largely by pattern matching, so there is a strong argument that making yourself look like candidates who companies want will increase your pass rate. …

Reading bios of founders and applying to companies where the CTO shares your background is probably an effective job-search strategy.

- Who Y Combinator companies want

What are we losing, by being snobbish

Ok, so our interview processes are killing diversity. So what?

If the best of the best are all young white man, who all know each other because they all went to the same school and worked at the same couple of companies and are all mostly almost the same age and all meet at the same meetups and talk at the same conferences and have the same interests and … What are the odds of that being true? Zero.

A political system, where all the control is in the hands of a small group of people, who all know each other and all socialize with each other, is called an oligarchy.

By definition oligarchy does not place the best people into the most important positions, because it doesn’t have access to the best people, its pool of possible candidates is diminished to the extreme because only members of the oligarchy can be given positions.

Instead of picking the best and brightest of all, who were born, they are forced to pick whoever is best amongst them. Preferably they would like to give all the oligarchy members important positions, but competence is always equally distributed in a society, so they can’t all be better than all the rest of the population.

That is what we are losing. We are losing innovation and richness of taste. It is not just the underrepresented groups that are being left behind, but everybody, who doesn’t want or can pretend to be the best thing that ever happened to development.

We are losing new ideas, better approaches, we could already have solutions to a million still unsolved problems, but we don’t, because we want to handpick the people, who will be allowed to work on them.

So,… if you are hiring, look at your already hired employees.

Did you succeed in bringing in a varied team of people or are they pretty uniform?

Think about questioning your interview process. Are you really asking things that you want to know?

Is it equally likely that all your job applicants would know how to correctly answer your questions? Or do some have advantages - informal because they are good friends with your employees or formal because they went to the same meetup?

Are you sure you are not missing out on somebody talented, just because they don’t fit your mental image of a dev?

And if you are searching for a job, just treat it like Tinder. Figure out what type of company you wish to work for then figure out what type of dev they are looking for and try to show that you have those very specific characteristics.

And try not to feel jaded, that the system is as shitty as it is. But the best jobs are usually still best accessible via acquaintances.

And who knows, maybe we will one day live in a society that will give everybody a chance.

External sources