Gauging talent is tricky. Solving how to separate the lemons from the peaches is a problem every company has to face when hiring new engineers. There are some common strategies used to navigate this problem and some lesser known ones. Having faced many of these exercises and even constructed them myself, here are a few things I’ve learnt.

There is a long lived debate in the industry about the best methods to assess technical candidates and consistency can differ between companies.

How it's done

The most common style of interview is a coding assessment where a candidate is given a simple problem and is asked to write the solution live or timed and then have their submission judged. This could be as simple as sorting a batch of blue and red balls to something tricker like finding the shortest sub array where the sum is k

Some companies try to be more pragmatic with their style of problems and attempt to reflect a task you’ll face on the job, and some just need to make sure you can actually solve something before moving on to a later stage. Another style of assessment is a code review exercise where there are some obvious flaws or bugs and they wish to see how you reason through a codebase to find these issues.

As the candidate becomes more senior the exercises tend to focus on higher level problems with a wider surface and several areas of ambiguity. System design interviews are common here and ask questions like; how would you design the twitter “for you” page, followed by questions of scale and reliability. A senior candidate can also face questions around soft skills such as dealing with a challenging colleague and covering topics such as failure and how you’d approach things if you had to do them again.

Finally an engineering candidate may face a take home project or home tasks that are to be submitted the following week.

There is on going debate about the merits of live coding vs take home task. Where those that face the take home say that without a concrete time limit this can be invasive to their personal time as these tasks can often involve more than an hours worth of work and these days claude can complete these tasks with little effort. Critics of live coding sessions say they are not a good emulation of a real day at work as you’re not watched like a hawk in the office and there can be a high degree of luck if you have been given a problem that you have faced in practice.

My opinion is that those that have strengths in one and not the other will always favour and make arguments for the thing they’re good at. In the past I’ve had a colleague complain an interview was too easy despite the metrics telling us that we were turning down 98% of applicants due to them failing to complete the task to our standard.

There are business realities and having the hardest exercise can filter good enough candidates and if you’re wanting to hire the top ten competitive programmers you will be paying a premium for their dedication when there’s a chance that their skill doesn’t lend itself to dealing with problems outside of their domain. You might benefit and afford having one of these engineers but probably not ten. You also need to reflect on some of the other biases that are affecting your pool of participants if you’re favouring the niches of competitive programmers.

Creating a task

Having done a fair share of take home tasks which cover developing a card processor, aggregating data from a zip file that doesn’t fit in memory, developing an api that aggregating exchange rates and providing analysis on US visa applicants I found the variety of problems and skills required with take home tasks to be exciting and more relevant to the role than most leetcode challenges.¹

I have also had the opportunity of writing a take home tasks for candidates which revealed two important lessons: learn how to provide feedback and assess how a candidate responds to feedback.

I learnt early on that feedback has to be very clear and unambiguous - especially in the case of turning down candidates.

On the list of assessment criteria I stated that if a candidate misinterprets rule A that this was fine to ignore as we are not assessing them on that rule but on how it combines with other rules. In a submission a candidate had made a mistake on unrelated rule B and I marked them down for not following instructions. Unfortunately I did not make it clear that I was referring to rule B and they attacked me with a long written email about rule A; the rule we were not fussed about interpretation.

In order to avoid this from happening again I made sure the notes for the assessment had the reminder to avoid being vague in feedback, try to be as clear as possible. When feedback is vague a candidate will always take the option that shows them in better light so be abundantly clear on the reason.

This is probably what leads companies to provide generic rejection emails as it allows them to avoid confrontation with a candidate, it also allows rejection that are purely a vibe check. Perfectly acceptable for the soulless business.

The other lesson I learnt from home tasks was how it can be used to assess the candidates reaction to feedback. After reviewing their task I’d bring it up in the subsequent interview discussing the pros and cons.

One candidate even pressed me on the validity on my feedback which allowed me to go into more detail and find a point of consensus with them. The reaction to the feedback by the candidate led us to a positive signal and we made sure we assessed all candidates on their reaction going forward.

The future

With Ai disrupting how much an engineer is able to achieve it doesnt undermine the value of the home task, I believe we can still use it to assess how a candidate responds to feedback and suggestion.

It is still a valuable signal if the applicant understands the system they have created and how their system might evolve, which can be tested by asking them simple questions about their submission. I might even go as far as suggesting something bad to see if they simply agree or push back. Similar to suggestions that might come from authority day to day. It seems with the adoption of ai engineers are happier than ever to accept changes than they used to be as these issues are dealt with a single prompt. We should still be able to recognise when an issue is not an issue and avoid codifying edge cases that are hallucinations by people.

There are some other styles of tasks that are slightly more experimental that I believe would lead to positive signals.

I remember being taught in school to read the entire test before starting. A teacher did this by providing a bunch of problems on a single two sided page where the last problem said “ignore all previous problems, sit quietly and you’ll pass with full marks”.

One day it would be interesting to provide a code base for a home task and specify a requirement for a feature. I’d like that requirement to already be implemented to understand if a candidate would blindly continue to implement something or submit the existing code base as task complete. Bonus points if they implement a test. Partial knowledge comes from many sources, one such source can be a PM or Sales, the basis of blindly following authority can lead to redundant work and more maintenance in the future. A candidate that closes the gap can be valuable.

A friend of mine qualified as a doctor and informed me of some of the tests they are faced with. There was one i found to be interesting and would consider giving software engineers. Have them walk into a room and sit at a table across from someone, instruct them that they need to convince that person to organise the shapes on their table to match the shapes on the candidate’s table. As soon as the candidate says “put the triangle in the top left” they should respond with “what’s a triangle”.

Not only do I believe that it can be a valuable assessment of how the candidate is able to walk through different levels of reasoning to find common ground it also captures the realities of working in cross functional teams with team mates of varying experience where questions like “we need to offer SS on the API” get met with “what’s SS?”