Gauging talent is tricky. Solving how to separate the lemons from the peaches is a problem every company has to face when hiring new engineers. There are some common strategies used to navigate this problem and some lesser known ones. Having faced many of these exercises and even constructed them myself, here are a few things I’ve learnt.

There is a long lived debate in the industry about the best methods to assess technical candidates and consistency can differ between companies.

How it's done

The most common style of interview is a coding assessment where a candidate is given a simple problem and is asked to write the solution live or timed and then have their submission judged. This could be as simple as sorting a batch of blue and red balls to something tricker like finding the shortest sub array where the sum is k

Some companies try to be more pragmatic with their style of problems and attempt to reflect a task you’ll face on the job, and some just need to make sure you can actually solve something before moving on to a later stage. Another style of assessment is a code review exercise where there are some obvious flaws or bugs and they wish to see how you reason through a codebase to find these issues.

As the candidate becomes more senior the exercises tend to focus on higher level problems with a wider surface and several areas of ambiguity. System design interviews are common here and ask questions like; how would you design the twitter “for you” page, followed by questions of scale and reliability. A senior candidate can also face questions around soft skills such as dealing with a challenging colleague and covering topics such as failure and how you’d approach things if you had to do them again.

Finally an engineering candidate may face a take home project or home tasks that are to be submitted the following week.

There is on going debate about the merits of live coding vs take home task. Where those that face the take home say that without a concrete time limit this can be invasive to their personal time as these tasks can often involve more than an hours worth of work and these days claude can complete these tasks with little effort. Critics of live coding sessions say they are not a good emulation of a real day at work as you’re not watched like a hawk in the office and there can be a high degree of luck if you have been given a problem that you have faced in practice.

My opinion is that those that have strengths in one and not the other will always favour and make arguments for the thing they’re good at. In the past I’ve had a colleague complain an interview was too easy despite the metrics telling us that we were turning down 98% of applicants due to them failing to complete the task to our standard.

There are business realities and having the hardest exercise can filter good enough candidates and if you’re wanting to hire the top ten competitive programmers you will be paying a premium for their dedication when there’s a chance that their skill doesn’t lend itself to dealing with problems outside of their domain. You might benefit and afford having one of these engineers but probably not ten. You also need to reflect on some of the other biases that are affecting your pool of participants if you’re favouring the niches of competitive programmers.

Creating a task

Having done a fair share of take home tasks which cover developing a card processor, aggregating data from a zip file that doesn’t fit in memory, developing an api that aggregating exchange rates and providing analysis on US visa applicants I found the variety of problems and skills required with take home tasks to be exciting and more relevant to the role than most leetcode challenges.¹

I have also had the opportunity of writing a take home tasks for candidates which revealed two important lessons: learn how to provide feedback and assess how a candidate responds to feedback.

I learnt early on that feedback has to be very clear and unambiguous - especially in the case of turning down candidates.

On the list of assessment criteria I stated that if a candidate misinterprets rule A that this was fine to ignore as we are not assessing them on that rule but on how it combines with other rules. In a submission a candidate had made a mistake on unrelated rule B and I marked them down for not following instructions. Unfortunately I did not make it clear that I was referring to rule B and they attacked me with a long written email about rule A; the rule we were not fussed about interpretation.

In order to avoid this from happening again I made sure the notes for the assessment had the reminder to avoid being vague in feedback, try to be as clear as possible. When feedback is vague a candidate will always take the option that shows them in better light so be abundantly clear on the reason.

This is probably what leads companies to provide generic rejection emails as it allows them to avoid confrontation with a candidate, it also allows rejection that are purely a vibe check. Perfectly acceptable for the soulless business.

The other lesson I learnt from home tasks was how it can be used to assess the candidates reaction to feedback. After reviewing their task I’d bring it up in the subsequent interview discussing the pros and cons.

One candidate even pressed me on the validity on my feedback which allowed me to go into more detail and find a point of consensus with them. The reaction to the feedback by the candidate led us to a positive signal and we made sure we assessed all candidates on their reaction going forward.

The future

With Ai disrupting how much an engineer is able to achieve it doesnt undermine the value of the home task, I believe we can still use it to assess how a candidate responds to feedback and suggestion.

It is still a valuable signal if the applicant understands the system they have created and how their system might evolve, which can be tested by asking them simple questions about their submission. I might even go as far as suggesting something bad to see if they simply agree or push back. Similar to suggestions that might come from authority day to day. It seems with the adoption of ai engineers are happier than ever to accept changes than they used to be as these issues are dealt with a single prompt. We should still be able to recognise when an issue is not an issue and avoid codifying edge cases that are hallucinations by people.

There are some other styles of tasks that are slightly more experimental that I believe would lead to positive signals.

I remember being taught in school to read the entire test before starting. A teacher did this by providing a bunch of problems on a single two sided page where the last problem said “ignore all previous problems, sit quietly and you’ll pass with full marks”.

One day it would be interesting to provide a code base for a home task and specify a requirement for a feature. I’d like that requirement to already be implemented to understand if a candidate would blindly continue to implement something or submit the existing code base as task complete. Bonus points if they implement a test. Partial knowledge comes from many sources, one such source can be a PM or Sales, the basis of blindly following authority can lead to redundant work and more maintenance in the future. A candidate that closes the gap can be valuable.

A friend of mine qualified as a doctor and informed me of some of the tests they are faced with. There was one i found to be interesting and would consider giving software engineers. Have them walk into a room and sit at a table across from someone, instruct them that they need to convince that person to organise the shapes on their table to match the shapes on the candidate’s table. As soon as the candidate says “put the triangle in the top left” they should respond with “what’s a triangle”.

Not only do I believe that it can be a valuable assessment of how the candidate is able to walk through different levels of reasoning to find common ground it also captures the realities of working in cross functional teams with team mates of varying experience where questions like “we need to offer SS on the API” get met with “what’s SS?”

The interesting thing about the startup world is that progression is typically, 1. working on someone elses idea. 2. working on your own idea, 3 funding other people to come up with ideas. A common mistake for many of the YC college dropouts tend to be; build first, ask questions later. When you're YC and are hedging your bets for a slice in a few potentially great businesses paying for this education is cheap.

It is common for startups to run head first through the phase: build it and they will come.

we built a couple things that just no one wanted and then realized we're going about this the wrong way.

Christina Cacioppo (Vanta) 2025²

This mistake isnt limited to the young cohort of YC startups. This is a fallacy that cause many Software projects to fail, it is a lesson that all creators have to learn.

Software Engineers

Software engineers hit a maturing point when they make the shift away from accidental engineering to intent driven decision making.

Early in their career they tend to enjoy the exploration of new ideas and the novelty of new software in business. Often referred to as CV driven development. They dive head first into developing features or solving problems. This characteristic isnt inherently a problem when guided by engineers that have their fair share of war stories, however as every engineer comes to learn; diving head first into the code will lead you down a painful path.

Established engineers approach a problem with a focus on what the user or customer intends to do and avoid relying on codified solutions.

It is still easy for experienced engineers to fall into the trap, blinded by excitement or neglicting to push back through fear or through finding it easier to go with the flow than challenge and presenting an alternative.¹

Without establishing customer intentions we are building with hope as the dominant strategy and are leaving our product up to chance. Leading with intent helps to create things that have purpose.

The most noticable example that highlights this behavour is when an engineer manages to close out work without writing code or when they establish that work shouldnt be done until something fundamental is proven. This can be as simple as, "is anyone asking for this?".

The Users

Companies place a heavy emphasis on valuing the customer as a means to avoid repeating their early mistake. The truth is we shouldn't be valuing what they say but we must try to care about the same things they care about. We need to discover their intent.

Product market fit is about creating a product that serves the intention of a large enough market. Before we can create a product we need to figure out what the customer wants to do and then figure out how to serve them. We can't setup a pillow renting business before we have asked if anyone would actually rent a pillow.

The "Customer Obsession" mantra isn't just corporate shill and it is a common value among companies because it is easily forgotten.

If I'd asked customers what they wanted, they would have told me, 'A faster horse!

Henry Ford

Steve Jobs used this quote to highlight why giving the customer what they want doesn't work. This doesn't mean you shouldn't speak to customers, but you should figure out not what they want but what they want to do. In the Henry Ford example, they want a faster horse, but their underlying intention is to get from A to B at speed.

The user will tell you want they want but it is our job to find out what they need.

Another easy habbit to fall into is being quick to provide answers without the full picture. Your first solution is often never the best and you might find that the customer might have already tried it. Instead of providing the answer, our first step should be to question why they want the answer in the first place.

Trying

Figuring out how to address a customers need is hard work and I've found people tend to be alergic to hard work; often coming up with reasons why their market is build differently or how something so common as product market fit doesnt apply to them. The failure of startups is a tale as old as time and the "I'm not like everyone else" is probably the reason people fall prey to common pitfalls.

There is nothing that should be more exciting than a hard problem, hard problems will make most people give up. This makes solving it all the more enjoyable.

Software is planning, and as the adage goes, failing to plan is planning to fail. This article goes into the development of Git and UNIX with the goal of dispelling the myth that the genius software engineer exists and when we hear something was written entirely over night there's key information being omitted.

Everyone likes the story of the programmer that disappears and resurfaces after a week with some revolutionising software. We all enjoy a compelling narrative but in reality great software takes time, additionally for most great projects the code is actually the smallest part of the project.

Planning

Projects don't go wrong they start wrong

How Big Things Get Done (2023)

We shouldn't view software creation from the point when coding starts. The process should include the upfront planning. In 1975 Brooke advocated that the coding portion of a software project should amount to 17% of the overall time and it should be noted that in the 1970s the most dominant programming languages were FORTRAN and COBOL.

If a software requirements error is detected and corrected during the plans and requirements phase, its correction is a relatively simple matter of updating the requirements specification.

Software Engineering Economics (1981)

When we think of planning we often think of planning permissions, regulation and endless bureaucracy. We don't have time for that, we are trying to move fast and break things. However, in software, the planning stage is usually when most of the thinking happens and the hard problems get ironed out. Thinking is hard and most people avoid it, make thinking easier and you can stand out from other developers.

The planning phase of a project is also the point in time when large changes are the cheapest.¹

UNIX

One fable within software is that UNIX was developed over the weekend by Ken Thompson. Ken is quite the mythical figure in the software world, Brian Kernighan attests to this in multiple interviews.

We must keep in mind though that Ken worked for three years on Multics which he carried many features forward to UNIX. Here's not saying Ken isn't a great programmer, but we shouldn't discount that his mind was focused on an operating system for quite an amount of time and probably made some contribution towards formulating a better system.

Imagine your best programmer said to you, "sure let me work on the project for two to three years then we will scrap it; and start again. You are bound to have the best in class software." Instead businesses want every project to have a fabled programmer and to continue the trope.

Ken Thompson developed UNIX over 3 weeks, there's a video where Brian Kernighan remarks that modern software engineers aren't as productive².

Git

At this point Git is the most popular form of version control for software, it's also famously known to have been written in a fairly short amount of time. Linus states that he wrote Git in 10 days and we mostly assume the timeline for the project was - I have an idea and ten days later we have Git.

However this is not the case. Linus wrote Git in 10 days but it took him 4 month of thinking about the problem until he had a solution that he was satisfied with. 10 days after that we had Git.

Being a Creator

When it comes to creation, sometimes we are too excited by the answer that we fail to think about the question. A lot of software gets written without getting to the root of the problem and thus fail to materialise any value.

Large projects fail when creation comes first and planning comes never. Even after you have complete knowledge of the problem, solutions may have challenges which require addressing upfront, addressing them when you are midway through a project is going to cost both time and money.

For some problems you can have an attitude of "we will work it out when we get there", but you don't want to have this attitude to all problems as when you get there you might be 95% of the way through your time and budget and you have crossed the point of no return and getting over this hurdle requires going over budget.

Use your planning to categorise problems into "it won't work unless we know this" and "we can figure this out when we get there".

Board Games

"Is it fun" is the most important question in board game design. Ideally you should find this out before you hire an artist, write storyboards or even print and design cards. In board game creation circles they advocate tearing up pieces of paper and scribbling out a prototype to answer this question. The same thing applies to software.

Determine if the question you are answering is the correct one and determine if your software will actual solve the problem, do this; Wizard of Oz style, before putting a shovel into the ground.

Assumptions

All projects start with assumptions. These are typically the first things that need to be addressed before you start coding. Sometimes asking someone if a solution exists or how something works can knock off some unknowns and you'll be in a better position to start than had you remained silent.

Design it Twice

A Philosophy of Software Design (2018)

John Ousterhout advocates that designing software once is not enough and when you are planning something out you should make multiple attempts at it before you move ahead. He's found this leads to better design.

Every project comes with its unknowns and challenges, we are probably never going to rebuild the exact same solution twice. So packing in as much learning as you can upfront about the challenges ahead will leave you at least the most prepared you could have been.

The beginning of project is the cheapest time to learn so we should maximise and front load learning. Learning midway through a project is an expensive way of learning. It's fine if we can afford these learnings but on big and critical projects you don't want to bring up the point that the work we've been doing for the last 6 months was all for nothing especially if we could have learnt this earlier on.

Break things down, explore and figure out how each piece of the project will work and that we are actually providing the right answer to the correct question, this is the only way to deliver on time and on budget.

If we jump straight into code our design strategy is hope.

This is now my third article on lists. As someone that uses the built-in python list on a fairly regular basis, I might have built up a false sense of security. I'm pretty familiar with these listy-boys. However, recently I found out that I was not thinking about them correctly. Readers might smack themselves if they're familiar with data-structures but don't know how lists are implemented internally. The built-in lists are dynamic arrays.

How else could they optimise a sweet O(1) lookup time on indexing: mylist[4]. Especially when analysts are trying to avoid the built-in iterator and cursing their code with: for i in len(mylist): mylist[i].

Another trait an established data-structurer with be familiar with when it comes to dynamic arrays is that the append and pop methods are an amortised O(1). Amortised; because occasionally you have to suffer a cost of realloc(ating) memory across larger arrays.

Where the list starts to suffer is from poping and inserting at arbitrary positions.

Linked-List

The data-structurer will have had the linked-list slammed into their head often enough that it will pain them to hear about it again. So theory aside, I'll give you that sweet O(1) append and last item pop that you expect from a performant Stack.

Python deque provides a comparatively larger performance hit on initialisation to list and has poor O(n) performance when you want any arbitrary item somewhere in the middle. It does, however, have O(1); popleft, pop, append and appendleft. Due to being a doubly-linked list (or double-ended queue to get the abbreviation deque)

Deque in the wild

I saw a nice little quote from an enginneer on Quora:

In 8 years of getting paid to write computer programs, this post is the only time I’ve typed ‘deque.’

There are many places deque is used in the stdlib, most commonly whenever someone needs a queue or stack such as constructing a traceback, parsing python's sytax tree and keeping track of context scope.

My little run-in with deque was using it instead of a recursive function to avoid python's

maximum recursion depth exceeded

This limit happens to be set to 10^4. The solution was to add child nodes to a deque and when you were done with analysing the current node, popleft the next node.

Python Queue

You might be tempted to ask, well if deque is for queues. What on earth is from queue import Queue.

These queues are different (although, still using deque under the hood). They are optimised for communication across threads, which need to involve locking mechanisms and support methods like put_nowait() and join(). These are not intended to be used as a collective data-structure, hence the lack of support for the in operator.

More information

There is some neat documentation in the cpython repo which contains more data-structures and other alternatives to the standard built-in list. Tools for working with lists

References

• How are lists implemented:
• https://stackoverflow.com/a/15121933/3407256
• https://stackoverflow.com/a/23487658/3407256