Gauging talent is tricky. Solving how to separate the lemons from the peaches is a problem every company has to face when hiring new engineers. There are some common strategies used to navigate this problem and some lesser known ones. Having faced many of these exercises and even constructed them myself, here are a few things I’ve learnt.

There is a long lived debate in the industry about the best methods to assess technical candidates and consistency can differ between companies.

How it's done

The most common style of interview is a coding assessment where a candidate is given a simple problem and is asked to write the solution live or timed and then have their submission judged. This could be as simple as sorting a batch of blue and red balls to something tricker like finding the shortest sub array where the sum is k

Some companies try to be more pragmatic with their style of problems and attempt to reflect a task you’ll face on the job, and some just need to make sure you can actually solve something before moving on to a later stage. Another style of assessment is a code review exercise where there are some obvious flaws or bugs and they wish to see how you reason through a codebase to find these issues.

As the candidate becomes more senior the exercises tend to focus on higher level problems with a wider surface and several areas of ambiguity. System design interviews are common here and ask questions like; how would you design the twitter “for you” page, followed by questions of scale and reliability. A senior candidate can also face questions around soft skills such as dealing with a challenging colleague and covering topics such as failure and how you’d approach things if you had to do them again.

Finally an engineering candidate may face a take home project or home tasks that are to be submitted the following week.

There is on going debate about the merits of live coding vs take home task. Where those that face the take home say that without a concrete time limit this can be invasive to their personal time as these tasks can often involve more than an hours worth of work and these days claude can complete these tasks with little effort. Critics of live coding sessions say they are not a good emulation of a real day at work as you’re not watched like a hawk in the office and there can be a high degree of luck if you have been given a problem that you have faced in practice.

My opinion is that those that have strengths in one and not the other will always favour and make arguments for the thing they’re good at. In the past I’ve had a colleague complain an interview was too easy despite the metrics telling us that we were turning down 98% of applicants due to them failing to complete the task to our standard.

There are business realities and having the hardest exercise can filter good enough candidates and if you’re wanting to hire the top ten competitive programmers you will be paying a premium for their dedication when there’s a chance that their skill doesn’t lend itself to dealing with problems outside of their domain. You might benefit and afford having one of these engineers but probably not ten. You also need to reflect on some of the other biases that are affecting your pool of participants if you’re favouring the niches of competitive programmers.

Creating a task

Having done a fair share of take home tasks which cover developing a card processor, aggregating data from a zip file that doesn’t fit in memory, developing an api that aggregating exchange rates and providing analysis on US visa applicants I found the variety of problems and skills required with take home tasks to be exciting and more relevant to the role than most leetcode challenges.¹

I have also had the opportunity of writing a take home tasks for candidates which revealed two important lessons: learn how to provide feedback and assess how a candidate responds to feedback.

I learnt early on that feedback has to be very clear and unambiguous - especially in the case of turning down candidates.

On the list of assessment criteria I stated that if a candidate misinterprets rule A that this was fine to ignore as we are not assessing them on that rule but on how it combines with other rules. In a submission a candidate had made a mistake on unrelated rule B and I marked them down for not following instructions. Unfortunately I did not make it clear that I was referring to rule B and they attacked me with a long written email about rule A; the rule we were not fussed about interpretation.

In order to avoid this from happening again I made sure the notes for the assessment had the reminder to avoid being vague in feedback, try to be as clear as possible. When feedback is vague a candidate will always take the option that shows them in better light so be abundantly clear on the reason.

This is probably what leads companies to provide generic rejection emails as it allows them to avoid confrontation with a candidate, it also allows rejection that are purely a vibe check. Perfectly acceptable for the soulless business.

The other lesson I learnt from home tasks was how it can be used to assess the candidates reaction to feedback. After reviewing their task I’d bring it up in the subsequent interview discussing the pros and cons.

One candidate even pressed me on the validity on my feedback which allowed me to go into more detail and find a point of consensus with them. The reaction to the feedback by the candidate led us to a positive signal and we made sure we assessed all candidates on their reaction going forward.

The future

With Ai disrupting how much an engineer is able to achieve it doesnt undermine the value of the home task, I believe we can still use it to assess how a candidate responds to feedback and suggestion.

It is still a valuable signal if the applicant understands the system they have created and how their system might evolve, which can be tested by asking them simple questions about their submission. I might even go as far as suggesting something bad to see if they simply agree or push back. Similar to suggestions that might come from authority day to day. It seems with the adoption of ai engineers are happier than ever to accept changes than they used to be as these issues are dealt with a single prompt. We should still be able to recognise when an issue is not an issue and avoid codifying edge cases that are hallucinations by people.

There are some other styles of tasks that are slightly more experimental that I believe would lead to positive signals.

I remember being taught in school to read the entire test before starting. A teacher did this by providing a bunch of problems on a single two sided page where the last problem said “ignore all previous problems, sit quietly and you’ll pass with full marks”.

One day it would be interesting to provide a code base for a home task and specify a requirement for a feature. I’d like that requirement to already be implemented to understand if a candidate would blindly continue to implement something or submit the existing code base as task complete. Bonus points if they implement a test. Partial knowledge comes from many sources, one such source can be a PM or Sales, the basis of blindly following authority can lead to redundant work and more maintenance in the future. A candidate that closes the gap can be valuable.

A friend of mine qualified as a doctor and informed me of some of the tests they are faced with. There was one i found to be interesting and would consider giving software engineers. Have them walk into a room and sit at a table across from someone, instruct them that they need to convince that person to organise the shapes on their table to match the shapes on the candidate’s table. As soon as the candidate says “put the triangle in the top left” they should respond with “what’s a triangle”.

Not only do I believe that it can be a valuable assessment of how the candidate is able to walk through different levels of reasoning to find common ground it also captures the realities of working in cross functional teams with team mates of varying experience where questions like “we need to offer SS on the API” get met with “what’s SS?”

The interesting thing about the startup world is that progression is typically, 1. working on someone elses idea. 2. working on your own idea, 3 funding other people to come up with ideas. A common mistake for many of the YC college dropouts tend to be; build first, ask questions later. When you're YC and are hedging your bets for a slice in a few potentially great businesses paying for this education is cheap.

It is common for startups to run head first through the phase: build it and they will come.

we built a couple things that just no one wanted and then realized we're going about this the wrong way.

Christina Cacioppo (Vanta) 2025²

This mistake isnt limited to the young cohort of YC startups. This is a fallacy that cause many Software projects to fail, it is a lesson that all creators have to learn.

Software Engineers

Software engineers hit a maturing point when they make the shift away from accidental engineering to intent driven decision making.

Early in their career they tend to enjoy the exploration of new ideas and the novelty of new software in business. Often referred to as CV driven development. They dive head first into developing features or solving problems. This characteristic isnt inherently a problem when guided by engineers that have their fair share of war stories, however as every engineer comes to learn; diving head first into the code will lead you down a painful path.

Established engineers approach a problem with a focus on what the user or customer intends to do and avoid relying on codified solutions.

It is still easy for experienced engineers to fall into the trap, blinded by excitement or neglicting to push back through fear or through finding it easier to go with the flow than challenge and presenting an alternative.¹

Without establishing customer intentions we are building with hope as the dominant strategy and are leaving our product up to chance. Leading with intent helps to create things that have purpose.

The most noticable example that highlights this behavour is when an engineer manages to close out work without writing code or when they establish that work shouldnt be done until something fundamental is proven. This can be as simple as, "is anyone asking for this?".

The Users

Companies place a heavy emphasis on valuing the customer as a means to avoid repeating their early mistake. The truth is we shouldn't be valuing what they say but we must try to care about the same things they care about. We need to discover their intent.

Product market fit is about creating a product that serves the intention of a large enough market. Before we can create a product we need to figure out what the customer wants to do and then figure out how to serve them. We can't setup a pillow renting business before we have asked if anyone would actually rent a pillow.

The "Customer Obsession" mantra isn't just corporate shill and it is a common value among companies because it is easily forgotten.

If I'd asked customers what they wanted, they would have told me, 'A faster horse!

Henry Ford

Steve Jobs used this quote to highlight why giving the customer what they want doesn't work. This doesn't mean you shouldn't speak to customers, but you should figure out not what they want but what they want to do. In the Henry Ford example, they want a faster horse, but their underlying intention is to get from A to B at speed.

The user will tell you want they want but it is our job to find out what they need.

Another easy habbit to fall into is being quick to provide answers without the full picture. Your first solution is often never the best and you might find that the customer might have already tried it. Instead of providing the answer, our first step should be to question why they want the answer in the first place.

Trying

Figuring out how to address a customers need is hard work and I've found people tend to be alergic to hard work; often coming up with reasons why their market is build differently or how something so common as product market fit doesnt apply to them. The failure of startups is a tale as old as time and the "I'm not like everyone else" is probably the reason people fall prey to common pitfalls.

There is nothing that should be more exciting than a hard problem, hard problems will make most people give up. This makes solving it all the more enjoyable.

Humans and computers are really bad at being random. An article recently published by the FT examined how people behaved under seemingly random situations and how efficient computers are at predicting our decisions even when we aim to act as randomly as we can.

Computers are equally bad at acting randomly; yet this hasn't stopped us from attempting to develop algorithmic approaches to generating randomness. Random number generators have many applications, from generating worlds , running probabilistic simulations to cryptography.

Randomness can be split into two categories, True Randomness and Pseudo Randomness. True random may only appear in life and nature where it is impossible to predict the next state despite knowing the current state. On the other hand even though pseudo random appears unpredictable it can be replicated if all the starting conditions are known.

Quality of Randomness

There are several qualities to the algorithms that provide us randomness by which we can compare them. The first quality is the algorithm's "period", the number of time we can generate an output before the numbers start to repeat themselves. The second is the number of dimensions the algorithm can pass the spectral test.

In the 1940s von Neumann presented the middle-square method for computing random numbers and it worked as follows:

1. Take an n-digit number
2. Square it
3. Take the middle n digits from the result

The alternative was to find a random number from a book. Whilst there's no set period for the middle square method it is often short and unpredictable which proves to be inconvenient in most applications.

An improvement was made in the 1950s with the development of linear congruential generators. However these fall short on spectral tests which means they start to develop hyperplanes in larger dimensions (the English translation: you start to notice patterns in 3D).

There are a number of tests that are used to measure the quality of a random algorithm which have become known as the Diehard tests. Among them is a suite of tests called "Craps test" where the algorithm plays 200k games of craps and the count of wins and throws should follow a specific distribution.

Contemporary Randomness

All algorithms that generate random numbers start with what is known as a seed. The seed is a number that is used to provide an initial state that all future states are based on. This allows us to replicate random generation or seemingly random simulations when provided with the initial seed.

There are some situations where a seed is not explicitly given when generating something random - however typically your computer will use the current time as a substitute for the lack of explicit seed.

Most contemporary languages such as python use an algorithm called the "Primitive Twisted Generalized Feedback Shift Register Sequence" this is known more commonly as the Mersenne Twister. Due to the existence of a Mersenne prime constant in the algorithm. You can read the C implementation of MT in the python source.

A note on Mersenne primes: these are prime numbers that are (2^n) - 1. One less than a power of two. These primes happen to be the quickest to prove as prime and therefore make up all the largest known prime numbers.

The period of the Mersenne Twister is (2^19937), which is a number I don't have time to type out. 19937 happens to be that Mersenne prime number in the algorithm and the algorithm passes spectral tests up to 623 dimensions.

Cryptography

The realm of cryptography relies on crypto-secure pseudo random number generators (CSPRNG). These are used when generating a password or private/public keys for encryption like RSA. They are required to be both random and unpredictable. Unfortunately the Mersenne Twister isn't safe for cryptography since with enough observations it becomes predictable. As evident in the blog series: Cracking Random Number Generators

Over time we've seen the pseudo random algorithms required in cryptography being broken and methods discovered in order to predict the random number used in the system. Cloudflare proposed a solution to this, instead of having the CPU generate a number using a predetermined algorithm, have a wall of lava lamps and take a photo of the wall whenever you need a new number. This works because images are stored on a computer as numbers and there is naturally occurring randomness which is extremely hard to reverse engineer.

The first software book I bought was Effective Python by Brett Slatkin. The book distills tips and tricks learnt by Brett over the course of his career. It was in this book I first encountered how to use the enumerate keyword. Over the course of my career a new hint like this one has sparked joy and I've experienced first hand how pulling one of these out of the back pocket during a live interview can intrigue an interviewer.

One can speak a language their whole life and every so often come across a new word that you've not encountered before but it does a very good job of describing the current situation. The same happen in software.

Python is a language with a large standard library that offers many tools, so much so that you can even import antigravity. Here's a shortlist of some of the tricks you can do when you have to write python under pressure.

Modular Operator

Running into a situation where you need to cycle through indexes in an array the modular operator % is the tool of choice.

Using it we can have x cycle from 0 -> 5 and back to 0. It's application extends to explaining how hash look ups work in a key value store and providing an answer to a simple question like how much remains after dividing 5 by 3.

# x cycles from 0 -> 5 and back to 0.
x = 0
x = (x+1) % 6

# How much remains after dividing 5 by 3
>>> 5 % 3
2

Floor division

When we don't care about the remainder we can use a floor division to provide a nice round int. This answers the simple question, how many times can 4 fit into 11:

>>> 11 // 4
2

Divmod

When we can't remember if we should be using % or // in the middle of a live interview then we can use the builtin keyword divmod to give us both answers.

>>> divmod(6, 4)
(1, 2)

dict.setdefault

There are some tricky problems that want you to set a key if it doesn't exist in a map but if it already exists then avoid updating the map but ensure the value being set is the same as the value that has already been set. Obviously you can do this without setdefault.

x = {"foo": 4}

new_value = 10

old_value = x.get("foo")
if not old_value:
    x["foo"] = new_value
else:
    if old_value == new_value:
        raise

Here's how you can do it with setdefault.

x = {"foo": 4}

new_value = 10

old_value = x.setdefault("foo", new_value)
if old_value == new_value:
    raise

Setdefault will insert the key with the new value if the key isn't already in the dictionary. When it is already in the dictionary it returns that value otherwise it sets the provided value and also returns that value.

Greatest common denominator

If you need to perfectly tile a 21x35 rectangle with squares, what is the size of largest square that will cover the area of this rectangle?

The answer is the largest number that the two numbers, 21 and 35, can be divided by.

>>> import math
>>> math.gcd(21, 35)
7

Thus the largest size square is 7x7.

Prefix sum

Prefix sums is a common technique used to solve coding problems such as "Subarray Sum Equals k". A prefix sum at index i represents the sum of all items from 0 to i. We can create a prefix sum in python using itertools.

>>> import itertools
>>> list(itertools.accumulate([1, 3, 4, 3, 2]))
[1, 4, 8, 11, 13]

Defaultdict

Defaultdict is a classic tool which allows you to specify the default instantiation for a key. If you need to track a list of elements for specific keys you can instantiate the default dict with list. To avoid checking if a key already exists and creating a new list if not.

>>> from collections import defaultdict
>>> tracking = defaultdict(list)
>>> tracking["x"].append(1)
>>> tracking
{"x": [1]}

It also plays nicely with counting.

>>> from collections import defaultdict
>>> tracking = defaultdict(int)
>>> tracking["x"] += 1
>>> tracking
{"x": 1}

Heaps

Tracking key usage or needing a priority ordered queue will require using a min/max heap. Fortunately python offers methods that transforms lists into these heaps.

from heapq import heapify

queue = [3, 1, 2, 4]
heapify(queue)

Relying on heappush and heappop allow us to dequeue or enqueue items to our heap while maintaining priority order.

deque vs list

I've covered deque before in essence we can't always rely on the builtin list as they are dynamic arrays, we need to remove from the front and pop from the back in constant time. The deque is a builtin solution for linked lists.

from queue import deque

q = deque()
q.append(1)
q.appendleft(2)
q.pop()
1

Bisect

Another one already covered. This is Python's own implementation of binary search. If we are given a sorted array and wish to insert a new item while maintaining order we can use bisect_left.

from bisect import bisect_left

items = [1, 2, 4, 5, 5, 6]
bisect_left(items, 5)
3

Startups fail for the same reason you might. The typical failure case for startups include no market need and failing to pivot quick enough. These are also risks faced by larger companies but they're harder to notice. A typical solution from leadership is to remove processes and promote behaving "like a startup".

One does not throw process out the window in order to succeed. If this were the case; injecting chaos into a team would be the dominant strategy.

The key to running a successful product team is to establish a culture that seeks to aggressively shorten the time it takes for the team to make mistakes and learn. Shipping the wrong thing in one week is better than shipping the wrong thing in 6 months.

Long Term Risks

Long term plans lack the existence of genuine feedback, instead feedback is received on the idea in our head and how the idea is perceived in someone else's head. This misalignment is one of the risks we bake into the development cycle among others that aren't obvious when we have a long development horizon.

A plan is the route we take from the current state of the world A and the world we wish to exist B. When we commit to the plan we are hoping that this new world will be the place everyone wants to be. If 11/12 startups fail this indicates we are generally bad at envisioning futures.

Quarterly releases has the risk of assuming what is true today will still be still in three months and that everything will go according to plan during this time and not be derailed by something that takes priority.

If things do go according to plan and we get to release, the best case scenario is that the vision is only three months old. However in reality one learns, ideas mature and the state of the real world can change in a short time.

Long on-going projects are hard to back out of. Admitting defeat is hard especially if a competitor fails finding product market fit for the very next thing you're waiting to release. There's the sunk cost fallacy, political stake, emotions and the missed opportunities from having to sit down and crunch in the last minutes of release. Even having your head in the problem can help you realise that it might not work out and we have to ask ourselves if we've reached the point of no return.

A project that is under an on-going and long product iteration cycle needs to constantly reassure stakeholder that everything is alright and will be alright. Additionally there's the pressure being applied from shiny new ideas that have novelty appeal and are more exciting to talk about than the thing we've been actively working on for the last two months.¹

These are all systemic risks that are not built into the product but are built across the delivery process. A company that releases quarterly has a minimum reaction time of 3 months.

What can we do?

We have two goals. Find out what is valuable and minimize the time it takes to deliver that value. Ideas are cheap, customers pay you for what you release they don't pay for what you plan.

Learning comes from the user and you can skip discussions or second opinions. Getting something into the users hands is the opportunity to learn far more than you might from a peer. Spending 2 weeks to fail is far cheaper than spending 3 months and now you can take what you've learnt into the next cycle.

Successful startups are skipping advise from the experts and just seeing what works. Senior leadership can sometimes hold innovation back and a little naivety can go along way. We need maturity to say "I don't know" and bravery to say "Let just give it a go".

Identify the processes that are busy work, work that feels productive but slows down your ability to release. This could be planning meetings with teams that have nothing to do with your project.

Focus on development that gets you signals instead of releasing fluff. You don't need to release the entire picture, if you have the opportunity to release only one thing and that thing helps you determine if your vision is wrong, you shouldn't be building anything else.

Do some now

Delivering something now is better than delivering it all at later.

Don't delay learning, if you release a product to the world after months of working on it, you're going to be doing a lot of "learning" after that release. Only if it's not incredibly hard to separate that parts that are cruft and the parts that are valuable in this release - it is far easier to learn iteratively than it is to learn while you're putting out fires.

There are no rules, you are allowed to release something and have users find it by chance or selective introduction. Releasing a feature is different from announcing a feature. Some users might never come into contact with the feature, and those that find it might ignore it since it doesn't fit their use case. It is fine for things to not be a fit for all customers, but the users that find it's useful have had value delivered to them without you needing to complete the entire project. To these customers you've done this quicker than expected and now you're being paid earlier than expected.

Value can be delivered even if it requires manual work. Managing to automate 40% of the workload while requiring the rest to be fine-tuned from the user will be seen from the determined customer as saving them 40% of the time, they won't see the job as half done. Their concern is getting to market, they have their own competitors and you've just given them a 40% head start. Had you not released early, they would have to do the 40% themselves anyway.

Waiting to serve all use cases delay some of your customers when they could be using that time to make their money. Doing 20% of the work might be 10% more than a competitor, and releasing sooner positions you better and can de-risk churn.

Are your customers even asking for it? You can get through releasing 10% of the plan and suddenly all your existing customers adopt the feature without you needing to sink time into the other 90% of the work that you had assumed would be needed to convince them. Ditch the plan, ask them what's next.

If there's one truth or take away from this post; it's that the importance of something being done diminishes over time so the sooner something is delivered the higher it's overall impact. No matter how small.