Gauging talent is tricky. Solving how to separate the lemons from the peaches is a problem every company has to face when hiring new engineers. There are some common strategies used to navigate this problem and some lesser known ones. Having faced many of these exercises and even constructed them myself, here are a few things I’ve learnt.

There is a long lived debate in the industry about the best methods to assess technical candidates and consistency can differ between companies.

How it's done

The most common style of interview is a coding assessment where a candidate is given a simple problem and is asked to write the solution live or timed and then have their submission judged. This could be as simple as sorting a batch of blue and red balls to something tricker like finding the shortest sub array where the sum is k

Some companies try to be more pragmatic with their style of problems and attempt to reflect a task you’ll face on the job, and some just need to make sure you can actually solve something before moving on to a later stage. Another style of assessment is a code review exercise where there are some obvious flaws or bugs and they wish to see how you reason through a codebase to find these issues.

As the candidate becomes more senior the exercises tend to focus on higher level problems with a wider surface and several areas of ambiguity. System design interviews are common here and ask questions like; how would you design the twitter “for you” page, followed by questions of scale and reliability. A senior candidate can also face questions around soft skills such as dealing with a challenging colleague and covering topics such as failure and how you’d approach things if you had to do them again.

Finally an engineering candidate may face a take home project or home tasks that are to be submitted the following week.

There is on going debate about the merits of live coding vs take home task. Where those that face the take home say that without a concrete time limit this can be invasive to their personal time as these tasks can often involve more than an hours worth of work and these days claude can complete these tasks with little effort. Critics of live coding sessions say they are not a good emulation of a real day at work as you’re not watched like a hawk in the office and there can be a high degree of luck if you have been given a problem that you have faced in practice.

My opinion is that those that have strengths in one and not the other will always favour and make arguments for the thing they’re good at. In the past I’ve had a colleague complain an interview was too easy despite the metrics telling us that we were turning down 98% of applicants due to them failing to complete the task to our standard.

There are business realities and having the hardest exercise can filter good enough candidates and if you’re wanting to hire the top ten competitive programmers you will be paying a premium for their dedication when there’s a chance that their skill doesn’t lend itself to dealing with problems outside of their domain. You might benefit and afford having one of these engineers but probably not ten. You also need to reflect on some of the other biases that are affecting your pool of participants if you’re favouring the niches of competitive programmers.

Creating a task

Having done a fair share of take home tasks which cover developing a card processor, aggregating data from a zip file that doesn’t fit in memory, developing an api that aggregating exchange rates and providing analysis on US visa applicants I found the variety of problems and skills required with take home tasks to be exciting and more relevant to the role than most leetcode challenges.¹

I have also had the opportunity of writing a take home tasks for candidates which revealed two important lessons: learn how to provide feedback and assess how a candidate responds to feedback.

I learnt early on that feedback has to be very clear and unambiguous - especially in the case of turning down candidates.

On the list of assessment criteria I stated that if a candidate misinterprets rule A that this was fine to ignore as we are not assessing them on that rule but on how it combines with other rules. In a submission a candidate had made a mistake on unrelated rule B and I marked them down for not following instructions. Unfortunately I did not make it clear that I was referring to rule B and they attacked me with a long written email about rule A; the rule we were not fussed about interpretation.

In order to avoid this from happening again I made sure the notes for the assessment had the reminder to avoid being vague in feedback, try to be as clear as possible. When feedback is vague a candidate will always take the option that shows them in better light so be abundantly clear on the reason.

This is probably what leads companies to provide generic rejection emails as it allows them to avoid confrontation with a candidate, it also allows rejection that are purely a vibe check. Perfectly acceptable for the soulless business.

The other lesson I learnt from home tasks was how it can be used to assess the candidates reaction to feedback. After reviewing their task I’d bring it up in the subsequent interview discussing the pros and cons.

One candidate even pressed me on the validity on my feedback which allowed me to go into more detail and find a point of consensus with them. The reaction to the feedback by the candidate led us to a positive signal and we made sure we assessed all candidates on their reaction going forward.

The future

With Ai disrupting how much an engineer is able to achieve it doesnt undermine the value of the home task, I believe we can still use it to assess how a candidate responds to feedback and suggestion.

It is still a valuable signal if the applicant understands the system they have created and how their system might evolve, which can be tested by asking them simple questions about their submission. I might even go as far as suggesting something bad to see if they simply agree or push back. Similar to suggestions that might come from authority day to day. It seems with the adoption of ai engineers are happier than ever to accept changes than they used to be as these issues are dealt with a single prompt. We should still be able to recognise when an issue is not an issue and avoid codifying edge cases that are hallucinations by people.

There are some other styles of tasks that are slightly more experimental that I believe would lead to positive signals.

I remember being taught in school to read the entire test before starting. A teacher did this by providing a bunch of problems on a single two sided page where the last problem said “ignore all previous problems, sit quietly and you’ll pass with full marks”.

One day it would be interesting to provide a code base for a home task and specify a requirement for a feature. I’d like that requirement to already be implemented to understand if a candidate would blindly continue to implement something or submit the existing code base as task complete. Bonus points if they implement a test. Partial knowledge comes from many sources, one such source can be a PM or Sales, the basis of blindly following authority can lead to redundant work and more maintenance in the future. A candidate that closes the gap can be valuable.

A friend of mine qualified as a doctor and informed me of some of the tests they are faced with. There was one i found to be interesting and would consider giving software engineers. Have them walk into a room and sit at a table across from someone, instruct them that they need to convince that person to organise the shapes on their table to match the shapes on the candidate’s table. As soon as the candidate says “put the triangle in the top left” they should respond with “what’s a triangle”.

Not only do I believe that it can be a valuable assessment of how the candidate is able to walk through different levels of reasoning to find common ground it also captures the realities of working in cross functional teams with team mates of varying experience where questions like “we need to offer SS on the API” get met with “what’s SS?”

The interesting thing about the startup world is that progression is typically, 1. working on someone elses idea. 2. working on your own idea, 3 funding other people to come up with ideas. A common mistake for many of the YC college dropouts tend to be; build first, ask questions later. When you're YC and are hedging your bets for a slice in a few potentially great businesses paying for this education is cheap.

It is common for startups to run head first through the phase: build it and they will come.

we built a couple things that just no one wanted and then realized we're going about this the wrong way.

Christina Cacioppo (Vanta) 2025²

This mistake isnt limited to the young cohort of YC startups. This is a fallacy that cause many Software projects to fail, it is a lesson that all creators have to learn.

Software Engineers

Software engineers hit a maturing point when they make the shift away from accidental engineering to intent driven decision making.

Early in their career they tend to enjoy the exploration of new ideas and the novelty of new software in business. Often referred to as CV driven development. They dive head first into developing features or solving problems. This characteristic isnt inherently a problem when guided by engineers that have their fair share of war stories, however as every engineer comes to learn; diving head first into the code will lead you down a painful path.

Established engineers approach a problem with a focus on what the user or customer intends to do and avoid relying on codified solutions.

It is still easy for experienced engineers to fall into the trap, blinded by excitement or neglicting to push back through fear or through finding it easier to go with the flow than challenge and presenting an alternative.¹

Without establishing customer intentions we are building with hope as the dominant strategy and are leaving our product up to chance. Leading with intent helps to create things that have purpose.

The most noticable example that highlights this behavour is when an engineer manages to close out work without writing code or when they establish that work shouldnt be done until something fundamental is proven. This can be as simple as, "is anyone asking for this?".

The Users

Companies place a heavy emphasis on valuing the customer as a means to avoid repeating their early mistake. The truth is we shouldn't be valuing what they say but we must try to care about the same things they care about. We need to discover their intent.

Product market fit is about creating a product that serves the intention of a large enough market. Before we can create a product we need to figure out what the customer wants to do and then figure out how to serve them. We can't setup a pillow renting business before we have asked if anyone would actually rent a pillow.

The "Customer Obsession" mantra isn't just corporate shill and it is a common value among companies because it is easily forgotten.

If I'd asked customers what they wanted, they would have told me, 'A faster horse!

Henry Ford

Steve Jobs used this quote to highlight why giving the customer what they want doesn't work. This doesn't mean you shouldn't speak to customers, but you should figure out not what they want but what they want to do. In the Henry Ford example, they want a faster horse, but their underlying intention is to get from A to B at speed.

The user will tell you want they want but it is our job to find out what they need.

Another easy habbit to fall into is being quick to provide answers without the full picture. Your first solution is often never the best and you might find that the customer might have already tried it. Instead of providing the answer, our first step should be to question why they want the answer in the first place.

Trying

Figuring out how to address a customers need is hard work and I've found people tend to be alergic to hard work; often coming up with reasons why their market is build differently or how something so common as product market fit doesnt apply to them. The failure of startups is a tale as old as time and the "I'm not like everyone else" is probably the reason people fall prey to common pitfalls.

There is nothing that should be more exciting than a hard problem, hard problems will make most people give up. This makes solving it all the more enjoyable.

Humans and computers are really bad at being random. An article recently published by the FT examined how people behaved under seemingly random situations and how efficient computers are at predicting our decisions even when we aim to act as randomly as we can.

Computers are equally bad at acting randomly; yet this hasn't stopped us from attempting to develop algorithmic approaches to generating randomness. Random number generators have many applications, from generating worlds , running probabilistic simulations to cryptography.

Randomness can be split into two categories, True Randomness and Pseudo Randomness. True random may only appear in life and nature where it is impossible to predict the next state despite knowing the current state. On the other hand even though pseudo random appears unpredictable it can be replicated if all the starting conditions are known.

Quality of Randomness

There are several qualities to the algorithms that provide us randomness by which we can compare them. The first quality is the algorithm's "period", the number of time we can generate an output before the numbers start to repeat themselves. The second is the number of dimensions the algorithm can pass the spectral test.

In the 1940s von Neumann presented the middle-square method for computing random numbers and it worked as follows:

1. Take an n-digit number
2. Square it
3. Take the middle n digits from the result

The alternative was to find a random number from a book. Whilst there's no set period for the middle square method it is often short and unpredictable which proves to be inconvenient in most applications.

An improvement was made in the 1950s with the development of linear congruential generators. However these fall short on spectral tests which means they start to develop hyperplanes in larger dimensions (the English translation: you start to notice patterns in 3D).

There are a number of tests that are used to measure the quality of a random algorithm which have become known as the Diehard tests. Among them is a suite of tests called "Craps test" where the algorithm plays 200k games of craps and the count of wins and throws should follow a specific distribution.

Contemporary Randomness

All algorithms that generate random numbers start with what is known as a seed. The seed is a number that is used to provide an initial state that all future states are based on. This allows us to replicate random generation or seemingly random simulations when provided with the initial seed.

There are some situations where a seed is not explicitly given when generating something random - however typically your computer will use the current time as a substitute for the lack of explicit seed.

Most contemporary languages such as python use an algorithm called the "Primitive Twisted Generalized Feedback Shift Register Sequence" this is known more commonly as the Mersenne Twister. Due to the existence of a Mersenne prime constant in the algorithm. You can read the C implementation of MT in the python source.

A note on Mersenne primes: these are prime numbers that are (2^n) - 1. One less than a power of two. These primes happen to be the quickest to prove as prime and therefore make up all the largest known prime numbers.

The period of the Mersenne Twister is (2^19937), which is a number I don't have time to type out. 19937 happens to be that Mersenne prime number in the algorithm and the algorithm passes spectral tests up to 623 dimensions.

Cryptography

The realm of cryptography relies on crypto-secure pseudo random number generators (CSPRNG). These are used when generating a password or private/public keys for encryption like RSA. They are required to be both random and unpredictable. Unfortunately the Mersenne Twister isn't safe for cryptography since with enough observations it becomes predictable. As evident in the blog series: Cracking Random Number Generators

Over time we've seen the pseudo random algorithms required in cryptography being broken and methods discovered in order to predict the random number used in the system. Cloudflare proposed a solution to this, instead of having the CPU generate a number using a predetermined algorithm, have a wall of lava lamps and take a photo of the wall whenever you need a new number. This works because images are stored on a computer as numbers and there is naturally occurring randomness which is extremely hard to reverse engineer.

A PR should never be rejected for architectural decisions. When code is written the overall system should have already been planned and agreed upon.

There are two common types of software documents, the architectural decision record [ADR] and the software design document [SDD]. Companies can combine both or do with only an SDD. There's no one size fit all and not all software needs much thought.

Planning software is a recurring theme on my blog and this article attempts to provide the structure to how I plan and design.

ADRs

Architectural Decision Records are typically how decisions are expressed given the context and knowledge at the time. Martin Fowler will cover the topic of ADRs better than I.

SDDs

A Software Design Document is the specification detailing the system and the system's intent before it is built in order to encourage feedback and catch plot holes early in the development cycle of software.

Other than allowing us to get feedback before commitment they are a useful practice for the following reasons:

1. Discovering knowledge gaps. The practice of planning the system allows us to make an initial pass at realising an idea. The practice can bring to light areas we hadn't noticed and reveal that our logic doesn't make sense on paper.

2. Reflection. We learn over the course of a project and having a document that captures our assumptions at the start provides us with a meaningful document that we may reflect on and identify why we were initially wrong.

3. Prosperity. Systems are often forgotten, having a document that outlines the project's initial stated aims can provide an answer in the future when we ask "why did we do this?" Or someone new to the company asks "why did you build it like this?".

4. Getting buy in. It is tough to convince others of an idea in your head. As soon as it's on paper and there's clear logic to your conviction there will be more willingness of others to support. Projects that stand up well to criticism are also more likely to be convincing and picked up.

Template

These documents depend on the problem they aim to address and typically include the following sections:

Context

Tell the audience the problem you aim to solve and why it is important to solve it. Generally I define the state of the system at the current point in time and expose the gaps it has.

E.g. users are wanting double the portion of donuts, we only have 1 donut machine, if we buy another donut machine we will be able to serve twice as many donuts and multiply our profit by a factor of 2.

Scope

Make the limitations clear upfront. Tell the reader what the solution is not aiming to solve. The perfect software doesn't exist and if we tried to build it we would run out of time and runway. Setting boundaries allow us to focus on the core problem we aim to solve.

E.g. The donut machine will not address the users asking for apple juice.

Proposal

Start at a high level and break it down into its components in the subsequent sections. When communicating systems we must understand that individuals consume information differently, some are more visual than others. Knowing this we can enrich the document by providing more than a single representation of the point we are trying to make.

Even if a decision is minor the trade-offs should be tabulated. The choice can be obvious but the practice of assessing an alternative can reveal better solutions, and avoid falling into a Cargo Cult trap.¹

Rejected Options

Avoid designing once, by including the rejected options we make this practice clear. Our first idea will not always be our best idea². Understanding the weaknesses of our rejected options can also provide answers to the curious reader.

If we need to improve the system in the future; remembering why we rejected the other options may allow us to avoid them when we need to pivot. Unless something has changed, we might have a better understanding of options in the future and they'll become opportunities.

Optional Sections

These sections don't apply to all software designs, you might use all of them on a large document. Smaller changes to a system disregard them.

Cost

Money and time.

Provide cost and time estimates, these can give other departments a head start when considering pricing or capacity. Start with estimates or source price pages, something is better than nothing when it comes to informing the stakeholder of a go/no-go project.

Cost is not only found in raw compute. There's also the ongoing cost of upkeep; ensuring the system is up-to-date. Some systems require a member from support staff or an analyst for input on a regular basis, these are flagged as operational cost.

Risk

Address what can go wrong and cover how likely it is to happen. On large projects, some risks might not have a mitigation but we can address how to measure it and minimise it's impact by catching it as early as possible.

Determining the project's risk comes from experience. They range from technical risks like an overview of technical short cuts and their cons, to softer risks such as the introduction of friction affecting product adoption or the consequence of design affecting popular perception.

Goals

Every system should have a goals otherwise it's literally pointless, without a clear purpose the systems is debt before it gets off the ground. This also helps to alleviate scope creep as clearly defined goals allows us to cut any work that does not achieve the original stated purpose of the system.

Milestones

Through my experience I have found high level milestones can be useful, however if they are inflexible or significantly detailed they hold a project back.

The benefit of high level milestones is that it indicates a larger vision for the project and the potential long term impact it may have. We tend to learn over the course of the project so these should be flexible as we are bound to discover things up until the release of the first milestone that will impact our initial plan.

Getting feedback from the first users of your system can throw your roadmap and milestones in the bin.

Deep Dives

As a company scales the systems become more complex, complexity can also be introduced by regulation and some consideration needs to be made before bringing on 3rd party providers. To deal with this I have a shortlist to assess risks when designing software.

Certifications may be required if you are operating in specific industries such as healthcare. It's best to know upfront that you're integrating with a HIPAA compliant party than after the integration has completed. Finding out too late can be a huge waste of time.

Regionally locked data might be a requirement, GDPR, for example, requires explicit consent from an individual if their data is going to be transferred across regional boundaries. If you are integrating with a 3rd party you may need to confirm what regions they are able to operate in.

Authentication and permissions are commonly needed if we have to limit access to specific users or to subsets of users. Determining how permissions are granted and how users authenticate can impact the design of our systems.

Localisation is required by most medium size software businesses as they'll operate in more than a single language or more than a single currency. Serving multiple geographies require these considerations.

Security audits and assessments can be required before being able to use a provider's service. We should ensure the software we use is trustworthy if we are likely to send sensitive data.

These topics may be brought up under our section on risk, or during the discussion of trade-offs between solutions.

FAQ

There are questions that are asked more frequently than others, it's always helpful to include them. Even the off hand questions that were brought up in passing. Good questions can be more interesting than the answer.

Feedback

Feedback is often an opinion and can come from someone with lesser context but more experience. Figure out the quickest way to test an opinion. Don't get caught up on addressing everything but ensure you de-risk concerns by having a plan if their concern becomes a reality or making it easy to roll back or having circuit breakers or feature flags.

As an example a conversation around "this might not scale" could be countered with "this might not get a lot of traffic", systems don't need to be implemented with scale in mind, but they should have an answer that can address scaling if it becomes a concern unless you have some concrete evidence that it won't be a concern; such as; we don't think this will be used often by users but if it does we have a way to limit it's usage and have isolated it from the rest of the system so that if it starts getting attention it won't affect the performance of the whole system.

Do we always need an SDD?

One thing that has held true in software is that the earlier something is found in the process of developing software the easier it is to correct course. Software design documents are a way to map out the unknowns and identify blind spots in ideas. Not all ideas are complete, sometimes ideas come without a solution, we don't want to find ourselves a month into development to realise we don't know how the solution is actually meant to look.

Not all changes require deep thought or detailed specification. There should be room to explore and learn by hacking something together, but the exploration needs the bigger picture and the goal. Without purpose we are wasting time.

This article is longer than many of the design documents I've written.

Keeping track of a constant set of items is fairly straight forward, however when the number of items start to grow larger than the capacity of a single machine things get expensive. There's a way around this and it's to rely on approximations instead of concrete numbers. There are two probabilistic data structures I'd like to cover in this post; Bloom Filters and Count-Min Sketch.

Generally these are applied when space is a constraint and you need predictable and consistent size. If you're counting or caching at scale there might be a chance that your database is relying on probability instead of certainty.

Hash Tables

A fundamental data structure in computer science is the hash table, useful in caching and counting, as well as representing objects in software.

A large hash table is often useful when you need a key value store such as one that would map user ID to a profile picture so that every request for a profile picture is speedy since hash tables operate in 0(1) for look ups.

This is done by having a number of buckets and a function that consistently converts a key to the index of a bucket. This is called the hash function. A cache or a key value store only requires computing the hash to locate the data.

If we were to hash the key "fox" (hash("fox")) and the resulting output was 5 we would know our data is in bucket 5.

Hash functions won't compute unique hashes and occasionally they can collide with existing keys stored in the hash table. So both hash("fox") and hash("cat") might end up pointing to bucket 5. They can reduce the chance of this happening by increasing the number of buckets. Having 10 keys and 1 million buckets means the chance of collision becomes extremely small.

In practice hash tables store linked lists in the bucket locations and when a collision occurs they iterate through the list until it finds the key. When storing a new key it appends to the end of the list if it's not there already. Redis uses this technique in addition to resizing the number of buckets dynamically in-order-to keep the length of these lists to a minimum.

We can use hash tables to determine if we've seen something before, by storing keys as we see them, if the key exists in the hash table we know that this isn't the first time we've seen the item.

Bloom Filters

When we start dealing with data streams or billions of users, storing everything in memory can be expensive. Instead we can reduce the total memory consumed by using a probabilistic data structure; the bloom filter.

Bloom filters rely on approximate set inclusion. So instead of yes this item is in the set or no it isn't; we get the following outcome:

1. This item is not in the set.
2. This item might be in the set.

This is done by having a consistent number of buckets and using more than one hash function. As you can see in the illustration below, the key is hashed three times and each resultant bucket is set to 1.

When items are queried we will know for sure that we haven't seen it before if any of the buckets return a 0. However if all the buckets result in a collision then we know that we might have seen it before.

Both the number of buckets and hashes can be configured which allows us to trade more space for a reduction in the probability that we return false negatives. (We might have seen it, when we haven't).

The bloom filter is applied in situations where space is limited and keeping track of every element isn't an option. If we wish to avoid making expensive queries for data that doesn't exist; a bloom filter can help us reduce the number of expensive queries.

Browsers have used bloom filters in the past by providing a preset filter of malicious URLs. When we visit a URL and it's not included in the filter we can proceed however if it might be in the filter we can query a server to help determine if it's safe or not. We avoid this query on the majority of URLs as most URLs are safe.

Count-Min Sketch

The last probabilistic data structure I'd like to cover is the Count-Min Sketch, like a bloom filter it has multiple hash functions, unlike the bloom filter it tracks the number of times a key lands in a bucket.

When queried it hashes the key and returns the minimum from the counts stored in the corresponding buckets. This allows us to determine an upper bound estimate for the number of times we've seen a key.

Count-Min Sketch is useful in large scale data processing, for example if we are interested in tracking the top-k searches we can do this normally by using a heap. If we have size restrictions and need to use constant space instead of O(n) space we can put the sketch in front of the heap.

Heap inserts are done in log(n) time which we can avoid doing if we know the item shouldn't be in the heap. Items that appear infrequently are then discarded before even making it to the heap. We do this by querying the sketch for an upper bound of the new item, for example 3, and if the kth item in our heap has 12 appearances then we can avoid adding the new item to the heap.

I have found it interesting that at scale we can use probability to optimise our systems but it also requires an understanding of how the data is distributed. These data structures work well on long tail distributions but when all items are as frequent as each other these become less useful. It would be interesting to discover how systems can map to different distributions of data and how these structures are set up to solve the given problems.