I have a frightening memory of joining a company and my first Pull Request required me to SSH into a compute instance and do a git pull in order to release my change into production. On that occasion I found that I was also deploying more than just my change as the branch was not entirely up to date.

There's a battle of trade-offs in software where things that sound good on paper are often placed in the backlog and forgotten about. Something about the process being good enough or being the way things have always been done that kicks valid improvements down the priority list. It is with luck that some are granted the privileged to determine what they get to work on and companies will give that chance to people that have agency and a strong conviction that they know better.

How it began

The other pattern I noticed at that company was a lack of dev and prod environments. Everyone with access to the project had access to everything. At this stage the company was on-boarding more engineers and data scientists and each of them were being granted full access to this monolithic environment.

It was even more exciting to find legacy projects without owners and services being deployed into production using the developer's container orchestrator of choice. Docker-compose, Mesos, self-hosted kubernetes or GKS.

The company was growing quickly and it felt like with every new developer there was a new way to release changes into production. Every fire I was pulled into appeared to be running on it's own tech stack and required learning how things operated from the ground up. Nothing seemed transferable from one project to the next.

These were the problems I had determined to solve.

Taking on the problem

Over a week I mustered together a team to tackle this complexity. The first thing was a meeting with the CTO to propose setting up two new project environments; one for dev and one for prod. This was an easy request as the CTO was horrified to learn that things were being deployed straight to production.

The next thing to setup was a CI/CD pipeline that is generic enough to be used across any service. This meant that if you wanted to use our new dev/prod environments your servers had to be deployed through our automated pipelines. To help the other teams we wrote a service template and helm chart that would play nice with CI/CD. This also meant that we restricted the deployments to a single hosted container orchestrator and allowed us to consolidate all the different styles of deployment. As a consequence we were able to help across more teams as they ran into issues due to us becoming more familiar with kubernetes and not having to understand an entirely new workflow or orchestrator.

The Fun Didn't Stop

We had an outage at around 2pm in the afternoon when the company lost connection to all the servers in the production environment. At this point we had around 5 teams deploying code on our stack. An urgent message in one of the slack channel asking if anyone had changed something revealed that someone had assigned their service a new IP range which masked our network bridge to the rest of the company.

This is when we introduced terraform. Any network changes or changes to the infrastructure could now be reviewed, reverted and audited since it was defined in code. If something went wrong we could investigate the changes as well as having our infra committed to version control. We saw the introduction of terraform improve the adoption of our stack as new engineers were frightened to work in the other cow-boy projects that had existed on infra configured by hand through the web UI.

We started to notice that other teams were adopting our tech stack as they no longer needed to spend time on defining the release process as we had a template that they could clone and get started almost immediately. Now they could tackle business problems instead of weighing up the trade-offs of each container orchestrator.

Our helm chart also helped, we got rid of the mountains of bespoke yaml used for several k8s deployments. We could also serve engineers with features such as specifying cron jobs with very little configuration in their service spec. Most teams were data intensive and relied on scheduled batch processes, so in some cases they had only adopted these cronjobs.

We also developed a library for new services that setup logging, sentry integration, trace IDs and database connections and introduced a standard for running database migrations in the service template. At this point is was quite important to encourage shared ownership of these codebases. The more open we were to changes from other teams the more likely they were to use it and benefit other teams from these changes. This broke down the silos that had existed previously, as improvements were no longer isolated to a single service but could now be utilised across the company. It also meant these libraries were improving without needing someone on my team working on it full-time.

Clear Sailing

My team of 5 was serving 80 engineers and data scientists across 12 teams. Upon reflection not everything went smoothly. There were people that wanted to maintain control over their entire stack, perhaps we were a bit stretched and couldn't focus on features they required at the time. They might have also had disagreements with choices we had made in the service template.

There were also features we developed that didn't get adopted, which makes sense to me now. These were things that generally slowed people down for very little benefit. Contract testing is an example of this. On paper having clear contracts between services and having a means to define and test these contracts sounds like a great idea. The nature of the services at the time, being cronjobs or at most two endpoints meant that their interfaces weren't growing in complexity and introducing this step in the build process wasn't a big enough bang for their buck.

Service to service authentication was another feature that we didn't need at that time. Our services were internal and in a VPC. Obviously if we were compromised not being able to send requests to the servers in that network would be a great thing to have, but I think it would have been better to sink time into features that would speed up the adoption of our workflow instead of features that added friction. Not to say we didn't need this auth layer but we could have potentially address this at a later point.

Improvements

There are improvements to the workflows that I would have enjoyed introducing. For example; using a CI/CD server that didn't rely on defining our workflows in Kotlin. I feel that Kotlin added a barrier to contributions and we didn't need this. There are other CI/CD tools like gocd.org and concourse-ci.org which have an easier way of defining workflows. Although now-a-days we can get a lot done with github workflows and reduce the reliance on having a CI server.

We attempted introducing Istio, however, at the time this was an aspirational feature but if we had succeeded we would have allowed our teams to run canary deployments, which would allow them to divert a small amount of traffic to a new version of their service and if anything goes wrong, it avoids rolling out the broken version to all customers.

I still seem to come across companies that only give leadership the ability to deploy to production and changes go out once mid week. When this is the practice many changes tend to accumulate and when they're released everything goes out as a big-bang deployment. When something breaks in these scenarios it's often harder to pinpoint what went wrong. Smaller frequent deployments grants autonomy and shortens the feedback loop which speeds up a developer finding out what went wrong, releases also tend to be less disruptive and engineer's have a higher confidence in the changes they release into the world.

The biggest take away from my experience in leading an infra team is that we learn by making mistakes, so we shouldn't try reduce the number of mistakes we make. We must focus on reducing the cost of the mistake through incremental change, rollbacks and observability. Many companies and teams try to make no mistake at all and in doing so they cost themselves growth.

It is not the job of sales to find customers with problems that we have solved, but also to find the problems that we can potentially solve. The role is one part customer relations and one part business development.

The challenge with this role is that features fall on a spectrum from nice-to-have to critical-for-business; being able to tell the difference is a key skill for anyone in sales. If we filter all the nice-to-have functionality out of the developer's backlog what remains is the high value and impactful functions that allow us to sign more deals.

Engineering teams often miss this business context and lack the skill to differentiate these features. Part of the problem stems from being a few steps removed from the customer and nice-to-have features can appear more exciting to work on but don't end up creating value.

Engineers that wish to be highly autonomous and are charged with seeking impactful work need to exercise their inner salesperson. Those that don't embrace these skills are at risk of wasting their time and working on the wrong thing.

Buy Signals

Those in sales and business dev look for hints that inform them of the project's potential to get to the deal signing stage. Focusing on projects with high potential allows us to avoid chasing cases that won't move forward or are likely to end without being closed.

If more than one customer is asking for a feature this can be an obvious indication that focusing on this can provide more impact. If you're working on internal software infrastructure, are you getting a feature request from more than one team, or is it from the esoteric data scientist that seems to have their own bespoke setup?

Sales rely on buy signals. A buy signal can indicate when a client is excited enough by a solution that they are willing to consider a purchase. The most obvious way they signal this is by giving you money. If they're trying to give you money before the solution is built, you'll know the solution is somewhat important to them.

We don't have to rely on the client putting down money for us to recognise a buy signal. Buy signals can appear from any skin a customer is willing to risk for a solution. If a customer is willing to vouch for the product to a superior this is a political stake. This allows us to move up the decision chain and provides a positive signal that a deal can make it to close. We also use this as a way of making progress in conversations without explicitly asking for money.

Software engineers should want to get closer to the projects that have the largest impact. We can use buy signals to determine the projects with the most buy-in from leadership and create the most business.

Discovery Questions

Developers can find themselves working on the right project but can end up developing the wrong thing without an accurate idea of the pain-point. We need to peel back the layers that surround a customer's pain to determine how we may best deliver.

Sales do this with discovery sessions or hearing it directly from the customer. Their goal is to avoid making incorrect assumptions about the problem which may lead to building the wrong solution. Software engineers could also benefit from this skill. If you're given a spec an engineer should ensure they're delivering the appropriate fix, otherwise they risk missing the target and have to do revisions or start again from scratch.

Sales does this with open ended questions, the more space they provide a customer to explore the problem the higher the chance we expose insight. Getting them to dig deeper on specific points can expose issues they might be having and asking how a solution might affect their business can help sales qualify this lead.

A classic technique of getting your customer to talk is called Mirroring covered in "Never Split the Difference" by Chris Voss.

Are your developers asking enough questions to expose misaligned assumptions?

Touch Points

Progress updates with customers allow you to show that you are keeping them in mind and their requests haven't been forgotten about. They also provide a heads up when new features are near release. They are used to build trust and allow the customer to feel like they are helping guide the process.

If your wins are also their wins they will feel like they have more skin in the project's success and become an internal champion across the industry and in their own business.

How often do you provide a task to a developer and only hear about it three months later when there's just two weeks until the deadline?

Regular touch points don't just serve the relationship you are building with your customers but also allow you to validate the project closer to real-time. Ensuring that you are headed in the correct direction is critical for success. If you're heading down the wrong path you'll want to know as soon as possible.

It's a shame that developers seem to be stereotyped as introverts lacking people skills that wish to be isolated from any call with the customer. The industry appears to lean into this idea by adding more barriers between the client and developer. The truth is the companies that have their developers closest to users and customers are the ones that succeed and the best way for an engineer to level up is to care more about the customer.

Determine Timelines

Software can rely on external processes and teams. Figure out what you're building and predict where you might need approvals or contracts, this will allow you to get the ball rolling in those departments. Sales already do this on their discovery calls by probing if the customer will need to bump other departments, bumping them now can lead to less delay.

Within a company we might require a third party tool for the new feature we're developing, so the sooner we loop in finance or the security team the better. We can have them work on their specifics in the background while we work on the feature. We can probe our PM before we start the project that we might require looping in these other teams.

We should also determine when something will be needed and understand the timeline. They could be aiming to deploy their MVP in a month which could affect how we prioritise building the solution. If the deadline is tight, perhaps we can determine if there's only a subset of features required for the MVP and avoid sinking time into the features that are expected at a later time.

Alignment

Developers can level themselves up by becoming more aligned with the business and not less. One way of doing this is by moving closer to the customer or learning how other roles operate. Software will continue to be a competitive industry and engineers can't afford to waste time on things that don't delight their users and/or provide impact.

We have two large tables and 100 guests coming to our wedding and we have to figure out how they will be seated. Defining the seating chart tends to be the most enjoyable part of wedding planning¹.

After drawing circles for all the seats and guests in excalidraw.com, I began connecting the circles with colourful lines to map out specific relationships. As an example, a couple at the wedding should be seated together.

It dawned on me that this is a constraint satisfaction problem (CSP); and modeling CSPs is something I've been doing for the last year.

Constraint Satisfaction Problems

CSPs are a family of problems where you are required to find the value to a number of variables given certain constraints to those variables. There are many areas that such problems occur, one such area is packing optimisations.

There are software engineering roles that rely on solving these type of questions; typically these roles will include the term "Formal Methods" under their list of responsibilities.

To solve a CSP problem, we begin by modelling the problem mathematically. There are a few notations that allow us to define the variables and constraints for these kinds of problems, the one I am familiar with is called SMT-language.

Here is a simple example where we wish to find valid values for x and y:

(declare-const x Int)
(declare-const y Int)
(assert (> x 0))
(assert (< y 10))
(assert (= (+ x y) 15))
(check-sat)
(get-model)

You may have noticed that this uses prefix notation, i.e. (> x 0) which is equivalent to (x > 0) in the familiar infix notation.

To find valid solutions we need a solver such as z3. If we provide z3 with the file above it will give us a solution for x and y that fits the constraint. E.g x = 15 and y = 0. There may be more than one valid solution. Sometimes there is not solution and it will return unsat. Short for unsatisfiable.

Knapsack problems

We can use z3 and smt to model and solve a classic knapsack problem. Given the following products:

• Product A has a size of 3 and value of 4.
• Product B has a size of 5 and a value of 7.

Pack a bag with a capacity of 16. Maximizing the total value of the bag.

(declare-const productA_count Int)
(declare-const productB_count Int)
(declare-const total_value Int)

(assert (>= productA_count 0))
(assert (>= productB_count 0))

; Product A: size=3, value=4
; Product B: size=5, value=7
; Bag capacity: 16

(assert (<= (+ (* 3 productA_count) (* 5 productB_count)) 16))
(assert (= total_value (+ (* 4 productA_count) (* 7 productB_count))))

(maximize total_value)
(check-sat)
(get-model)

Solving this tells us that if we pack two of each product we maximize the total value of the bag; this being 22.

Seating

Getting back into the seating chart. I wrote (vibed) a program that allowed me to draw different connections between each of my guests. These connections account for the different constraints we wanted to map onto the guests. So if A and B are a couple, ensuring they sit next to each other is a constraint.

Here is a complete list of the constraints that were modeled:

1. Sit next to each other.
2. Sit opposite each other. Or constraint 1.
3. Sit diagonally across from each other. Or constraint 2.
4. Not next to, not opposite and not diagonal from each other.
5. A should be next to B OR C.

After drawing all these constraints between our guests this is how the wedding looks:

Green is constraint 1, yellow 2, orange 3, dashed red 4 and dashed purple is constraint 5.

We are seating everyone at two large tables. As would have been done in a classic viking longhouse. In order to model the guests at these tables each guest will need a variable for the position, the side and the table.

The groom will have an assigned seat, in the middle of the first table facing inward, towards the guests. As guest number 1 he would have the following variables:

# table_1: int = 0
# pos_1: int = 12
# side_1: bool = true

This would seat me in the middle of the table looking out across the room. We then define these variables for all 100 guests. We also need to include a constraint that all the table_{n} variables must be 0 or 1, since there are only two tables. Additionally the pos_{n} variables have to be between [0 and 25) since there are only 25 seats on each side of the tables.

Constraints

Now we model each constraint listed above. The first constraint states that guest A and B must be next to each other. If we take guests 11 and 12 this would look like the following:

(assert (= table_11 table_12))
(assert (= side_11 side_12))
(assert (or (= pos_11 (+ pos_12 1) (= pos_12 (+ pos_11 1)))))

I.e. Same table, same side but their positions are offset by 1 or -1.

The next constraint allows the guest to not only sit next to someone but as an alternative they can be opposite each other. To do this we assign the above constraint to same_side_const and define a new constraint assign it to opposite_const and in order to satisfy either of them we say. (assert (or same_side_const opposite_const)). Here's the definition of opposite_const:

(assert (= table_11 table_12))
(assert (distinct side_11 side_12))
(assert (= pos_11 pos_12))

I.e. Same table, different sides but same position.

For the 3rd constraint we merge constraint 1 and 2 together. B needs to have a different side to A and the position needs to be offset by 1 or -1.

Using the tool above to define the graph constraints I could then export the constraints to be passed into a script that generates the smt file for z3. It then provided a solution that I could import into the same tool to see a visual representation of how it would seat all the guests. You can see that below:

As you'll notice from the colour of each line, all the constraints are satisfied.²

Results

So after 5 hours was this seating chart useful? Not really...

The guests that are on the edge of each of the clusters weren't really people we wanted to sit together. However the arrangement within each cluster was great.

Was this fun? Totally!

I'm not going to do this again, because I'm not going to be married again, but in the event that I have to create a seating chart for 100 people in the future there are certain things that might provide a better outcome.

Perhaps it would be better to rely on more flexible constraints, as an example instead of saying A and B need to be in close vicinity I would instead say A should sit next to at least B, C or D. Giving more flexibility to who A can be sat next to.

After presenting this analysis to my future wife she showed me how she had laid out the tables. You can see this below.

Since the constraints are already linking the guests, I was able to spot some improvements to the chart and get some value from this effort.

There was another improvement to the modelling I could have made when looking at the final chart.

If we wanted A to sit next to C and A was in a couple with B. I had not considered that it was probably fine for B to sit next to C instead of A. If C would get on with A there's a chance they would also get on with their partner B. So I could have relaxed this constraint and had C sit next to either A or B.

Creating a seating chart is not a recurring problem of mine, so sadly there's no need for me to refine this solution further.

Guests forgetting to tell me that they're coming has also caused me to pull some hair out, as with most math, when it attempts to model the real world.

A tech stack is more often inherited than it is chosen. On the rare occasion someone gets the chance to decide which tech the company gets to use for the next decade. I've seen this go well and I've seen this go belly up.

If you're in the right club, you know the answer; stick with boring.

The Postgres Paradox

When thinking about how long a technology will last I often use the current age of the software to determine how much longer it will be around. So if it's 2 years old, it might be around 2 more years. Postgres dates back 35 years, so it will probably be around for 35 more years.¹

We can view this in terms of experienced employees; we are more likely to find someone with 20 years of Postgres experience than we are to find someone with 20 years in a tech that's only existed for 2 years.

Taking the age of the tech into consideration allows us to earmark the hiring pool and the likelihood that fixes or workarounds exist for problems we might run into. Diving into newer tech increases the chance we are the first to come across certain problems.

The bleeding edge might appear shiny and new but we aren't here to ship blog posts we are here to ship features and delight customers.

These avoidable issues are hard to solve because of two reasons:

1. They're novel, we're the first to run into them.
2. The people that have faced this issue are expensive, because not many of them exist.

Make a Difference

Starting a business is a risk, picking fancy new tech is a risk. These risks add up. Where you decide to take the risk depends on where you are trying to drive innovation.

In 2015, London, two neobanks were founded; this is the tech they are built on and the tech's age at the time. Revolut is built on Postgres (25yo) and Java (19yo), the other is built on Cassandra (7yo) and Golang (6yo). The former has 50million customers, the later has 10million.

Risks are allowed to be taken, but we should limit the number of unknowns and not make things harder than they need to be. If we pick our entire stack on the principle of shiny and new we are going to be fighting our tech more often that we are solving customer use-cases.

Get excited about delighting users, not the fancy tech.

Don't be First, Be Fast

Early market leaders have much greater long-term success and enter an average of 13 years after pioneers.

Golder and Tellis (1993)

People make the assumption that if they're not using the latest technology they're not staying competitive, when they are instead just becoming a case-study for the industry.²

The study by Golder and Tellis tells us that first movers have a failure rate that is 6 times higher than fast followers.

If the tech stack mattered, you'd see it in advertising, but you don't, because the customer doesn't care. Your tech stack doesn't determine your product market fit³

Your investors might care (🤷).

If I had to choose between customers and investors I'd pick customers.⁴

The Right Tool for The Job

Ask a graduate what tool or language they would use for their next project and they'll give you a political answer: "I would pick the best tool for the job". Here's a list of programming languages. There are 50 that start with the letter A. Let me know when you've found the one that's best suited for the job.

Perhaps we should define what "best for the job", means. If we are in a team of 12 engineers that are maintaining 2 Postgres databases; introducing a MySQL database on the basis that it addresses a specific use-case better than postgres is not a good idea.

If we hire a DB expert in the future are we going to ensure they're versed in both Postgres and MySQL? It's niche but we could find them at a price.

We don't have to hire this DB expert, our team has built up experience with Postgres, now they'll need to familiarise themselves with the maintenance process of a new db. Just bumping your database version is now slightly tricker and you're monitoring news and bug fixes for both releases.

We could define "best for the job" as perhaps the one we are most familiar with, and this is often the best choice. If you can skip learning a new language or learning the framework step, you're going to get to market faster than others. Don't let learning the tech be the procrastination for learning the market.

In a startup we are testing our market assumptions, this is the largest risk to the company, we shouldn't slow down the speed we get to market. It's better to know we are wrong sooner rather than later.

Getting Experience

Engineers at startups might use the job as an opportunity to focus on CV driven develop instead of focusing on the needs of the business.

I've seen a graduate write every new project in a new programming language.

Anyone know Elixir? We're trying to fix a bug in production and the developer that wrote this is no longer at the company.

From a slack thread.

Starting every new project in a new language isn't a fast way to learn how to deliver. It's a fast-way to get 3 months of experience 5 times. Learn how to deal with a language's pain points and how to solve issues with a specific framework is what makes you valuable. If you're looking to stretch yourself, then use the same language and framework for larger projects.

If you've only got 3 months of experience in a language and framework you're probably not that hard to replace.

The new thing won’t be better, you just aren’t aware of all of the ways it will be terrible yet

Slide 84

Propaganda

Propaganda doesn't just occur in war or between competing states, corporate propaganda is a well established strategy available to large technical institutions.

Let's say you are working on an internal framework that holds up your billion dollar social media platform, you wish to hire more engineers to maintain and build the project. Training new hires on this framework is time consuming, wouldn't it be great if the engineers we hired were already familiar with the framework?

They can also hire developer advocates, these are the influencers of the software engineering world, corporate mercenaric missonaries that are very good at making the work they're doing exciting.

Ignore the hype, your job is to keep things simple.

Bleeding Edge

One reason you might wish to be in the bleeding edge of tech is to gain the expertise and then build around some of the fundamentals you've learnt. Perhaps your customers are others trying to enter the market, this is perhaps a safety-net for the risk you take, but the market can also flatten out and you're left holding a bag of air.

There are a few parts that make up the blog, the static site generator, the API and the repo that hosts all the HTML. This article provides an overview of these pieces and how they're all brought together to create the website.

Framework

The framework is called Pelican a python based static site generator. Each post is written in markdown which starts with meta tags such as Title and Category. Pelican then uses this and a custom theme template to create HTML. The theme uses jinja templating to construct template HTML pages.

Pelican also provides a config file which I have used for feature flags and to enable extensions. The extensions I've enabled include footnotes and code-highlighting. Additionally Pelican hosts a local version of my blog which it recompiles whenever it detects a change to any of the files.

In the config there's a list which contains all the links rendered in the sidebar. Extending or updating these links is only a matter of making changes to this list.

There are two directories inprogress/ and content/. inprogress/ stores drafts that are half complete, completed or minor thoughts for potential blog posts. When these are ready to be published they are moved over to content/ and when these are pushed to the repo, using a github action, pelican automatically renders the entire blog into HTML. The action then creates a commit for any changes and pushes the changes to a public repo.

The public repo contains the entire rendering of the blog and theme. Relying on Github pages, this is then hosted.

The CV

Within the same repo as the drafts and published content there's a CV written in Latex. This latex file can be rendered to a PDF. I have a bash script which allows me to recompile this PDF whenever changes are made to the Latex script.

The API

There's an API that contains edge functions written in Go. These serve requests for the comment sections that sit below each blog post. There are three endpoints, one fetches the count of comments given a list of article URLs. This is used on index pages.

There's an endpoint to fetch comments given an article URL and an endpoint to post a comment to an article URL. The repo containing these endpoints has an action which deploys updates on each new commit.

Finally the API handles the mailing list. It handles registering new emails, verifying the email is valid and lastly triggers an edge function to look for new posts using the blog's RSS feed in order to send updates to recipients in the mailing list.