Over a weekend I had a quick play around with the config provider to traefik, experimenting with reverse proxies. I'm fairly familiar with the redis-cli and it's fairly easy to build a redis client in any language so getting off the grounds is a 10min job at most.

I've always got redis container running locally, so I'm sticking with that, to get this running I do:

docker run --name some-redis -p 6379:6379 -d redis

Since I am running traefik on docker-compose and I'll need it to connect to redis, I have to create a network and redis should be connected to this network:

docker network create mynet
docker network connect mynet some-redis

The next step is the compose file for traefik:

# docker-compose.yml
version: '3'

networks:
  maunet:
    external: true

services:

  reverse-proxy:
    image: traefik:v2.5
    command:
      - "--api.insecure=true"
      - "--providers.redis"
      - "--providers.redis.endpoints=some-redis:6379"
    ports:
      # HTTP
      - "80:80"
      # Web UI
      - "8080:8080"
    networks:
      - maunet

And up:

docker-compose up -d

The traefik frontend should be accessible at 127.0.0.1:8080.

Configuration

A few of the traefik concepts didn't make sense at first, but it didn't take long to understand what they're talking about.

• I have another container running a simple rest server, it's serving on port 3000 which is exposed at port 53782 this container is called 'some-server'.

• A hurdle that took me a while to figure out was to also connect this container to the network we have traefik and redis running on:

docker network connect mynet some-server

• I need to tell traefik a service (server) exists:

set traefik/http/services/<service-name>/loadbalancer/servers/0/url "http://some-server:3000"

• Now assign this service to a router:

set traefik/http/routers/newroute/service some-server

• Lastly give the router a rule: (Note those are backticks around the forward slash.)

set traefik/http/routers/newroute/rule "Path(`/`)"

Now hitting 127.0.0.1:80 will forward the request to your server. You can also do all the other interesting things that traefik provides like loadbalancing and stripping path prefixes.

References

• https://doc.traefik.io/traefik/routing/providers/kv/

This is now my third article on lists. As someone that uses the built-in python list on a fairly regular basis, I might have built up a false sense of security. I'm pretty familiar with these listy-boys. However, recently I found out that I was not thinking about them correctly. Readers might smack themselves if they're familiar with data-structures but don't know how lists are implemented internally. The built-in lists are dynamic arrays.

How else could they optimise a sweet O(1) lookup time on indexing: mylist[4]. Especially when analysts are trying to avoid the built-in iterator and cursing their code with: for i in len(mylist): mylist[i].

Another trait an established data-structurer with be familiar with when it comes to dynamic arrays is that the append and pop methods are an amortised O(1). Amortised; because occasionally you have to suffer a cost of realloc(ating) memory across larger arrays.

Where the list starts to suffer is from poping and inserting at arbitrary positions.

Linked-List

The data-structurer will have had the linked-list slammed into their head often enough that it will pain them to hear about it again. So theory aside, I'll give you that sweet O(1) append and last item pop that you expect from a performant Stack.

Python deque provides a comparatively larger performance hit on initialisation to list and has poor O(n) performance when you want any arbitrary item somewhere in the middle. It does, however, have O(1); popleft, pop, append and appendleft. Due to being a doubly-linked list (or double-ended queue to get the abbreviation deque)

Deque in the wild

I saw a nice little quote from an enginneer on Quora:

In 8 years of getting paid to write computer programs, this post is the only time I’ve typed ‘deque.’

There are many places deque is used in the stdlib, most commonly whenever someone needs a queue or stack such as constructing a traceback, parsing python's sytax tree and keeping track of context scope.

My little run-in with deque was using it instead of a recursive function to avoid python's

maximum recursion depth exceeded

This limit happens to be set to 10^4. The solution was to add child nodes to a deque and when you were done with analysing the current node, popleft the next node.

Python Queue

You might be tempted to ask, well if deque is for queues. What on earth is from queue import Queue.

These queues are different (although, still using deque under the hood). They are optimised for communication across threads, which need to involve locking mechanisms and support methods like put_nowait() and join(). These are not intended to be used as a collective data-structure, hence the lack of support for the in operator.

More information

There is some neat documentation in the cpython repo which contains more data-structures and other alternatives to the standard built-in list. Tools for working with lists

References

• How are lists implemented:
• https://stackoverflow.com/a/15121933/3407256
• https://stackoverflow.com/a/23487658/3407256

There are some bugs and problems that give a thrill once they're solved. The best bugs are the ones that teach you something, the worst bugs are the ones that indicate that you've not improved your spelling and the difference between an l and a 1 is large.

Figuring out why I was facing the following traceback in heroku provided a time consuming learning experience, but probably one that I won't forget.

heroku[web.1]: State changed from crashed to starting
heroku[web.1]: State changed from starting to crashed
heroku[web.1]: State changed from crashed to starting
app[web.1]: Traceback (most recent call last):
app[web.1]: File "/home/app/server/bin/server", line 3, in <module>
app[web.1]: from api.server import deploy
app[web.1]: ModuleNotFoundError: No module named 'api'

Why can't the module be found 🤔!

Worst of all, the dockerfile builds locally, and when I run it. It's executing /home/app/server/bin/server and ready to receive traffic...

FROM python:3.8.2

ENV USER appuser
ENV HOME /home/${USER}

RUN mkdir -p ${HOME}/server
WORKDIR ${HOME}/server

ENV PATH ${HOME}/server/bin:${HOME}/.local/bin:$PATH

USER appuser

CMD ["server"]

I won't go into how long I spent on thinking it had something to do with permission. Looking back, it's quite clear that a permission has nothing to do with it, since the logs would say so.

Take a step back

The difference between the image on heroku and the image running locally. What probably made me assume it was a permission error was the quote from their docs:

containers are not run with root privileges in Heroku

So there was some funky business they're doing to the user I provided.

Get closer to the problem

Finding out that I could get inside the heroku container certainly helped me figure out the problem:

heroku run bash

I could now recreate the Module not found error. I tried using pipenv to install the module that was missing, however that didn't work either. hmm.. Where are these modules installed??

Show me the site-packages:

python -m site

Heroku

sys.path = [
    '/',
    '/usr/local/lib/python38.zip',
    '/usr/local/lib/python3.8',
    '/usr/local/lib/python3.8/lib-dynload',
    '/home/appuser/server/.local/lib/python3.8/site-packages',
    '/home/appuser/server',
    '/usr/local/lib/python3.8/site-packages',
]
USER_BASE: '/home/appuser/server/.local' (exists)
USER_SITE: '/home/appuser/server/.local/lib/python3.8/site-packages' (exists)
ENABLE_USER_SITE: True

Local Docker Container

sys.path = [
    '/home/appuser/server',
    '/usr/local/lib/python38.zip',
    '/usr/local/lib/python3.8',
    '/usr/local/lib/python3.8/lib-dynload',
    '/home/hints/.local/lib/python3.8/site-packages',
    '/usr/local/lib/python3.8/site-packages',
]
USER_BASE: '/home/appuser/.local' (exists)
USER_SITE: '/home/appuser/.local/lib/python3.8/site-packages' (exists)
ENABLE_USER_SITE: True

Right so they are not referencing the same site-packages. Are the site packages even installed in the heroku container?!?

$ ls /home/appuser/.local/lib/python3.8/site-packages
Flask-1.1.2.dist-info               mccabe-0.6.1.dist-info
Flask_Alembic-2.0.1.dist-info       mccabe.py
Flask_Cors-3.0.8.dist-info          more_itertools
Flask_JWT_Extended-3.24.1.dist-info more_itertools-8.2.0.dist-info
Flask_SQLAlchemy-2.4.1.dist-info    oauthlib

🎉 so they aren't missing...

Locally

echo $HOME
/home/hints

Heroku

$ echo $HOME
/home/hints/server

Ah.. So $HOME has something to do with it.

Well I have $HOME=/home/appuser and WORKDIR=/home/appuser/server perhaps heroku is setting the work directory to the home directory. 🤷‍♂️

ENV USER hints
ENV HOME /home/${USER}

RUN mkdir -p ${HOME}
WORKDIR ${HOME}

And sure enough fixing my deployment.

Easily one of my favourite patterns from the gang of four is the chain of responsibility pattern. It aims to avoid coupling the object that sends a request to the handlers of the request. This is especially useful if the structure to our handlers follows a sense of hierarchy.

I've had very few opportunities to implement design patterns, but gaming has really helped me to envision a context where several designs can be applied as handy solutions to managing complexity.

So we will apply the chain of responsibility to a situation where we have gathered four of the most powerful wizards in a room, each of whom will test their power by casting a spell.

For this example we have our wizards as the objects which send requests.

class Wizard:
    """Create a wizard."""
    def __init__(self, name: str, intelligence: int):
        #: Identify the wizard by name
        self.name = name
        #: Intelligence as a proxy of a wizard's power.
        self.intelligence = intelligence

    def cast_spell(self, spell: Spell):
        """Have the wizard cast a spell."""
        print(f"{self.name} casts {spell.name} spell.")
        spell.cast(self)

We will make each behaviour of the spell, which is determined by the power of the wizard casting it, as an object handling a request. If the wizard does not meet the handler's requirements it passes the request on, to it's successor. This is where we have applied a sense of hierarchy to the handler.

To begin we have an Abstract Handler:

from abc import ABC
from abc import abstractmethod


class AbstractHandler(ABC):
    def __init__(self, successor=None):
        self._successor = successor

    def handle(self, creature):
        reaction = self._handle(creature)
        if not reaction:
            self._successor.handle(create)

    @abstractmethod
    def _handle(self, spell):
        raise NotImplementedError("Must provide implementation in subclass.")

Now we can define our handler hierarchy:

class LowPowerFireSpell(AbstractHandler):
    def _handle(self, creature):
        if creature.intelligence < 10:
            print("Spell backfires.")
            return True


class MediumPowerFireSpell(AbstractHandler):
    def _handle(self, creature):
        if creature.intelligence < 20:
            print("Small fire ball is cast.")
            return True


class HighPowerFireSpell(AbstractHandler):
    def _handle(self, creature):
        if creature.intelligence < 30:
            print("A fire ball blazes across the room")
            return True


class GodlikePowerFireSpell(AbstractHandler):
    def _handle(self, creature):
        print("A Massive column of fire burns through the room!")
        return True

Finally a single object to identify the spell chain.

class Spell:
    def __init__(self, name: str):
        #: Identifying the spell by a name
        self.name = name
        #: How the spell behaves at differing levels of user's power
        self.chain = LowPowerFireSpell(
            MediumPowerFireSpell(
                HighPowerFireSpell(
                    GodlikePowerFireSpell()
                )
            )
        )

    def cast(self, creature):
        self.chain.handle(creature)

To test their skill, we have a spell which casts a "Fire Ball". We have the following wizards present:

if __name__ == '__main__':
    fire_spell = Spell('Fire Ball')
    merlin = Wizard("Merlin", 8)
    albus = Wizard("Albus", 18)
    howl = Wizard("Howl", 28)
    gandalf = Wizard("Gandalf", 38)

They are all gathered in a room, and take turns casting the same spell.

room = [merlin, albus, howl, gandalf]
for wizard in room:
    wizard.cast_spell(fire_spell)
    print("")

To which we should see the spell behave according to their intelligence:

(myenv) pc-name ~ $ python script.py
Merlin casts Fire Ball spell.
Spell backfires.

Albus casts Fire Ball spell.
Small fire ball is cast.

Howl casts Fire Ball spell.
A fire ball blazes across the room

Gandalf casts Fire Ball spell.
A Massive column of fire burns through the room!

(myenv) pc-name ~ $

Implementation can be found here

Before I started reading the gang of 4 ["Go4"], I was convinced I would not need an iterator. After all, in python, they are already implemented:

x = ['a', 'b', 'c']

for i in x:
    print(i)

In the frame of python lists as aggregate objects, the intent of an iterator is satisfied.

Provide a way to access the elements of an aggregate object sequentially without exposing its underlying representations.

Displaying notes

Here I explain, how the iterator became my favourite design pattern.

I have written about representation of lists before, and I don't think this will be my last time. When implementing the drop-downs in foolscap to allow a user to see sections contained in their note, I ran into an issue of complexity where I had blocks of conditionals dictating what should be displayed to a user. I wanted the user to have an indication that a note contains sections, and for them to be able to toggle a drop-down to see each section. Similar to:

 (+) | circleci  |
     | docker    |

Where my "circleci" note would contain sections and the "docker" note does not. Then expanding:

 <-> | circleci  |
  └─ | -workflow |
     | docker    |

This is where I realised the abstraction power of an iterator, And I could hide the collapsed sections behind an "if" conditional in an iterator.

class Menu:

    def __init__(self, items):
        self.items = items

    def visible(self):
        """Yield the next appropriate item for display."""
        for item in self.items:
            yield item.title
            if hasattr(item, 'expand') and item.expand:
                for sub_item in item.sub_items:
                    yield sub_item.title

Supporting further traversal policies in my Menu class is straightforward, and drawing becomes absurdly simplified:

def draw(menu):
    """Draw all viewable menu items."""
    for item in menu.visible():
        draw_item(item)

After this I thought of a more complex aggregate structure that I could traverse, like a tree depth first.

Realising the abstraction strength of the iterator, one finds its definition of intent far more compelling.

Provide a way to access the elements of an aggregate object sequentially without exposing its underlying representations.