Over a weekend I had a quick play around with the config provider to traefik, experimenting with reverse proxies. I'm fairly familiar with the redis-cli and it's fairly easy to build a redis client in any language so getting off the grounds is a 10min job at most.

I've always got redis container running locally, so I'm sticking with that, to get this running I do:

docker run --name some-redis -p 6379:6379 -d redis

Since I am running traefik on docker-compose and I'll need it to connect to redis, I have to create a network and redis should be connected to this network:

docker network create mynet
docker network connect mynet some-redis

The next step is the compose file for traefik:

# docker-compose.yml
version: '3'

networks:
  maunet:
    external: true

services:

  reverse-proxy:
    image: traefik:v2.5
    command:
      - "--api.insecure=true"
      - "--providers.redis"
      - "--providers.redis.endpoints=some-redis:6379"
    ports:
      # HTTP
      - "80:80"
      # Web UI
      - "8080:8080"
    networks:
      - maunet

And up:

docker-compose up -d

The traefik frontend should be accessible at 127.0.0.1:8080.

Configuration

A few of the traefik concepts didn't make sense at first, but it didn't take long to understand what they're talking about.

• I have another container running a simple rest server, it's serving on port 3000 which is exposed at port 53782 this container is called 'some-server'.

• A hurdle that took me a while to figure out was to also connect this container to the network we have traefik and redis running on:

docker network connect mynet some-server

• I need to tell traefik a service (server) exists:

set traefik/http/services/<service-name>/loadbalancer/servers/0/url "http://some-server:3000"

• Now assign this service to a router:

set traefik/http/routers/newroute/service some-server

• Lastly give the router a rule: (Note those are backticks around the forward slash.)

set traefik/http/routers/newroute/rule "Path(`/`)"

Now hitting 127.0.0.1:80 will forward the request to your server. You can also do all the other interesting things that traefik provides like loadbalancing and stripping path prefixes.

References

• https://doc.traefik.io/traefik/routing/providers/kv/

There are some bugs and problems that give a thrill once they're solved. The best bugs are the ones that teach you something, the worst bugs are the ones that indicate that you've not improved your spelling and the difference between an l and a 1 is large.

Figuring out why I was facing the following traceback in heroku provided a time consuming learning experience, but probably one that I won't forget.

heroku[web.1]: State changed from crashed to starting
heroku[web.1]: State changed from starting to crashed
heroku[web.1]: State changed from crashed to starting
app[web.1]: Traceback (most recent call last):
app[web.1]: File "/home/app/server/bin/server", line 3, in <module>
app[web.1]: from api.server import deploy
app[web.1]: ModuleNotFoundError: No module named 'api'

Why can't the module be found 🤔!

Worst of all, the dockerfile builds locally, and when I run it. It's executing /home/app/server/bin/server and ready to receive traffic...

FROM python:3.8.2

ENV USER appuser
ENV HOME /home/${USER}

RUN mkdir -p ${HOME}/server
WORKDIR ${HOME}/server

ENV PATH ${HOME}/server/bin:${HOME}/.local/bin:$PATH

USER appuser

CMD ["server"]

I won't go into how long I spent on thinking it had something to do with permission. Looking back, it's quite clear that a permission has nothing to do with it, since the logs would say so.

Take a step back

The difference between the image on heroku and the image running locally. What probably made me assume it was a permission error was the quote from their docs:

containers are not run with root privileges in Heroku

So there was some funky business they're doing to the user I provided.

Get closer to the problem

Finding out that I could get inside the heroku container certainly helped me figure out the problem:

heroku run bash

I could now recreate the Module not found error. I tried using pipenv to install the module that was missing, however that didn't work either. hmm.. Where are these modules installed??

Show me the site-packages:

python -m site

Heroku

sys.path = [
    '/',
    '/usr/local/lib/python38.zip',
    '/usr/local/lib/python3.8',
    '/usr/local/lib/python3.8/lib-dynload',
    '/home/appuser/server/.local/lib/python3.8/site-packages',
    '/home/appuser/server',
    '/usr/local/lib/python3.8/site-packages',
]
USER_BASE: '/home/appuser/server/.local' (exists)
USER_SITE: '/home/appuser/server/.local/lib/python3.8/site-packages' (exists)
ENABLE_USER_SITE: True

Local Docker Container

sys.path = [
    '/home/appuser/server',
    '/usr/local/lib/python38.zip',
    '/usr/local/lib/python3.8',
    '/usr/local/lib/python3.8/lib-dynload',
    '/home/hints/.local/lib/python3.8/site-packages',
    '/usr/local/lib/python3.8/site-packages',
]
USER_BASE: '/home/appuser/.local' (exists)
USER_SITE: '/home/appuser/.local/lib/python3.8/site-packages' (exists)
ENABLE_USER_SITE: True

Right so they are not referencing the same site-packages. Are the site packages even installed in the heroku container?!?

$ ls /home/appuser/.local/lib/python3.8/site-packages
Flask-1.1.2.dist-info               mccabe-0.6.1.dist-info
Flask_Alembic-2.0.1.dist-info       mccabe.py
Flask_Cors-3.0.8.dist-info          more_itertools
Flask_JWT_Extended-3.24.1.dist-info more_itertools-8.2.0.dist-info
Flask_SQLAlchemy-2.4.1.dist-info    oauthlib

🎉 so they aren't missing...

Locally

echo $HOME
/home/hints

Heroku

$ echo $HOME
/home/hints/server

Ah.. So $HOME has something to do with it.

Well I have $HOME=/home/appuser and WORKDIR=/home/appuser/server perhaps heroku is setting the work directory to the home directory. 🤷‍♂️

ENV USER hints
ENV HOME /home/${USER}

RUN mkdir -p ${HOME}
WORKDIR ${HOME}

And sure enough fixing my deployment.