substracker.co.uk

Production

Ensure that the following files are in place on the server:

• datafiles/master_add_data.csv
• datafiles/smd_average_line.csv

They are both contained in the Notes/Projects/Surgetracker/resources folder and should be copied to the server. The master data file particularly is not kept under version control for obvious reasons.

• cd code/substracker
• git pull
• make

---

Log of work to deployment

We will be deploying on Digital Ocean.

Reference tutorial: How to Scale and Secure a Django Application with Docker, Nginx and Let's Encrypt This is saved as a PDF as ~/Documents/Notes/Project/Surgetracker.

The tutorial references Digital Ocean Spaces for the static files, which I think we want to use. The settings.py file in the polls-docker branch of the demo Django repository is instructive - here is where the config for Spaces and the static files is. These are put in the env file that we use with our Docker commands when running the stuff later on. Also, refer to the preceeding tutorial for futher details on this. The sample app uses django-storages for handling static files. Ref: https://www.digitalocean.com/community/tutorials/how-to-set-up-object-storage-with-django.

Instead of two web application servers, we are going to use one, and forgo the failover, load-balancing stuff - for now.

Servers

1. web1 - 2gb RAM droplet for Django, Gunicorn and memcached.
2. pg1 - A database - not sure whether to go for the managed database at this point, which is $15
3. ng1 - a nginx box which accepts HTTPS requests and forwards to the web application using Nginx and certbot.

web1 and ng1 should have Docker pre-installed, so we don't have to bother with that. Enable all free cloud networks, monitoring and anything else we can get.

The main tutorial makes clear that you can configure the nginx container on one of the web containers if you want. That might make sense at first, to get things up and running.

Steps to deployment

NB: There are GitHub actions available off the shelf to test the Django application build Docker containers and push them to a registry. I have a registry at https://cloud.digitalocean.com/registry?i=410f5e.

1. Set up the three servers above. Either use a generic droplet for the database or the managed solution (preferable). Suggest to go for droplets with Docker already set up to save a bit of time. This probably means Ubuntu servers, which is fine.
2. Do whatever is necessary to get ssh access to each one, and ensure they are updated.
3. Set up Digital Ocean spaces.
4. Add the necessary packages to requirements.txt (boto and django-storages by the looks of it). Refer to this tutorial.
5. Also refer to the settings.py file in the polls example repository, in the Docker branch.
6. Create a new staging branch in our repo. Create a conf directory and within that, docker and nginx. Put settings-prod.py at the root of it. Configure settings.py so that it makes sense.
7. Create an env file. We will use this to base our docker run commands off later.
8. Reconfigure our Dockerfile - copy it into conf and rename it for production. Or put it in the production subfolder. Probably best to leave our current configuration intact, even though we are making a new branch. Ensure the gunicorn CMD at the bottom of the Dockerfile.
9. Follow the tutorial through Step 1. This is the first web container.
10. Step 3 is the nginx container. By the looks of things, the instructions are pretty much identical, even though we are only using one container.
11. Set up the necessary DNS for staging.twentyfoursoftware.co.uk or whatever we need for this.
12. Step 4 is all about certbot and Let's Encrypt.

---

Steps take to deploy

These are raw steps and might not be in the most sensible order - I am simply recording the steps taken.

1. Set up new Debian 12 droplet (2Gb) (surge-web-1).
2. Created new user surge.
3. Installed git and Docker.
4. Created ~/code/ and cloned `surgetracker from github.
5. Created production versions of main Dockerfile, in surgetracker/conf/prod. The main thing that changes at this point is to add the gunicorn line at the bottom in place of the manage.py line. And add the EXPOSE line.
6. docker build -f conf/prod/Dockerfile -t surgetracker .
7. On the server, created the env file as suggested, with the following params:

DJANGO_SECRET_KEY=
DEBUG=True
DJANGO_ALLOWED_HOSTS=
DATABASE_ENGINE=postgresql_psycopg2
DATABASE_NAME=polls
DATABASE_USERNAME=
DATABASE_PASSWORD=
DATABASE_HOST=
DATABASE_PORT=
STATIC_ACCESS_KEY_ID=
STATIC_SECRET_KEY=
STATIC_BUCKET_NAME=
STATIC_ENDPOINT_URL=
DJANGO_LOGLEVEL=info

1. Generated the SECRET_KEY with openssl rand -base64 32.
2. Moved settings from the DO Managed Database panel into the config - this was pretty straightforward.
3. Backed up the env file in Docments/sync.
4. Created a new conf/prod/settings_prod.py that I am going to have to point the relevant commands to.
5. Followed the documentation for django-storages to set up the settings_prod.py file. Most of the settings are from the default Amazon S3 settings, with a few changes for Digital Ocean Spaces. These are highlighed in the storages, docs.
6. Added boto3 and django-storages to the requirements file. This means I need to rebuild that image.
7. I also added DJANGO_SETTINGS_MODULE=conf.prod.settings_prod to the env file.

After playing around with the env file, getting it right, I ran the first command to run the container:

docker run --env-file env surgetracker sh -c "python manage.py makemigrations && python manage.py migrate" and it worked!

Then I ran another instance of the container, this time as an interactive shell, so that I could create the superuser.

docker run -i -t --env-file env surgetracker sh, the python manage.py createsuperuser. Ctrl+D to get out of the container.

Then I ran the following command to upload the static files to Spaces, and this fucking worked too:

docker run --env-file env surgetracker sh -c "python manage.py collectstatic --noinput"

There they are listed at https://cloud.digitalocean.com/spaces/surgetracker?path - holy shit.

I tried running docker run -i -t --env-file env surgetracker sh and then python mangage.py import_csv master_add_data.csv to import the data, but got a CSV error. No drama - can fix that. This is after I used scp to copy that file onto the server before rebuilding the image.

Now we run the app with docker run --env-file env -p 80:8000 surgetracker. After one cock up because I left mysite in there from the tutorial, replacing it with surgetracker, boom. Note, I had to clone the repo with the changed commit, then remove the old image, then rebuild the image, then run. I ran the migrate commands but they weren't needed. Yayoo! And the site is live at http://64.227.37.12/ - albeit without static files, or at least the logo, for now - but that is a fucking massive achievement.

Now that we have confirmed that it works, we can run it in detached mode and log off the server:

docker run -d --rm --name polls --env-file env -p 80:8000 surgetracker

This gets me to the end of Stage 1 of the tutorial.

Proxy

Set up a new Debian droplet. Update it. Installed Docker. Created a new "surge" user account to hold the configs. Enabled Docker for non-root.

• created a new user
• disabled ssh access to root account
• restarted sshd
• visudo for surge user

Created a new conf directory in /home/surge/conf.

Created conf/nginx.conf.

upstream django {
	server APP_SERVER_1_IP;
	server APP_SERVER_2_IP;
}

server {
	listen 80 default_server;
	return 444;
}

server {
	listen 80;
	listen [::]:80;
	server_name your_domain.com;
	return 301 https://$server_name$request_uri;
}

server {
	listen 443 ssl http2;
	listen [::]:443 ssl http2;
	server_name your_domain.com;

	# SSL
	ssl_certificate /etc/letsencrypt/live/your_domain.com/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/live/your_domain.com/privkey.pem;

	ssl_session_cache shared:le_nginx_SSL:10m;
	ssl_session_timeout 1440m;
	ssl_session_tickets off;

	ssl_protocols TLSv1.2 TLSv1.3;
	ssl_prefer_server_ciphers off;

	ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";

	client_max_body_size 4G;
	keepalive_timeout 5;

        location / {
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_set_header Host $http_host;
          proxy_redirect off;
          proxy_pass http://django;
        }

	location ^~ /.well-known/acme-challenge/ {
		root /var/www/html;
	}
}

Made the necessary changes and read about the nginx config here: https://www.digitalocean.com/community/tutorials/understanding-the-nginx-configuration-file-structure-and-configuration-contexts#understanding-nginx-configuration-contexts This is also a useful tool to use to tweak the nginx config, apparently: https://www.digitalocean.com/community/tools/nginx

Here is the command to run the nginx docker container for the first time:

docker run --rm --name nginx -p 80:80 -p 443:443 \ -v ~/conf/nginx.conf:/etc/nginx/conf.d/nginx.conf:ro \ -v /var/www/html:/var/www/html \ nginx:1.25.1

As expected, this command throws an error because there are no Let's Encrypt certicates available. We fix that next.

Certbot

https://www.digitalocean.com/community/tutorials/how-to-scale-and-secure-a-django-application-with-docker-nginx-and-let-s-encrypt#step-4-configuring-certbot-and-let-s-encrypt-certificate-renewal

I created an A record to for staging.twentyfoursoftware.co.uk pointing to the proxy's public ip.

Then put in this command to provision a staging version of the certificates using certbot:

docker run -it --rm -p 80:80 --name certbot \ -v "/etc/letsencrypt:/etc/letsencrypt" \ -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \ certbot/certbot certonly --standalone --staging -d staging.twentyfoursoftware.co.uk

After testing this with the domain in place (albeit having to handle the TLS warning), you run this without staging mode:

docker run -it --rm -p 80:80 --name certbot \ -v "/etc/letsencrypt:/etc/letsencrypt" \ -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \ certbot/certbot certonly --standalone -d staging.twentyfoursoftware.co.uk

You have to select option 2 to renew the certificate with this otherwise you will get an error.

Also, I got a 500 error when testing this with staging.twentyfoursoftware.co.uk - it was having problems proxying to the Django server. **And this was because I'd not started the web container! Once I did that - it worked beautifully.

Although immediately after trying to log in, I got a Django Forbidden 403 error - Origin checking failed - https://staging.twentyfoursoftware.co.uk does not match any trusted origins, which I guess is because I did not put this in the settings.py file. Fixing that now. Looks like you have to have CSRF_TRUSTED_ORIGINS = ['https://front.bluemix.net'] in the settings.

And once it's all working, you run this in detached mode to run the proxy in the background:

docker run --rm --name nginx -d -p 80:80 -p 443:443 \ -v ~/conf/nginx.conf:/etc/nginx/conf.d/nginx.conf:ro \ -v /etc/letsencrypt:/etc/letsencrypt \ -v /var/lib/letsencrypt:/var/lib/letsencrypt \ -v /var/www/html:/var/www/html \ nginx:1.25.1

Log out of both server. We're in bizzo.

Getting bulk SMD data in

The python manage.py import_csv command relies on DEBUG being False (in production) and datafiles/master_add_data.csv being present. Run the web container with the new make run command, then get into the container with docker run -it --env-file env surgetracker bash and then python manage.py import_csv. All existing SMD data will be lost and replaced with the CSV data.

Keeping SSL certs up to date

With the nginx container running, do the following to do a dry-run of the certificate renewal. We are using the webroot plugin to use the /var/www/html/ directory for the letsencrypt bit. See the docs: https://eff-certbot.readthedocs.io/en/stable/using.html#webroot

docker run -it --rm --name certbot \ -v "/etc/letsencrypt:/etc/letsencrypt" \ -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \ -v "/var/www/html:/var/www/html" \ certbot/certbot renew --webroot -w /var/www/html --dry-run

Then, if we weren't doing --dry-run, we'd use docker kill -s HUP nginx to reload nginx to use the new certs. Obviously, we'd do the above command without --dry-run to do it live. Apparently there is guidance about automating renewal here: https://www.digitalocean.com/community/tutorials/how-to-secure-a-containerized-node-js-application-with-nginx-let-s-encrypt-and-docker-compose#step-6-renewing-certificates.

crontab

This is the line in root's crontab (do sudo crontab -e to edit it) that runs the certbot renew each midday. This is obviously in the proxy repo... (https://github.com/yulqen/substracker-nginx-config).

0 12 * * * /home/surge/conf/ssl_renew.bash >> /var/log/cron.log 2>&1

The script is as follows:

#!/bin/bash
DOCKER="/usr/bin/docker"
cd /home/surge/conf/
$DOCKER run --rm  --name certbot -v "/etc/letsencrypt:/etc/letsencrypt"   -v "/var/lib/letsencrypt:/var/lib/letsencrypt"   -v "/var/www/html:/var/www/html"   certbot/certbot renew --webroot -w /var/www/html
$DOCKER system prune -af

Cloud firewall

We don't want any access to our web droplet so let's set up a Digital Ocean cloud firewall round it. This is the next step in the big tutorial. Docs: https://docs.digitalocean.com/products/networking/firewalls/how-to/create/

Simply adding a DO cloud firewall, even though only applied to the web droplet, blocked access to the proxy too - don't know why. In the default state, the web droplet can currently be access because Docker opens ports, and port 80 is exposed. Verify by doing: http://64.227.37.12/.

The tutorial gets us to add the following rules into /etc/ufw/after.rules on the web droplet, which is configured to take into account DO's VPCs:

# BEGIN UFW AND DOCKER
*filter
:ufw-user-forward - [0:0]
:DOCKER-USER - [0:0]
-A DOCKER-USER -j RETURN -s 10.0.0.0/8
-A DOCKER-USER -j RETURN -s 172.16.0.0/12
-A DOCKER-USER -j RETURN -s 192.168.0.0/16

-A DOCKER-USER -p udp -m udp --sport 53 --dport 1024:65535 -j RETURN

-A DOCKER-USER -j ufw-user-forward

-A DOCKER-USER -j DROP -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 192.168.0.0/16
-A DOCKER-USER -j DROP -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 10.0.0.0/8
-A DOCKER-USER -j DROP -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 172.16.0.0/12
-A DOCKER-USER -j DROP -p udp -m udp --dport 0:32767 -d 192.168.0.0/16
-A DOCKER-USER -j DROP -p udp -m udp --dport 0:32767 -d 10.0.0.0/8
-A DOCKER-USER -j DROP -p udp -m udp --dport 0:32767 -d 172.16.0.0/12

-A DOCKER-USER -j RETURN
COMMIT
# END UFW AND DOCKER

Then do sudo systemctl restart ufw. ufw wasn't actually running so I then moved on to https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-20-04. Then I made sure I still had ssh access with sudo ufw allow ssh. Then sudo ufw enable. Direct http access to the droplet should not be disabled. Check by trying to browse to http://64.227.37.12/. And we can continue to get in on the proxy! And ssh.

That is the end of the DOLLARS tutorial! https://www.digitalocean.com/community/tutorials/how-to-scale-and-secure-a-django-application-with-docker-nginx-and-let-s-encrypt

Thank you!