Github Self-Hosted Runner

We use persistent EC2 instances to run github actions runner.

• jodapp.qa.deploy
• jodapp.prod.deploy

The term "deploy instances" will refer to these instances

To ensure short-running deployment processes, we pre-install required dependencies/software into these instances.

• If we did not, each time you run github actions we would need to download portable versions of the build dependencies.

Deploy Instances Software Requirements

packages/software	installed as
docker	system-wide via ubuntu (sudo); github-runner in docker group
mise	github-runner user
Ruby	github-runner user
Kamal gem	github-runner user
Github actions runner	github-runner user

Deploy Instance Users

We use least-privileges access for github action runners. There are two users:

• ubuntu:
  • default admin user to manage the instance
  • installs and manages the Github Actions runner agent
• Root / systemd
  • responsible for launching the runner service
  • installed the github actions service via /opt/actions-runner/svc.sh
    • configured to start the process with github-runner user.
    • on boot, root user invokes the service but drops privileges to github-runner for the actual runner process

Files and directories

path	owner	description
/opt/actions-runner	ubuntu (after chown step)	main working folder for github actions
/opt/actions-runner/_work	ubuntu (after chown step)	job data is checked out
/opt/actions-runner/_diag	ubuntu (after chown step)	diagnostics log directory
/opt/hostedtoolcache	ubuntu (after chown step)	main working folder for github actions

SSH keys

Each deploy instance requires to generate their own SSH keys.

• This will allow kamal to use ssh to run the docker commands on the application instances (i.e. jodapp.qa.api, jodapp.prod.web, etc)

SSH keys will have the same name as the instance name. For example the QA instance would have:

• jodapp.qa.deploy (private key)
• jodapp.qa.deploy.pub (public key)

Public keys need to be added to the application instances .ssh/authorized_keys file.

Used by kamal which uses ssh remotely run docker commands on the application instances.

Overview

1. Spin up a new deploy instance on EC2.
2. In deploy instance, generate SSH key for jodapp.{env}.deploy
3. In application instances, update ~/.ssh/authorized_key to include jodapp.{env}.deploy.pub for kamal to ssh into
4. Update Github Action Environment Secrets KAMAL_SSH_KEY for the repo to include this ssh key
5. Install build dependencies
6. Install mise to manage ruby versions
7. Install ruby v3.4.3 with mise
8. Install kamal gem
9. Create folders for github runner
10. Install github actions runner following github actions instructions
11. As github-runner, install github actions runner software

• requires organisation access

12. Start github actions runner software using /opt/actions-runner/svc.sh to load it into systemd

References

• https://codehirise.com/post/setup-github-action-private-runner-in-ubuntu-vm

Install Steps

1. Generate SSH Key for Kamal

# generate key
ssh-keygen -t ed25519 -C "jodapp.{env}.deploy"
# save in /home/ubuntu/.ssh/jodapp.prod.deploy

# print the public key
# copy it to all the application servers ~/.ssh/authorized_keys
cat ~/.ssh/jodapp.{env}.deploy

# copy it to KAMAL_SSH_KEY environment secret for all repos and for all repo env being deployed
cat ~/.ssh/jodapp.{env}.deploy.pub

2. Main Installation

#!/usr/bin/env bash
#
# Complete setup for a persistent GitHub Actions self-hosted runner
# on Ubuntu 24.04 EC2, with Docker, Ruby 3.4.3, and Kamal pre-installed.
#
# Once this runs, you’ll have:
#  • A “github-runner” service account
#  • Docker engine available to that account
#  • Ruby 3.4.3 & Kamal installed under the runner account
#  • GitHub runner installed in /opt/actions-runner & running as systemd
#  • /opt/hostedtoolcache writable for setup‐* actions
#  • logrotate pruning _diag logs to 3 days

# set -euo pipefail
# ───────────────────────────────────────────────────────────────────────────────
#  -e: exit immediately if any command fails (non-zero)
#  -u: treat any unset variable as an error
#  -o pipefail: if any command in a pipeline fails, the whole pipeline fails
#  → this makes your script “fail fast” on mistakes or missing vars.

#######################################
# 1) Update system & install prerequisites
#######################################
sudo apt update && sudo apt upgrade
sudo apt install -y \
  curl wget git build-essential \
  apt-transport-https ca-certificates gnupg lsb-release \
  libssl-dev zlib1g-dev libreadline-dev \
  autoconf libyaml-dev libffi-dev libgmp-dev rustc

#######################################
# 2) Install Docker Engine & grant ubuntu access
#######################################
# uninstall all conflicting packages
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done

curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
# sudo install -m 0755 -d /etc/apt/keyrings
# sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
# sudo chmod a+r /etc/apt/keyrings/docker.asc

echo \
  "deb [arch=$(dpkg --print-architecture) \
   signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] \
   https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list >/dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Add ubuntu to docker group so it can run docker commands without sudo
sudo usermod -aG docker ubuntu

#######################################
# 3) Prepare runner directories (owned by ubuntu)
#######################################
sudo mkdir -p /opt/actions-runner
sudo chown ubuntu:ubuntu /opt/actions-runner

sudo mkdir -p /opt/hostedtoolcache
sudo chown ubuntu:ubuntu /opt/hostedtoolcache

#######################################
# 5) Install Ruby 3.4.3 & Kamal using mise using ubuntu user
#######################################
# Install mise (Ruby version manager) for this user
curl https://mise.run/bash | sh

# Ensure mise is on PATH in this shell
source ~/.bashrc

# Install & activate Ruby 3.4.3
mise install ruby@3.4.3
mise use --global ruby@3.4.3

# Install the Kamal gem for deployments
gem install kamal

#######################################
# 5) Download & extract GitHub Actions runner
# You can also follow Github instructions
# https://github.com/organizations/jod-app/settings/actions/runners/new?arch=arm64&os=linux
#######################################
# Detect architecture (aarch64→arm64, else x64)
ARCH=$(uname -m)
if [[ "$ARCH" == "aarch64" ]]; then
  RUNNER_ARCH=arm64
else
  RUNNER_ARCH=x64
fi

# Fetch latest release tag (like “v2.327.1”) then strip “v” for filename
TAG=$(curl -s https://api.github.com/repos/actions/runner/releases/latest \
  | grep -Po '"tag_name": "\K[^"]+')
RUNNER_VERSION=${TAG#v}
ASSET="actions-runner-linux-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz"
URL="https://github.com/actions/runner/releases/download/${TAG}/${ASSET}"

# Perform download & extraction
cd /opt/actions-runner
curl -fsSL -o ${ASSET} ${URL}
# Optional: validate checksum
# echo '<SHA256>  ${ASSET}' | sha256sum -c
tar xzf ${ASSET}
rm ${ASSET}

# Ensure ubuntu owns all runner files
sudo chown -R ubuntu:ubuntu /opt/actions-runner

#######################################
# 6) Register the runner with GitHub
#######################################
cd /opt/actions-runner
./config.sh --token REGISTRATION_TOKEN

#######################################
# 7) Install & start as a systemd service
#######################################
# Install service as the ubuntu user
sudo ./svc.sh install ubuntu

# Start and enable on boot
sudo ./svc.sh start
sudo ./svc.sh status

#######################################
# 9) Cron cleanup of old logs & workspaces
#######################################
sudo tee /etc/cron.daily/github-runner-cleanup >/dev/null << 'EOF'
#!/bin/sh
# remove actions-runner logs older than 3 days
find /opt/actions-runner/_diag -maxdepth 1 -type f \
     \( -name '*.log' -o -name '*.log.*' \) -mtime +3 -delete
EOF
# Make it executable
sudo chmod +x /etc/cron.daily/github-runner-cleanup