Operations Wiki

Runbook, release flow, and infrastructure notes.

This page explains how the production instance is laid out, what the GitHub Actions deploy job does, and which commands are useful when something looks wrong.

Topology

Runtime chain

Client
  -> Caddy :443 / :80
  -> reverse_proxy 127.0.0.1:8000
  -> FastAPI container (wordtoken)
  -> Hugging Face cache volume
  -> XLM-RoBERTa + BiLSTM + CRF inference

The public site terminates TLS in Caddy. The API container is only bound on loopback, so traffic reaches it through the reverse proxy and not directly from the internet.

Startup

Model lifecycle

  • The first cold boot downloads roughly 1.1 GB of model artifacts from Hugging Face.
  • Downloaded files are cached in the Docker volume wordtoken_huggingface_cache.
  • The health endpoint stays degraded until the model is fully loaded.
  • Subsequent restarts are much faster because the cache is reused.
Release Flow

GitHub Actions deployment

  1. Push or merge a change into main.
  2. GitHub Actions checks out the repository and runs the test suite.
  3. The workflow syncs the repository to /opt/wordtoken over SSH.
  4. The remote deploy script rebuilds the Docker image and replaces the running container.
  5. The script waits for http://127.0.0.1:8000/health to return successfully before completing.
Repo Secrets

Required GitHub configuration

Name Use
DEPLOY_SSH_KEY Private key used by the workflow to SSH into the production host.
DEPLOY_HOST Production IPv4 or hostname.
DEPLOY_USER SSH user on the target server.
DEPLOY_PORT SSH port, typically 22.
DEPLOY_PATH Remote application directory, usually /opt/wordtoken.
SITE_URL Public health-check target used after deploy, for example https://wordtoken.ygn.app.
Troubleshooting

Useful commands

docker logs --tail=200 wordtoken
docker ps
curl -sS http://127.0.0.1:8000/health
systemctl status caddy
journalctl -u caddy -n 100 --no-pager
Recovery Notes

Common symptoms

  • 502/connection reset: the container is still loading the model or crashed during startup.
  • Health says degraded: check container logs for Hugging Face download errors or missing runtime dependencies.
  • TLS problems: inspect systemctl status caddy and ensure DNS still points to the server.
  • Slow cold boots: keep the Hugging Face cache volume intact between releases.