Add initial infrastructure and backup scripts for Gitea and homelab deployment

- Create README.md with project layout and quick start instructions - Implement backup scripts for Gitea, including database and repository exports - Add systemd service and timer for automated Gitea backups - Develop bootstrap scripts for homelab and VPS setup - Document architecture and restore procedures - Configure Caddy reverse proxy and Docker Compose for service management - Establish secrets management guidelines
2026-03-04 14:42:46 -05:00
commit c93dcb5daf
21 changed files with 531 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,37 @@
+# ops
+
+Infrastructure-as-code repo for reproducible VPS edge + homelab deployment.
+
+## Layout
+
+- `bootstrap/` host bootstrap scripts
+- `edge/caddy/` VPS edge reverse proxy stack
+- `stacks/` app stack modules
+- `backups/` backup + restore scripts
+- `secrets/` encrypted secret placeholders and guidance
+- `docs/` architecture and restore runbooks
+
+## Quick Start
+
+1. Fill `secrets/*.age` with encrypted values.
+2. Update domain/IP placeholders in `edge/caddy/Caddyfile`.
+3. Copy `.env.example` files to `.env` per stack.
+4. Run bootstrap scripts on target hosts.
+
+## Ops Repo Source Strategy
+
+When the ops repo lives on self-hosted Gitea, bootstrap should not depend on one source.
+
+- Primary source: self-hosted Gitea URL (`OPS_REPO_PRIMARY_URL`)
+- Optional mirror source: secondary git host (`OPS_REPO_FALLBACK_URL`)
+- Last-resort source: local bundle file (`OPS_BUNDLE_PATH`, default `/srv/backups/ops/latest/ops.bundle`)
+
+The bootstrap scripts automatically try those in that order.
+
+Current default primary URL is `https://git.sketchferret.com/sketchferret/ops.git`.
+
+## Retention Policy
+
+- Keep all backup files for 7 days (daily recovery points).
+- From day 8 to day 365, keep one backup per ISO week per folder.
+- Keep files under any `latest/` folder (for bootstrap fallback pointers).
--- a/backups/scripts/gitea-dump.sh
+++ b/backups/scripts/gitea-dump.sh
@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+TS="$(date +%F_%H%M%S)"
+OUT="/srv/backups/gitea/dumps"
+mkdir -p "$OUT"
+
+docker exec gitea sh -lc "gitea dump -c /data/gitea/conf/app.ini -f /tmp/gitea-dump-$TS.zip"
+docker cp "gitea:/tmp/gitea-dump-$TS.zip" "$OUT/"
+docker exec gitea rm -f "/tmp/gitea-dump-$TS.zip"
+
+echo "Wrote: $OUT/gitea-dump-$TS.zip"
--- a/backups/scripts/gitea-mirror-export.sh
+++ b/backups/scripts/gitea-mirror-export.sh
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+GITEA_URL="${GITEA_URL:-https://git.sketchferret.com}"
+TOKEN_FILE="${TOKEN_FILE:-/srv/secrets/gitea_token}"
+OUT="${OUT:-/srv/backups/git-mirrors}"
+OWNER="${OWNER:-spencer}"
+
+if [[ ! -f "$TOKEN_FILE" ]]; then
+  echo "Missing token file: $TOKEN_FILE"
+  exit 1
+fi
+
+TOKEN="$(cat "$TOKEN_FILE")"
+mkdir -p "$OUT/$OWNER"
+
+repos_json="$(curl -fsSL -H "Authorization: token $TOKEN" "$GITEA_URL/api/v1/users/$OWNER/repos?limit=1000")"
+
+mapfile -t urls < <(python3 - <<'PY' "$repos_json"
+import json,sys
+for repo in json.loads(sys.argv[1]):
+    print(repo["clone_url"])
+PY
+)
+
+for url in "${urls[@]}"; do
+  name="$(basename "$url" .git)"
+  target="$OUT/$OWNER/$name.git"
+
+  auth_url="$url"
+  if [[ "$url" == https://* ]]; then
+    auth_url="https://${TOKEN}:x-oauth-basic@${url#https://}"
+  fi
+
+  if [[ -d "$target" ]]; then
+    git -C "$target" remote set-url origin "$auth_url"
+    git -C "$target" fetch --prune
+  else
+    git clone --mirror "$auth_url" "$target"
+  fi
+
+done
+
+echo "Mirror export complete: $OUT/$OWNER"
--- a/backups/scripts/ops-bundle.sh
+++ b/backups/scripts/ops-bundle.sh
@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+OPS_REPO_PATH="${OPS_REPO_PATH:-/srv/ops}"
+OUT_BASE="${OUT_BASE:-/srv/backups/ops}"
+TS="$(date +%F_%H%M%S)"
+OUT_DIR="$OUT_BASE/$TS"
+LATEST_DIR="$OUT_BASE/latest"
+
+if [[ ! -d "$OPS_REPO_PATH/.git" ]]; then
+  echo "Missing git repo at $OPS_REPO_PATH"
+  exit 1
+fi
+
+mkdir -p "$OUT_DIR"
+git -C "$OPS_REPO_PATH" bundle create "$OUT_DIR/ops.bundle" --all
+
+mkdir -p "$LATEST_DIR"
+cp "$OUT_DIR/ops.bundle" "$LATEST_DIR/ops.bundle"
+
+echo "Wrote bundle: $OUT_DIR/ops.bundle"
+echo "Updated latest: $LATEST_DIR/ops.bundle"
--- a/backups/scripts/pg-dump.sh
+++ b/backups/scripts/pg-dump.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+TS="$(date +%F_%H%M%S)"
+OUT="/srv/backups/gitea/pg"
+mkdir -p "$OUT"
+
+PGUSER="${PGUSER:-gitea}"
+PGDATABASE="${PGDATABASE:-gitea}"
+PGPASSFILE="${PGPASSFILE:-/srv/secrets/postgres_password}"
+
+if [[ ! -f "$PGPASSFILE" ]]; then
+  echo "Missing $PGPASSFILE"
+  exit 1
+fi
+
+export PGPASSWORD="$(cat "$PGPASSFILE")"
+docker exec -e PGPASSWORD="$PGPASSWORD" gitea-db sh -lc "pg_dump -U '$PGUSER' -d '$PGDATABASE'" > "$OUT/gitea-pg-$TS.sql"
+
+echo "Wrote: $OUT/gitea-pg-$TS.sql"
--- a/backups/scripts/restore-gitea.md
+++ b/backups/scripts/restore-gitea.md
@@ -0,0 +1,14 @@
+# Gitea Restore (Postgres)
+
+1. Stop gitea stack:
+   - `cd /srv/ops/stacks/gitea && docker compose down`
+2. Restore filesystem data to:
+   - `/srv/data/gitea`
+   - `/srv/data/gitea-postgres` (or restore logical SQL below)
+3. Start only database:
+   - `cd /srv/ops/stacks/gitea && docker compose up -d gitea-db`
+4. Import SQL dump (if using logical dump):
+   - `cat /srv/backups/gitea/pg/<dump>.sql | docker exec -i gitea-db psql -U gitea -d gitea`
+5. Start gitea:
+   - `docker compose up -d gitea`
+6. Validate web + SSH endpoints.
--- a/backups/scripts/retention.sh
+++ b/backups/scripts/retention.sh
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+BASE="${BASE:-/srv/backups}"
+KEEP_DAILY_DAYS="${KEEP_DAILY_DAYS:-7}"
+KEEP_WEEKLY_DAYS="${KEEP_WEEKLY_DAYS:-365}"
+
+python3 - <<'PY' "$BASE" "$KEEP_DAILY_DAYS" "$KEEP_WEEKLY_DAYS"
+import datetime as dt
+import os
+import sys
+
+base = sys.argv[1]
+keep_daily_days = int(sys.argv[2])
+keep_weekly_days = int(sys.argv[3])
+now = dt.datetime.now().timestamp()
+
+if not os.path.isdir(base):
+	print(f"Backup base not found: {base}")
+	sys.exit(0)
+
+files = []
+for root, _, names in os.walk(base):
+	parts = set(os.path.normpath(root).split(os.sep))
+	preserve_latest = "latest" in parts
+	for name in names:
+		path = os.path.join(root, name)
+		if not os.path.isfile(path):
+			continue
+		try:
+			mtime = os.path.getmtime(path)
+		except OSError:
+			continue
+		age_days = (now - mtime) / 86400
+		files.append((path, root, mtime, age_days, preserve_latest))
+
+to_delete = set()
+
+# Delete anything older than weekly window (unless it's in a latest folder)
+for path, _, _, age_days, preserve_latest in files:
+	if not preserve_latest and age_days > keep_weekly_days:
+		to_delete.add(path)
+
+# For weekly window, keep one file per week per directory (latest by mtime)
+weekly_candidates = {}
+for path, root, mtime, age_days, preserve_latest in files:
+	if preserve_latest:
+		continue
+	if keep_daily_days < age_days <= keep_weekly_days:
+		iso = dt.datetime.fromtimestamp(mtime).isocalendar()
+		key = (root, iso.year, iso.week)
+		best = weekly_candidates.get(key)
+		if best is None or mtime > best[1]:
+			weekly_candidates[key] = (path, mtime)
+
+weekly_keep = {v[0] for v in weekly_candidates.values()}
+
+for path, _, _, age_days, preserve_latest in files:
+	if preserve_latest:
+		continue
+	if keep_daily_days < age_days <= keep_weekly_days and path not in weekly_keep:
+		to_delete.add(path)
+
+deleted = 0
+for path in sorted(to_delete):
+	try:
+		os.remove(path)
+		print(path)
+		deleted += 1
+	except OSError:
+		pass
+
+print(f"Retention complete for {base} (deleted {deleted} files)")
+PY
+
+echo "Policy: daily <= ${KEEP_DAILY_DAYS}d, weekly <= ${KEEP_WEEKLY_DAYS}d"
--- a/backups/systemd/gitea-backup.service
+++ b/backups/systemd/gitea-backup.service
@@ -0,0 +1,7 @@
+[Unit]
+Description=Run Gitea backup scripts
+After=docker.service
+
+[Service]
+Type=oneshot
+ExecStart=/bin/bash -lc '/srv/ops/backups/scripts/gitea-dump.sh && /srv/ops/backups/scripts/pg-dump.sh && /srv/ops/backups/scripts/gitea-mirror-export.sh && /srv/ops/backups/scripts/ops-bundle.sh && /srv/ops/backups/scripts/retention.sh'
--- a/backups/systemd/gitea-backup.timer
+++ b/backups/systemd/gitea-backup.timer
@@ -0,0 +1,9 @@
+[Unit]
+Description=Nightly Gitea backup timer
+
+[Timer]
+OnCalendar=*-*-* 02:30:00
+Persistent=true
+
+[Install]
+WantedBy=timers.target
--- a/bootstrap/homelab.sh
+++ b/bootstrap/homelab.sh
@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+OPS_REPO_PRIMARY_URL="${OPS_REPO_PRIMARY_URL:-https://git.sketchferret.com/sketchferret/ops.git}"
+OPS_REPO_FALLBACK_URL="${OPS_REPO_FALLBACK_URL:-}"
+OPS_BUNDLE_PATH="${OPS_BUNDLE_PATH:-/srv/backups/ops/latest/ops.bundle}"
+TS_HOSTNAME="${TS_HOSTNAME:-homelab}"
+
+echo "[1/5] Install packages"
+apt-get update
+apt-get install -y ca-certificates curl git ufw docker.io docker-compose-plugin age
+
+echo "[2/5] Install tailscale"
+curl -fsSL https://tailscale.com/install.sh | sh
+
+echo "[3/5] Prepare directories"
+mkdir -p /srv/{ops,secrets,data,backups}
+chmod 700 /srv/secrets
+
+echo "[4/5] Sync ops repo"
+if [[ ! -d /srv/ops/.git ]]; then
+  if git clone "$OPS_REPO_PRIMARY_URL" /srv/ops; then
+    echo "Cloned ops from primary"
+  elif [[ -n "$OPS_REPO_FALLBACK_URL" ]] && git clone "$OPS_REPO_FALLBACK_URL" /srv/ops; then
+    echo "Cloned ops from fallback mirror"
+  elif [[ -f "$OPS_BUNDLE_PATH" ]]; then
+    rm -rf /srv/ops
+    mkdir -p /srv/ops
+    git clone "$OPS_BUNDLE_PATH" /srv/ops
+    echo "Cloned ops from local bundle: $OPS_BUNDLE_PATH"
+  else
+    echo "Unable to fetch ops repo from primary, fallback, or bundle"
+    exit 1
+  fi
+else
+  if ! git -C /srv/ops pull --ff-only; then
+    if [[ -n "$OPS_REPO_FALLBACK_URL" ]]; then
+      git -C /srv/ops remote set-url origin "$OPS_REPO_FALLBACK_URL"
+      git -C /srv/ops pull --ff-only || true
+    fi
+  fi
+fi
+
+echo "[5/5] Bring up tailscale"
+if [[ -f /srv/ops/secrets/tailscale_authkey.age ]]; then
+  if [[ ! -f /srv/secrets/ops.agekey ]]; then
+    echo "Missing /srv/secrets/ops.agekey (age private key)"
+    exit 1
+  fi
+  age -d -i /srv/secrets/ops.agekey -o /srv/secrets/tailscale_authkey /srv/ops/secrets/tailscale_authkey.age
+  chmod 600 /srv/secrets/tailscale_authkey
+  tailscale up --authkey="$(cat /srv/secrets/tailscale_authkey)" --hostname="$TS_HOSTNAME"
+else
+  echo "tailscale_authkey.age not found; run tailscale up manually"
+fi
+
+echo "Done: Homelab bootstrap complete"
--- a/bootstrap/vps.sh
+++ b/bootstrap/vps.sh
@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+OPS_REPO_PRIMARY_URL="${OPS_REPO_PRIMARY_URL:-https://git.sketchferret.com/sketchferret/ops.git}"
+OPS_REPO_FALLBACK_URL="${OPS_REPO_FALLBACK_URL:-}"
+OPS_BUNDLE_PATH="${OPS_BUNDLE_PATH:-/srv/backups/ops/latest/ops.bundle}"
+TS_HOSTNAME="${TS_HOSTNAME:-vps-edge}"
+
+echo "[1/6] Install packages"
+apt-get update
+apt-get install -y ca-certificates curl git ufw docker.io docker-compose-plugin age
+
+echo "[2/6] Install tailscale"
+curl -fsSL https://tailscale.com/install.sh | sh
+
+echo "[3/6] Configure firewall"
+ufw allow OpenSSH
+ufw allow 80/tcp
+ufw allow 443/tcp
+ufw --force enable
+
+echo "[4/6] Prepare directories"
+mkdir -p /srv/{ops,secrets,data,backups}
+chmod 700 /srv/secrets
+
+echo "[5/6] Sync ops repo"
+if [[ ! -d /srv/ops/.git ]]; then
+  if git clone "$OPS_REPO_PRIMARY_URL" /srv/ops; then
+    echo "Cloned ops from primary"
+  elif [[ -n "$OPS_REPO_FALLBACK_URL" ]] && git clone "$OPS_REPO_FALLBACK_URL" /srv/ops; then
+    echo "Cloned ops from fallback mirror"
+  elif [[ -f "$OPS_BUNDLE_PATH" ]]; then
+    rm -rf /srv/ops
+    mkdir -p /srv/ops
+    git clone "$OPS_BUNDLE_PATH" /srv/ops
+    echo "Cloned ops from local bundle: $OPS_BUNDLE_PATH"
+  else
+    echo "Unable to fetch ops repo from primary, fallback, or bundle"
+    exit 1
+  fi
+else
+  if ! git -C /srv/ops pull --ff-only; then
+    if [[ -n "$OPS_REPO_FALLBACK_URL" ]]; then
+      git -C /srv/ops remote set-url origin "$OPS_REPO_FALLBACK_URL"
+      git -C /srv/ops pull --ff-only || true
+    fi
+  fi
+fi
+
+echo "[6/6] Bring up tailscale and caddy"
+if [[ -f /srv/ops/secrets/tailscale_authkey.age ]]; then
+  if [[ ! -f /srv/secrets/ops.agekey ]]; then
+    echo "Missing /srv/secrets/ops.agekey (age private key)"
+    exit 1
+  fi
+  age -d -i /srv/secrets/ops.agekey -o /srv/secrets/tailscale_authkey /srv/ops/secrets/tailscale_authkey.age
+  chmod 600 /srv/secrets/tailscale_authkey
+  tailscale up --authkey="$(cat /srv/secrets/tailscale_authkey)" --hostname="$TS_HOSTNAME" --ssh
+else
+  echo "tailscale_authkey.age not found; run tailscale up manually"
+fi
+
+cd /srv/ops/edge/caddy
+docker compose up -d
+
+echo "Done: VPS bootstrap complete"
--- a/deploy.sh
+++ b/deploy.sh
@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+if [[ "$#" -eq 0 ]]; then
+  echo "Usage: $0 <stack> [stack...]"
+  echo "Example: $0 gitea kuma"
+  exit 1
+fi
+
+for stack in "$@"; do
+  stack_dir="/srv/ops/stacks/$stack"
+  if [[ ! -d "$stack_dir" ]]; then
+    echo "Skipping unknown stack: $stack"
+    continue
+  fi
+
+  if [[ -f "$stack_dir/.env.example" && ! -f "$stack_dir/.env" ]]; then
+    cp "$stack_dir/.env.example" "$stack_dir/.env"
+    echo "Created $stack_dir/.env from .env.example; fill secrets before production"
+  fi
+
+  echo "Deploying stack: $stack"
+  (cd "$stack_dir" && docker compose up -d)
+done
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -0,0 +1,32 @@
+# Architecture
+
+## Model
+
+- VPS is public edge (Caddy + VPN client).
+- Homelab hosts internal application stacks.
+- Traffic path: Internet -> VPS Caddy -> VPN -> homelab service.
+
+## State Conventions
+
+- `/srv/ops` cloned repo
+- `/srv/secrets` decrypted runtime secrets (not committed)
+- `/srv/data/<stack>` persistent bind mounts
+- `/srv/backups` backup artifacts
+
+## Deployment Order
+
+1. Edge bootstrap on VPS
+2. Homelab bootstrap
+3. Bring up proxy/network dependencies
+4. Bring up core stacks (gitea, db)
+5. Bring up secondary stacks (kuma, apps)
+
+## Bootstrap Paradox Mitigation
+
+Because ops is hosted on Gitea inside the homelab, bootstrap uses three repo sources:
+
+1. Primary Gitea repo
+2. Optional fallback mirror (secondary git host)
+3. Local git bundle backup (`/srv/backups/ops/latest/ops.bundle`)
+
+Nightly backups include both full Gitea backups and standalone repo exports/bundles.
--- a/docs/RESTORE.md
+++ b/docs/RESTORE.md
@@ -0,0 +1,25 @@
+# Restore Runbook
+
+## VPS Restore
+
+1. Provision host and SSH access.
+2. Ensure `ops.bundle` exists at `/srv/backups/ops/latest/ops.bundle` (or set `OPS_BUNDLE_PATH`).
+3. Run `bootstrap/vps.sh`.
+4. Confirm VPN up and Caddy healthy.
+5. Validate DNS + TLS endpoints.
+
+## Homelab Restore
+
+1. Provision host and SSH access.
+2. Ensure `ops.bundle` exists at `/srv/backups/ops/latest/ops.bundle` (or set `OPS_BUNDLE_PATH`).
+3. Run `bootstrap/homelab.sh`.
+4. Restore data under `/srv/data/*` and `/srv/backups/*` as needed.
+5. Start stacks with `docker compose up -d` per stack.
+6. Run health checks and verify service endpoints.
+
+## Data Priorities
+
+- Gitea app data + DB dump
+- Repo mirror exports
+- Proxy config and certificates
+- Encrypted secret source files
--- a/edge/caddy/Caddyfile
+++ b/edge/caddy/Caddyfile
@@ -0,0 +1,15 @@
+git.sketchferret.com {
+  reverse_proxy 100.115.54.124:4445
+}
+
+home.sketchferret.com {
+  reverse_proxy 100.115.54.124:5300
+}
+
+esphome.sketchferret.com {
+  reverse_proxy 100.115.54.124:6052
+}
+
+calfill.sketchferret.com {
+  reverse_proxy 100.115.54.124:5300
+}
--- a/edge/caddy/compose.yml
+++ b/edge/caddy/compose.yml
@@ -0,0 +1,15 @@
+services:
+  caddy:
+    image: caddy:2.8.4
+    restart: unless-stopped
+    ports:
+      - "80:80"
+      - "443:443"
+    volumes:
+      - ./Caddyfile:/etc/caddy/Caddyfile:ro
+      - caddy_data:/data
+      - caddy_config:/config
+
+volumes:
+  caddy_data:
+  caddy_config:
--- a/secrets/README.md
+++ b/secrets/README.md
@@ -0,0 +1,15 @@
+# Secrets
+
+Do not commit plaintext secrets.
+
+## Pattern
+
+- Commit encrypted blobs only (`*.age`).
+- Decrypt to `/srv/secrets/*` at bootstrap/runtime.
+- Keep private decryption key outside git.
+
+## Expected encrypted files
+
+- `tailscale_authkey.age`
+- `gitea_token.age` (optional)
+- `postgres_password.age` (optional)
--- a/stacks/gitea/.env.example
+++ b/stacks/gitea/.env.example
@@ -0,0 +1,3 @@
+POSTGRES_DB=gitea
+POSTGRES_USER=gitea
+POSTGRES_PASSWORD=REPLACE_ME
--- a/stacks/gitea/compose.yml
+++ b/stacks/gitea/compose.yml
@@ -0,0 +1,29 @@
+services:
+  gitea:
+    image: gitea/gitea:1.23.8
+    restart: unless-stopped
+    environment:
+      - USER_UID=1000
+      - USER_GID=1000
+      - GITEA__database__DB_TYPE=postgres
+      - GITEA__database__HOST=gitea-db:5432
+      - GITEA__database__NAME=${POSTGRES_DB}
+      - GITEA__database__USER=${POSTGRES_USER}
+      - GITEA__database__PASSWD=${POSTGRES_PASSWORD}
+    ports:
+      - "4445:3000"
+      - "2222:2222"
+    volumes:
+      - /srv/data/gitea:/data
+    depends_on:
+      - gitea-db
+
+  gitea-db:
+    image: postgres:14.15
+    restart: unless-stopped
+    environment:
+      - POSTGRES_DB=${POSTGRES_DB}
+      - POSTGRES_USER=${POSTGRES_USER}
+      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
+    volumes:
+      - /srv/data/gitea-postgres:/var/lib/postgresql/data
--- a/stacks/kuma/.env.example
+++ b/stacks/kuma/.env.example
@@ -0,0 +1 @@
+# no required vars by default
--- a/stacks/kuma/compose.yml
+++ b/stacks/kuma/compose.yml
@@ -0,0 +1,8 @@
+services:
+  kuma:
+    image: louislam/uptime-kuma:1.23.16
+    restart: unless-stopped
+    ports:
+      - "3001:3001"
+    volumes:
+      - /srv/data/kuma:/app/data