Initial commit: Open sourcing all of the Maple Open Technologies code.

cloud/infrastructure/production/operations/BACKEND_UPDATES.md

@@ -0,0 +1,196 @@
+# Docker Image Updates & Deployment
+
+**Quick Reference for MapleFile & MaplePress Backend**
+
+## Images
+
+- MapleFile: `registry.digitalocean.com/ssp/maplefile-backend:prod`
+- MaplePress: `registry.digitalocean.com/ssp/maplepress-backend:prod`
+
+## Build & Push
+
+```bash
+cd ~/go/src/codeberg.org/mapleopentech/monorepo/cloud/maplefile-backend
+task deploy
+# Note the Image ID and git commit from output
+docker images registry.digitalocean.com/ssp/maplefile-backend:prod
+```
+
+## Deploy to Production
+
+**CRITICAL**: Docker Swarm caches images. You MUST verify Image IDs match across all nodes.
+
+### Step 1: Note Your Local Image ID
+
+```bash
+docker images registry.digitalocean.com/ssp/maplefile-backend:prod
+# Example: IMAGE ID = 74b2fafb1f69
+```
+
+### Step 2: Find Worker Node & Pull Images
+
+```bash
+# SSH to manager
+ssh dockeradmin@<MANAGER_IP>
+
+# Find which worker runs the service
+docker service ps maplefile_backend
+# Note: NODE column (e.g., mapleopentech-swarm-worker-8-prod)
+
+# Pull on manager
+docker pull registry.digitalocean.com/ssp/maplefile-backend:prod
+
+# Pull on worker
+ssh dockeradmin@<WORKER_NODE>
+docker pull registry.digitalocean.com/ssp/maplefile-backend:prod
+exit
+```
+
+### Step 3: Verify Image IDs Match
+
+```bash
+# On manager
+docker images registry.digitalocean.com/ssp/maplefile-backend:prod
+
+# On worker
+ssh dockeradmin@<WORKER_NODE>
+docker images registry.digitalocean.com/ssp/maplefile-backend:prod
+exit
+
+# ALL THREE (local, manager, worker) must show SAME Image ID
+```
+
+### Step 4: Remove & Recreate Service
+
+```bash
+# On manager - remove service
+docker service rm maplefile_backend
+
+# Redeploy stack
+cd ~/stacks
+docker stack deploy -c maplefile-stack.yml maplefile
+```
+
+### Step 5: Verify Deployment
+
+```bash
+docker service logs maplefile_backend --tail 50
+
+# Confirm these match your build:
+# 🚀 Starting MapleFile Backend v0.1.0
+# 📝 Git Commit: <your-commit-sha>
+# 🕐 Build Time: <your-build-timestamp>
+```
+
+## For MaplePress
+
+Same process, replace `maplefile` with `maplepress`:
+
+```bash
+docker service ps maplepress_backend
+# Pull on both nodes
+docker service rm maplepress_backend
+docker stack deploy -c maplepress-stack.yml maplepress
+```
+
+## Why Remove & Recreate?
+
+Docker Swarm's `docker service update --force` does NOT reliably use new images even after pulling. The `--resolve-image always` and `--with-registry-auth` flags also fail with mutable `:prod` tags.
+
+**Only remove & recreate guarantees the new image is used.**
+
+## Rollback
+
+### Quick Rollback
+
+```bash
+# Automatic rollback to previous version
+docker service rollback maplefile_backend
+```
+
+### Rollback to Specific Version
+
+```bash
+# Find previous image digest
+docker service ps maplefile_backend --no-trunc
+
+# Rollback to specific digest
+docker service update --image registry.digitalocean.com/ssp/maplefile-backend:prod@sha256:def456... maplefile_backend
+```
+
+## Troubleshooting
+
+### Health Check Failures
+
+```bash
+# Check logs
+docker service logs maplefile_backend --tail 100
+
+# Rollback if needed
+docker service rollback maplefile_backend
+```
+
+### Image Pull Authentication Error
+
+```bash
+# Re-authenticate
+doctl registry login
+
+# Retry
+docker service update --image registry.digitalocean.com/ssp/maplefile-backend:prod maplefile_backend
+```
+
+### Service Stuck Starting
+
+```bash
+# Common causes: database migrations failing, missing env vars, health check issues
+# Check logs
+docker service logs maplefile_backend --tail 50
+
+# Rollback if urgent
+docker service rollback maplefile_backend
+```
+
+## Standard Deployment Workflow
+
+```bash
+# 1. Local: Build & push (note the git commit and Image ID)
+cd ~/go/src/codeberg.org/mapleopentech/monorepo/cloud/maplefile-backend
+task deploy
+# Example output: "Deployed version d90b6e2b - use this to verify on production"
+
+# Note the local Image ID for verification
+docker images registry.digitalocean.com/ssp/maplefile-backend:prod
+# Example: IMAGE ID = 74b2fafb1f69
+
+# 2. Find which worker is running the service
+ssh dockeradmin@<MANAGER_IP>
+docker service ps maplefile_backend
+# Note the worker node (e.g., mapleopentech-swarm-worker-8-prod)
+
+# 3. Pull the new image on the MANAGER node
+docker pull registry.digitalocean.com/ssp/maplefile-backend:prod
+
+# 4. Pull the new image on the WORKER node
+ssh dockeradmin@<WORKER_NODE>
+docker pull registry.digitalocean.com/ssp/maplefile-backend:prod
+exit
+
+# 5. Force restart on manager
+ssh dockeradmin@<MANAGER_IP>
+docker service update --force maplefile_backend
+
+# 6. Verify git commit matches what you deployed
+docker service logs maplefile_backend --tail 50
+# Look for: 📝 Git Commit: d90b6e2b...
+```
+
+**Key points**:
+- You MUST pull the image on **BOTH manager and worker nodes**
+- Use `docker images` to verify Image ID matches your local build
+- Use `docker service update --force` to restart with the new image
+- Check startup logs for Git Commit to verify correct version is running
+
+---
+
+**Last Updated**: November 2025