# Deploy MaplePress Backend: Part 1

**Audience**: Junior DevOps Engineers, Infrastructure Team
**Time to Complete**: 60-90 minutes
**Prerequisites**:
- Completed guides 01-04.5 (Swarm, Cassandra, Redis, Meilisearch, Spaces)
- Backend Docker image ready to deploy
- Domain name `getmaplepress.ca` configured
- DigitalOcean API token for registry access
- DigitalOcean Spaces configured (from 04.5_spaces.md)

---

## Overview

This guide sets up **worker-6** from scratch and deploys the MaplePress backend. Part 2 (06_caddy.md) will add the Caddy reverse proxy.

**What you'll build:**
- Fresh worker-6 droplet with Docker and dockeradmin user
- Worker-6 joined to existing Docker Swarm
- Backend service connected to both networks (private databases + public reverse proxy)
- Backend service ready for Caddy reverse proxy (deployed in Part 2)
- Pull backend image from DigitalOcean Container Registry

**Architecture:**
```
Internet (HTTPS) → Caddy (worker-6) → Backend (worker-6) → Cassandra/Redis/Meilisearch (private network)
                    [Part 2: 06_caddy.md]    [Part 1: This guide]
```

---

## Step 1: Create Worker-6 Droplet

### 1.1 Create Droplet in DigitalOcean

1. Log into DigitalOcean dashboard
2. Click **Create** → **Droplets**
3. Configure:
   - **Region**: Same as your other workers (e.g., Toronto)
   - **Image**: Ubuntu 24.04 LTS x64
   - **Size**: Basic shared CPU, 2 GB / 2 vCPU ($18/mo)
   - **Hostname**: `mapleopentech-swarm-worker-6-prod`
   - **VPC Network**: Select same VPC as your swarm (mapleopentech-vpc-prod)
   - **SSH Keys**: Add your SSH key
4. Click **Create Droplet**
5. Wait 1-2 minutes for droplet to provision

### 1.2 Record IP Addresses

Once created, copy both IPs:
- **Public IPv4**: `<worker-6-public-ip>` (e.g., 157.230.45.67)
- **Private IPv4**: `<worker-6-private-ip>` (e.g., 10.116.0.8)

**Update your local `.env` file:**

```bash
# On your local machine
cd ~/monorepo/cloud/infrastructure/production
vi .env

# Add these lines:
WORKER_6_PUBLIC_IP=157.230.45.67
WORKER_6_PRIVATE_IP=10.116.0.8
```

Save: `Ctrl+O`, `Enter`, `Ctrl+X`

**✅ Checkpoint:** You should be able to ping worker-6:

```bash
ping <worker-6-public-ip>
# Should get responses
```

---

## Step 2: Initial Server Setup

### 2.1 SSH to Worker-6

```bash
# From your local machine
ssh root@<worker-6-public-ip>

# Should connect successfully
```

### 2.2 Update System Packages

```bash
# Update package lists
apt update

# Upgrade all packages
apt upgrade -y

# Install essential tools
apt install -y curl wget git vim apt-transport-https ca-certificates gnupg lsb-release software-properties-common
```

This takes 2-5 minutes.

### 2.3 Install Docker

```bash
# Add Docker GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Add Docker repository
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

# Update package list with Docker packages
apt update

# Install Docker
apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

# Verify installation
docker --version
# Should show: Docker version 27.x.x or higher

# Check Docker is running
docker ps
# Should show empty list (not error)
```

**✅ Checkpoint:** `docker --version` shows version 27+

### 2.4 Create dockeradmin User

```bash
# Create user
adduser dockeradmin
# Enter password when prompted
# SAVE THIS PASSWORD IN YOUR PASSWORD MANAGER!

# Add to sudo group (can run admin commands)
usermod -aG sudo dockeradmin

# Add to docker group (can run docker commands)
usermod -aG docker dockeradmin

# Copy SSH keys to new user
rsync --archive --chown=dockeradmin:dockeradmin ~/.ssh /home/dockeradmin

# Verify SSH keys copied
ls -la /home/dockeradmin/.ssh/
# Should show: authorized_keys
```

### 2.5 Test dockeradmin Access

**From your local machine (open new terminal):**

```bash
# Test SSH login
ssh dockeradmin@<worker-6-public-ip>

# Should login WITHOUT password prompt (using SSH key)

# Test docker access
docker ps
# Should show empty list (NOT permission denied)

# Test sudo access
sudo ls /root
# Enter dockeradmin password when prompted
# Should list root directory contents

# Exit back to local machine
exit
```

**✅ Checkpoint:** Can SSH as dockeradmin without password, run docker commands

---

## Step 3: Configure Firewall

**SSH back to worker-6 as root:**

```bash
ssh root@<worker-6-public-ip>
```

### 3.1 Setup UFW Firewall

```bash
# Enable firewall (force to avoid prompt)
ufw --force enable

# Allow SSH (CRITICAL - do this first!)
ufw allow 22/tcp

# Allow HTTP and HTTPS (for NGINX)
ufw allow 80/tcp
ufw allow 443/tcp

# Allow Docker Swarm (only from private VPC network)
ufw allow from 10.116.0.0/16 to any port 2377 proto tcp  # Swarm management
ufw allow from 10.116.0.0/16 to any port 7946            # Node communication
ufw allow from 10.116.0.0/16 to any port 4789 proto udp  # Overlay network

# Check firewall status
ufw status verbose
```

**Expected output:**

```
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
2377/tcp                   ALLOW       10.116.0.0/16
7946                       ALLOW       10.116.0.0/16
4789/udp                   ALLOW       10.116.0.0/16
```

**✅ Checkpoint:** UFW active, ports open correctly

---

## Step 4: Join Worker-6 to Docker Swarm

### 4.1 Get Swarm Join Token

**From your local machine, SSH to manager:**

```bash
ssh dockeradmin@<manager-public-ip>

# Get worker join token
docker swarm join-token worker
```

**Copy the entire output command.** It looks like:

```bash
docker swarm join --token SWMTKN-1-xxxxxx... 10.116.0.2:2377
```

**Important:** Use the **private IP** shown in the command (e.g., `10.116.0.2:2377`), NOT the public IP.

### 4.2 Join Worker-6 to Swarm

**SSH to worker-6 as dockeradmin:**

```bash
# From your local machine
ssh dockeradmin@<worker-6-public-ip>

# Paste the join command from manager
docker swarm join --token SWMTKN-1-xxxxxx... <manager-private-ip>:2377
```

**Expected output:**

```
This node joined a swarm as a worker.
```

### 4.3 Verify Worker-6 Joined

**Back on manager:**

```bash
# List all nodes
docker node ls
```

**You should see:**

```
ID                            HOSTNAME                            STATUS    AVAILABILITY   MANAGER STATUS
abc123...                     mapleopentech-swarm-manager-1-prod      Ready     Active         Leader
def456...                     mapleopentech-swarm-worker-1-prod       Ready     Active
...
xyz789...                     mapleopentech-swarm-worker-6-prod       Ready     Active         <-- NEW!
```

**✅ Checkpoint:** Worker-6 appears in `docker node ls` with status `Ready Active`

### 4.4 Label Worker-6 for Backend

**On manager:**

```bash
# Add backend label (tells swarm to deploy backend here)
docker node update --label-add backend=true mapleopentech-swarm-worker-6-prod

# Verify label
docker node inspect mapleopentech-swarm-worker-6-prod --format '{{.Spec.Labels}}'
# Should show: map[backend:true]
```

**✅ Checkpoint:** Worker-6 has `backend=true` label

---

## Step 5: Configure DNS

Before deploying, point your domain to worker-6.

### 5.1 Update DNS Records

1. Log into your domain registrar (where you bought getmaplepress.ca)
2. Find DNS settings / DNS records
3. Add/update these A records:

| Type | Name | Value | TTL |
|------|------|-------|-----|
| A | @ | `<worker-6-public-ip>` | 3600 |
| A | www | `<worker-6-public-ip>` | 3600 |

Replace `<worker-6-public-ip>` with your actual public IP (e.g., 157.230.45.67)

Example:
- **@ record**: Points `getmaplepress.ca` → 157.230.45.67
- **www record**: Points `www.getmaplepress.ca` → 157.230.45.67

### 5.2 Wait for DNS Propagation

DNS changes take 5-10 minutes (sometimes up to 1 hour).

**Test from your local machine:**

```bash
# Test root domain
dig getmaplepress.ca +short
# Should return: <worker-6-public-ip>

# Test www subdomain
dig www.getmaplepress.ca +short
# Should return: <worker-6-public-ip>
```

**⚠️ Don't proceed until both commands return the correct IP!**

**✅ Checkpoint:** DNS resolves to worker-6 public IP

---

## Step 6: Create Docker Networks

We need two overlay networks for our services:

1. **mapleopentech-private-prod** - Backend connects to databases (already exists from guides 02-04)
2. **mapleopentech-public-prod** - NGINX and Backend communicate (new)

**On manager:**

```bash
ssh dockeradmin@<manager-public-ip>

# Check private network exists
docker network ls | grep mapleopentech-private-prod
# Should show: mapleopentech-private-prod (created in previous guides)

# Create public network
docker network create --driver overlay --attachable mapleopentech-public-prod

# Verify both exist
docker network ls | grep maple
```

**Expected output:**

```
abc123...   mapleopentech-private-prod   overlay   swarm
def456...   mapleopentech-public-prod    overlay   swarm
```

**Why two networks?**
- **Private**: Backend talks to Cassandra, Redis, Meilisearch (no internet access)
- **Public**: NGINX forwards requests to Backend (internet-facing)
- Backend joins BOTH networks to receive requests and access databases

**✅ Checkpoint:** Both `mapleopentech-private-prod` and `mapleopentech-public-prod` networks exist

---

## Step 7: Authenticate with DigitalOcean Registry

Worker-6 needs to authenticate with DO registry to pull your private backend image.

### 7.1 Install doctl on Worker-6

**SSH to worker-6 as dockeradmin:**

```bash
ssh dockeradmin@<worker-6-public-ip>

# Download doctl
cd ~
wget https://github.com/digitalocean/doctl/releases/download/v1.98.1/doctl-1.98.1-linux-amd64.tar.gz

# Extract
tar xf doctl-*.tar.gz

# Move to system path
sudo mv doctl /usr/local/bin

# Clean up
rm doctl-*.tar.gz

# Verify installation
doctl version
# Should show: doctl version 1.98.1
```

### 7.2 Authenticate doctl

**You need your DigitalOcean API token.** Get it from:
1. DigitalOcean dashboard → API → Tokens/Keys
2. Click "Generate New Token" if you don't have one
3. Copy the token (looks like: `dop_v1_xxxxx...`)

**On worker-6:**

```bash
# Initialize doctl with your API token
doctl auth init
# Paste your API token when prompted
```

**Expected output:**

```
Please authenticate doctl for use with your DigitalOcean account. You can generate a token in the control panel at https://cloud.digitalocean.com/account/api/tokens

Enter your access token: dop_v1_xxxxx...

Validating token... OK
```

### 7.3 Login to Registry

```bash
# Login to DigitalOcean Container Registry
doctl registry login
```

**Expected output:**

```
Logging Docker in to registry.digitalocean.com
Login Succeeded
```

### 7.4 Test Image Pull

```bash
# Try pulling your backend image
docker pull registry.digitalocean.com/ssp/maplepress_backend:prod
```

**If you get "manifest not found" error:**

Check what tags exist:

```bash
# List available tags
doctl registry repository list-tags maplepress_backend
```

**If no `:prod` tag exists:**

You need to rebuild and push your image with the correct tag:

```bash
# From your local machine
cd ~/monorepo/cloud/maplepress-backend

# Build and push (this creates :prod tag)
task deploy
```

Then retry the pull on worker-6.

**✅ Checkpoint:** `docker images | grep maplepress` shows the backend image

---

## Step 8: Create Docker Secrets

**On manager:**

```bash
ssh dockeradmin@<manager-public-ip>
```

**Important:** Before proceeding, you should have completed **04.5_spaces.md** which creates the `spaces_access_key` and `spaces_secret_key` Docker secrets. If you haven't done that guide yet, the Spaces secrets will be missing and you'll need to create them manually in Step 8.3.

### 8.1 Generate and Create JWT Secret

```bash
# Generate JWT secret (base64 encoded, 64 characters)
JWT_SECRET=$(openssl rand -base64 64 | tr -d '\n')

# SAVE THIS! Print to screen
echo "JWT Secret: $JWT_SECRET"
# Copy this to your password manager!

# Create Docker secret (use -n to avoid adding newline)
echo -n "$JWT_SECRET" | docker secret create maplepress_jwt_secret -
```

### 8.2 Generate and Create IP Encryption Key

```bash
# Generate IP encryption key (32 hex characters)
IP_ENCRYPTION_KEY=$(openssl rand -hex 16)

# SAVE THIS! Print to screen
echo "IP Encryption Key: $IP_ENCRYPTION_KEY"
# Copy this to your password manager!

# Create Docker secret
echo $IP_ENCRYPTION_KEY | docker secret create maplepress_ip_encryption_key -
```

### 8.3 Verify All Secrets Exist

```bash
# List all secrets
docker secret ls
```

**You should see:**

```
ID                          NAME                              CREATED
abc123...                   maplepress_jwt_secret             1 second ago
abc124...                   maplepress_ip_encryption_key      1 second ago
def456...                   redis_password                    from guide 03
ghi789...                   meilisearch_master_key            from guide 04
jkl012...                   spaces_access_key                 from guide 04.5
mno345...                   spaces_secret_key                 from guide 04.5
```

**If redis_password, meilisearch_master_key, or Spaces secrets are missing:**

Create them now:

```bash
# Redis password (if missing)
REDIS_PASSWORD=$(openssl rand -base64 32 | tr -d "=+/" | cut -c1-32)
echo "Redis Password: $REDIS_PASSWORD"  # SAVE THIS!
echo $REDIS_PASSWORD | docker secret create redis_password -

# Meilisearch key (if missing)
MEILI_KEY=$(openssl rand -base64 32 | tr -d "=+/" | cut -c1-32)
echo "Meilisearch Key: $MEILI_KEY"  # SAVE THIS!
echo $MEILI_KEY | docker secret create meilisearch_master_key -

# Spaces secrets (if missing)
# These should have been created in 04.5_spaces.md
# If not, create them now with your DigitalOcean Spaces credentials

# Replace with your actual Spaces access key from DigitalOcean dashboard
echo -n "DO00ABC123XYZ456" | docker secret create spaces_access_key -

# Replace with your actual Spaces secret key from DigitalOcean dashboard
echo -n "abc123def456ghi789jkl012mno345pqr678stu901" | docker secret create spaces_secret_key -

# Verify they were created
docker secret ls | grep spaces
# Should show: spaces_access_key and spaces_secret_key
```

**✅ Checkpoint:** All six secrets exist: `maplepress_jwt_secret`, `maplepress_ip_encryption_key`, `redis_password`, `meilisearch_master_key`, `spaces_access_key`, `spaces_secret_key`

---

## Step 9: Deploy Backend Service

### 9.1 Create Stacks Directory

**On manager:**

```bash
# Create directory for stack files
mkdir -p ~/stacks
cd ~/stacks
```

### 9.2 Create Backend Stack File

```bash
vi maplepress-stack.yml
```

**Paste this:**

```yaml
version: '3.8'

networks:
  mapleopentech-private-prod:
    external: true
  mapleopentech-public-prod:
    external: true

secrets:
  maplepress_jwt_secret:
    external: true
  maplepress_ip_encryption_key:
    external: true
  redis_password:
    external: true
  meilisearch_master_key:
    external: true
  spaces_access_key:
    external: true
  spaces_secret_key:
    external: true

services:
  backend:
    image: registry.digitalocean.com/ssp/maplepress_backend:prod
    hostname: maplepress-backend
    networks:
      - mapleopentech-public-prod     # Receives requests from Caddy
      - mapleopentech-private-prod    # Accesses databases
    secrets:
      - maplepress_jwt_secret
      - maplepress_ip_encryption_key
      - redis_password
      - meilisearch_master_key
      - spaces_access_key
      - spaces_secret_key
    environment:
      # Application
      - APP_ENVIRONMENT=production
      - APP_VERSION=1.0.0
      - SERVER_HOST=0.0.0.0
      - SERVER_PORT=8000

      # Database (Cassandra)
      - DATABASE_HOSTS=cassandra-1:9042,cassandra-2:9042,cassandra-3:9042
      - DATABASE_KEYSPACE=maplepress
      - DATABASE_CONSISTENCY=QUORUM
      - DATABASE_REPLICATION=3
      - DATABASE_MIGRATIONS_PATH=file://migrations

      # Search (Meilisearch)
      - MEILISEARCH_HOST=http://meilisearch:7700

      # Object Storage (DigitalOcean Spaces)
      - AWS_ENDPOINT=https://nyc3.digitaloceanspaces.com
      - AWS_REGION=nyc3
      - AWS_BUCKET_NAME=maplepress-prod

      # Security (CORS)
      - SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com

      # Logging
      - LOGGER_LEVEL=info
      - LOGGER_FORMAT=json
    entrypoint: ["/bin/sh", "-c"]
    command:
      - |
        cd /app
        export APP_JWT_SECRET=$$(cat /run/secrets/maplepress_jwt_secret)
        export SECURITY_IP_ENCRYPTION_KEY=$$(cat /run/secrets/maplepress_ip_encryption_key)
        export CACHE_PASSWORD=$$(cat /run/secrets/redis_password)
        export MEILISEARCH_API_KEY=$$(cat /run/secrets/meilisearch_master_key)
        export AWS_ACCESS_KEY=$$(cat /run/secrets/spaces_access_key)
        export AWS_SECRET_KEY=$$(cat /run/secrets/spaces_secret_key)
        export CACHE_HOST=redis
        export CACHE_PORT=6379
        export CACHE_DB=0
        exec /app/maplepress-backend daemon
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.backend == true  # Only deploy to worker-6
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 3
      update_config:
        parallelism: 1
        delay: 10s
        order: start-first  # Zero-downtime updates
      resources:
        limits:
          memory: 1G
        reservations:
          memory: 512M
    healthcheck:
      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "--header=X-Tenant-ID: healthcheck", "http://localhost:8000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 60s
```

Save: `Ctrl+O`, `Enter`, `Ctrl+X`

### 9.3 Deploy Backend

```bash
# Deploy the maplepress stack (backend service)
docker stack deploy -c maplepress-stack.yml maplepress

# Check service created
docker service ls | grep backend
# Should show: maplepress_backend   0/1   registry.digitalocean.com/ssp/maplepress_backend:prod
```

### 9.4 Watch Backend Start

```bash
# Watch service come up
watch docker service ps maplepress_backend
# Press Ctrl+C when you see "Running" status

# Or check directly
docker service ps maplepress_backend
```

**Expected output after 1-2 minutes:**

```
ID             NAME                   NODE                                CURRENT STATE         ERROR
abc123...      maplepress_backend.1   mapleopentech-swarm-worker-6-prod       Running 1 minute ago
```

**If stuck in "Preparing" or "Starting":**

Check logs:

```bash
docker service logs -f maplepress_backend
# Look for errors
# Press Ctrl+C to exit
```

**Common issues:**
- **Image pull failed**: Worker-6 must authenticate with registry (see Step 7.3)
- **No suitable node**: Worker-6 missing `backend=true` label (see Step 4.4)
- **Secrets missing**: Create all 4 secrets (see Step 8)

### 9.5 Verify Backend Health

**Important:** The backend container runs on worker-6, not the manager. You must SSH to worker-6 to test directly.

**From your local machine, SSH to worker-6:**

```bash
ssh dockeradmin@<worker-6-public-ip>

# Find backend container ID
docker ps | grep backend

# Test health endpoint
docker exec $(docker ps -q --filter "name=maplepress_backend") \
  wget --no-verbose --tries=1 --spider --header="X-Tenant-ID: healthcheck" http://localhost:8000/health

# Should return: HTTP/1.1 200 OK
```

**Alternative: Check from manager (without SSHing to worker-6):**

```bash
# On manager
docker service ps maplepress_backend
# Should show: Running X minutes ago

# Check logs for successful startup
docker service logs maplepress_backend --tail 30
# Should show: migrations completed, schedulers running
```

**✅ Checkpoint:** Backend service running, health check passes

---

## Troubleshooting

### Quick Reference: Complete Backend Deployment Process

**IMPORTANT:** Follow this exact process when deploying backend updates to avoid caching issues and CORS errors.

```bash
# ==============================================================================
# STEP 1: Build and Push (Local Machine)
# ==============================================================================
cd ~/go/src/codeberg.org/mapleopentech/monorepo/cloud/maplepress-backend
task deploy

# ==============================================================================
# STEP 2: Force Pull on Worker-6 (Prevents Docker Image Caching)
# ==============================================================================
ssh dockeradmin@<worker-6-public-ip>
docker pull registry.digitalocean.com/ssp/maplepress_backend:prod
docker images --digests registry.digitalocean.com/ssp/maplepress_backend:prod
# Note the digest - should be recent
exit

# ==============================================================================
# STEP 3: Redeploy Stack (Manager Node)
# ==============================================================================
ssh dockeradmin@<manager-public-ip>
cd ~/stacks

# Remove existing stack
docker stack rm maplepress
sleep 10

# Remove old config (Docker configs are immutable)
docker config rm maplepress_caddyfile 2>/dev/null || true

# Deploy fresh
docker stack deploy -c maplepress-stack.yml maplepress

# ==============================================================================
# STEP 4: Verify Deployment
# ==============================================================================
# Check both services are running
docker service ls | grep maplepress
# Should show:
#   maplepress_backend         1/1
#   maplepress_backend-caddy   1/1

# Check service status
docker service ps maplepress_backend
docker service ps maplepress_backend-caddy

# ==============================================================================
# STEP 5: Test CORS (Local Machine)
# ==============================================================================
curl -v -H "Origin: https://getmaplepress.com" https://getmaplepress.ca/health 2>&1 | grep "access-control-allow-origin"
# Should show: access-control-allow-origin: https://getmaplepress.com

# ==============================================================================
# STEP 6: Test in Browser
# ==============================================================================
# Visit https://getmaplepress.com
# Open DevTools (F12) → Network tab
# Verify API calls to https://getmaplepress.ca succeed (status 200, not 0)
```

**Why each step matters:**
- **Step 2** prevents Docker from using cached old images
- **Step 3** ensures Docker configs are updated (they're immutable)
- **Step 5** verifies CORS before testing in browser (saves time)

---

### Common Problems

### Problem: Backend Won't Start

**Symptom:** `docker service ps maplepress_backend` shows "Starting" for > 2 minutes

**Check:**

```bash
# View logs
docker service logs maplepress_backend --tail 100

# Common issues:
# 1. Can't reach databases
#    - Verify Cassandra running: docker service ls | grep cassandra
#    - Verify Redis running: docker service ls | grep redis
#    - Check backend is on mapleopentech-private-prod network

# 2. Secrets missing
docker secret ls
# Should show: maplepress_jwt_secret, maplepress_ip_encryption_key, redis_password, meilisearch_master_key

# 3. Image pull failed
docker service ps maplepress_backend
# If you see "image not found", verify worker-6 authenticated with registry
ssh dockeradmin@<worker-6-public-ip>
docker pull registry.digitalocean.com/ssp/maplepress_backend:prod
```

### Problem: Migrations Error

**Symptom:** Backend logs show `failed to open source, "file://migrations": open .: no such file or directory`

**Cause:** Docker image was built without migrations directory or worker-6 has cached old image.

**Fix:**

```bash
# 1. Verify Dockerfile includes migrations
# On your local machine:
cd ~/monorepo/cloud/maplepress-backend
grep "COPY.*migrations" Dockerfile
# Should show: COPY --from=build-env /app/migrations ./migrations

# 2. Rebuild and push image with migrations
task deploy

# 3. Clear cache on worker-6 and force fresh pull
ssh dockeradmin@<worker-6-public-ip>
docker system prune -af

# 4. Authenticate and pull fresh image
doctl registry login
docker pull registry.digitalocean.com/ssp/maplepress_backend:prod

# 5. Verify new image has /app and migrations
docker run --rm registry.digitalocean.com/ssp/maplepress_backend:prod ls -la /app
# Should show: migrations/ directory

# 6. Redeploy backend from manager
ssh dockeradmin@<manager-public-ip>
cd ~/stacks
docker stack deploy -c maplepress-stack.yml maplepress
```

### Problem: CORS Errors from Frontend

**Symptom:** Frontend at `https://getmaplepress.com` gets CORS errors when calling backend at `https://getmaplepress.ca`

Browser console shows:
```
Access to fetch at 'https://getmaplepress.ca/health' from origin 'https://getmaplepress.com'
has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present
```

**Root Causes:**

1. **Backend not deployed** - The maplepress stack doesn't exist
2. **Old image cached** - Worker-6 has old code without CORS support
3. **Wrong CORS origins configured** - Environment variable has wrong domain

**Diagnostic Steps:**

```bash
# 1. Check if backend stack exists
docker stack ls | grep maplepress
# Should show: maplepress (with 2 services)

# 2. Check if backend is running
docker service ls | grep maplepress_backend
# Should show: maplepress_backend  1/1

# 3. Test CORS directly
curl -v -H "Origin: https://getmaplepress.com" https://getmaplepress.ca/health 2>&1 | grep -i "access-control-allow-origin"
# Should show: access-control-allow-origin: https://getmaplepress.com

# 4. If CORS header is missing, check backend logs
docker service logs maplepress_backend --tail 50 | grep -i "cors\|origin"
# Should show CORS debug/warning messages when requests arrive
```

**Fix:**

```bash
# 1. Verify CORS configuration in stack file
cat ~/stacks/maplepress-stack.yml | grep CORS
# Should show: - SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com

# 2. If missing, add to stack file
vi ~/stacks/maplepress-stack.yml
# Add under environment section:
#   - SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com

# 3. Force fresh deployment (on local machine first)
cd ~/go/src/codeberg.org/mapleopentech/monorepo/cloud/maplepress-backend
task deploy

# 4. Force pull fresh image on worker-6
ssh dockeradmin@<worker-6-public-ip>
docker pull registry.digitalocean.com/ssp/maplepress_backend:prod
docker images --digests registry.digitalocean.com/ssp/maplepress_backend:prod
exit

# 5. Completely redeploy stack (on manager)
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile 2>/dev/null || true
docker stack deploy -c maplepress-stack.yml maplepress

# 6. Verify CORS is working
curl -v -H "Origin: https://getmaplepress.com" https://getmaplepress.ca/health 2>&1 | grep "access-control-allow-origin"
# Should show: access-control-allow-origin: https://getmaplepress.com

# 7. Check backend logs show CORS activity
docker service logs maplepress_backend --tail 20
# Should show debug messages when CORS requests arrive
```

**Prevention:**

- Always follow the complete deployment process (build → push → force pull → redeploy)
- Test CORS after every backend update
- Never skip the force pull step on worker-6 (prevents image caching issues)
```

### Problem: JWT Secret Validation Error

**Symptom:** Backend panics with `JWT secret does not appear to be randomly generated` or `SECURITY: Invalid JWT secret in production environment`

**Cause:** JWT secret contains newlines or wasn't generated with proper base64 encoding.

**Fix:**

```bash
# On manager
ssh dockeradmin@<manager-public-ip>

# 1. Remove old JWT secret
docker secret rm maplepress_jwt_secret

# 2. Generate proper base64 JWT secret (no newlines)
JWT_SECRET=$(openssl rand -base64 64 | tr -d '\n')

# 3. SAVE THIS! Print to screen
echo "JWT Secret: $JWT_SECRET"
# Copy to your password manager!

# 4. Create secret with no newline
echo -n "$JWT_SECRET" | docker secret create maplepress_jwt_secret -

# 5. Redeploy backend to pick up new secret
cd ~/stacks
docker stack deploy -c maplepress-stack.yml maplepress

# 6. Watch backend start successfully
docker service logs -f maplepress_backend
# Should NOT see JWT validation errors
```

**Important:** Always use `tr -d '\n'` to remove newlines and `echo -n` to avoid adding newlines when creating secrets.

### Problem: Can't Connect to Databases

**Symptom:** Backend logs show connection errors to Cassandra, Redis, or Meilisearch

**Check:**

```bash
# 1. Verify all database services are running
docker service ls
# Should show: cassandra_cassandra-1, cassandra_cassandra-2, cassandra_cassandra-3, redis_redis, meilisearch_meilisearch

# 2. Verify backend is on mapleopentech-private-prod network
docker service inspect maplepress_backend --format '{{range .Spec.TaskTemplate.Networks}}{{.Target}} {{end}}'
# Should include mapleopentech-private-prod

# 3. Test DNS resolution from backend container
ssh dockeradmin@<worker-6-public-ip>
docker exec $(docker ps -q --filter "name=maplepress_backend") nslookup cassandra-1
docker exec $(docker ps -q --filter "name=maplepress_backend") nslookup redis
docker exec $(docker ps -q --filter "name=maplepress_backend") nslookup meilisearch
# All should resolve to IPs
```

### Problem: Worker-6 Pulling Old/Cached Image

**Symptom:** Changes not reflected after deploying, logs show old errors

**Cause:** Docker Swarm worker has cached old image and isn't pulling fresh one.

**Fix:**

```bash
# On worker-6
ssh dockeradmin@<worker-6-public-ip>

# 1. Remove all cached images and containers
docker system prune -af
# WARNING: This removes ALL unused images, be careful!

# 2. Verify authentication with registry
doctl registry login

# 3. Force pull fresh image
docker pull registry.digitalocean.com/ssp/maplepress_backend:prod

# 4. Verify it's the new image (check digest)
docker images registry.digitalocean.com/ssp/maplepress_backend:prod --digests

# 5. From manager, force service update
ssh dockeradmin@<manager-public-ip>
docker service update --force maplepress_backend

# Watch it redeploy
watch docker service ps maplepress_backend
```

---

## Next Steps

✅ **You now have:**
- Worker-6 droplet configured and joined to Docker Swarm
- Backend service running on worker-6
- Backend connected to both networks (private for databases, public for Caddy)
- Docker secrets configured for sensitive credentials
- Backend pulling from DigitalOcean Container Registry
- Health checks passing
- Backend ready to receive requests from Caddy reverse proxy

**Next guide:**
- **06_caddy.md** - Deploy Caddy reverse proxy with automatic SSL/TLS
- Configure HTTPS with Let's Encrypt (automatic certificate management)
- Security headers and rate limiting
- Make your backend accessible via https://getmaplepress.ca

**Maintenance commands:**

```bash
# View backend logs
docker service logs -f maplepress_backend

# Update backend to new version (IMPORTANT: Follow this complete process)
# Step 1: Build and push new image (on local machine)
cd ~/go/src/codeberg.org/mapleopentech/monorepo/cloud/maplepress-backend
task deploy

# Step 2: Force pull fresh image on worker-6 (prevents caching issues)
ssh dockeradmin@<worker-6-public-ip>
docker pull registry.digitalocean.com/ssp/maplepress_backend:prod
exit

# Step 3: Remove and redeploy stack (ensures fresh image)
ssh dockeradmin@<manager-public-ip>
cd ~/stacks
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress

# Step 4: Verify deployment
docker service ps maplepress_backend
docker service ps maplepress_backend-caddy

# Step 5: Test CORS is working (from local machine)
curl -v -H "Origin: https://getmaplepress.com" https://getmaplepress.ca/health | grep -i "access-control"
# Should show: access-control-allow-origin: https://getmaplepress.com

# Scale backend (add more replicas)
vi maplepress-stack.yml  # Change replicas: 1 to replicas: 3
docker stack deploy -c maplepress-stack.yml maplepress

# Quick restart (without redeploying)
docker service update --force maplepress_backend
```

---

**Last Updated**: January 2025
**Maintained By**: Infrastructure Team
