# Deploy MapleFile Backend: Part 1

**Audience**: Junior DevOps Engineers, Infrastructure Team
**Time to Complete**: 60-90 minutes
**Prerequisites**:
- Completed guides 01-04.5 (Swarm, Cassandra, Redis, Meilisearch, Spaces)
- Backend Docker image ready to deploy
- Domain name `maplefile.ca` configured
- DigitalOcean API token for registry access
- DigitalOcean Spaces configured (from 04.5_spaces.md)

---

## Overview

This guide sets up **worker-8** from scratch and deploys the MapleFile backend. Part 2 (10_maplefile_caddy.md) will add the Caddy reverse proxy.

**What you'll build:**
- Fresh worker-8 droplet with Docker and dockeradmin user
- Worker-8 joined to existing Docker Swarm
- Backend service connected to both networks (private databases + public reverse proxy)
- Backend service ready for Caddy reverse proxy (deployed in Part 2)
- Pull backend image from DigitalOcean Container Registry

**Architecture:**
```
Internet (HTTPS) → Caddy (worker-8) → Backend (worker-8) → Cassandra/Redis (private network)
                    [Part 2: 10_maplefile_caddy.md]    [Part 1: This guide]
```

**Key Differences from MaplePress:**
- MapleFile uses the **same shared Cassandra cluster** as MaplePress (different keyspace)
- Uses same shared Redis and infrastructure services
- Uses S3-compatible storage (DigitalOcean Spaces)
- Different domain name and CORS origins

---

## Step 1: Create Worker-8 Droplet

### 1.1 Create Droplet in DigitalOcean

1. Log into DigitalOcean dashboard
2. Click **Create** → **Droplets**
3. Configure:
   - **Region**: Same as your other workers (Toronto - tor1)
   - **Image**: Ubuntu 24.04 LTS x64
   - **Size**: Basic shared CPU, 2 GB / 2 vCPU ($18/mo)
   - **Hostname**: `mapleopentech-swarm-worker-8-prod`
   - **VPC Network**: Select same VPC as your swarm (default-tor1)
   - **SSH Keys**: Add your SSH key
4. Click **Create Droplet**
5. Wait 1-2 minutes for droplet to provision

### 1.2 Record IP Addresses

Once created, copy both IPs:
- **Public IPv4**: `<worker-8-public-ip>` (e.g., 143.110.212.253)
- **Private IPv4**: `<worker-8-private-ip>` (e.g., 10.137.0.16)

**Update your local `.env` file:**

```bash
# On your local machine
cd ~/monorepo/cloud/infrastructure/production
vi .env

# Add these lines (use YOUR actual IPs from the environment details you provided):
SWARM_WORKER_8_HOSTNAME=mapleopentech-swarm-worker-8-prod
SWARM_WORKER_8_PUBLIC_IP=143.110.212.253
SWARM_WORKER_8_PRIVATE_IP=10.137.0.16
```

Save: `Ctrl+O`, `Enter`, `Ctrl+X`

**✅ Checkpoint:** You should be able to ping worker-8:

```bash
ping 143.110.212.253
# Should get responses
```

---

## Step 2: Initial Server Setup

### 2.1 SSH to Worker-8

```bash
# From your local machine
ssh root@143.110.212.253

# Should connect successfully
```

### 2.2 Update System Packages

```bash
# Update package lists
apt update

# Upgrade all packages
apt upgrade -y

# Install essential tools and utilities
apt install -y \
  curl wget git vim nano \
  apt-transport-https ca-certificates gnupg lsb-release software-properties-common \
  htop iotop nethogs \
  net-tools dnsutils iputils-ping traceroute \
  jq tree unzip zip \
  build-essential \
  screen tmux
```

This takes 2-5 minutes.

### 2.3 Install Docker

```bash
# Add Docker GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Add Docker repository
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

# Update package list with Docker packages
apt update

# Install Docker
apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

# Verify installation
docker --version
# Should show: Docker version 27.x.x or higher

# Check Docker is running
docker ps
# Should show empty list (not error)
```

**✅ Checkpoint:** `docker --version` shows version 29+

### 2.4 Create dockeradmin User

```bash
# Create user
adduser dockeradmin
# Enter password when prompted
# SAVE THIS PASSWORD IN YOUR PASSWORD MANAGER!

# Add to sudo group (can run admin commands)
usermod -aG sudo dockeradmin

# Add to docker group (can run docker commands)
usermod -aG docker dockeradmin

# Copy SSH keys to new user
rsync --archive --chown=dockeradmin:dockeradmin ~/.ssh /home/dockeradmin

# Verify SSH keys copied
ls -la /home/dockeradmin/.ssh/
# Should show: authorized_keys
```

### 2.5 Test dockeradmin Access

**From your local machine (open new terminal):**

```bash
# Test SSH login
ssh dockeradmin@143.110.212.253

# Should login WITHOUT password prompt (using SSH key)

# Test docker access
docker ps
# Should show empty list (NOT permission denied)

# Test sudo access
sudo ls /root
# Enter dockeradmin password when prompted
# Should list root directory contents

# Exit back to local machine
exit
```

**✅ Checkpoint:** Can SSH as dockeradmin without password, run docker commands

---

## Step 3: Configure Firewall

**SSH back to worker-8 as root:**

```bash
ssh root@143.110.212.253
```

### 3.1 Setup UFW Firewall

```bash
# Enable firewall (force to avoid prompt)
ufw --force enable

# Allow SSH (CRITICAL - do this first!)
ufw allow 22/tcp

# Allow HTTP and HTTPS (for Caddy)
ufw allow 80/tcp
ufw allow 443/tcp

# Allow Docker Swarm (only from private VPC network)
# Use YOUR VPC subnet from .env: SWARM_VPC_SUBNET=10.137.0.0/16
ufw allow from 10.137.0.0/16 to any port 2377 proto tcp  # Swarm management
ufw allow from 10.137.0.0/16 to any port 7946            # Node communication
ufw allow from 10.137.0.0/16 to any port 4789 proto udp  # Overlay network

# Check firewall status
ufw status verbose
```

**Expected output:**

```
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
2377/tcp                   ALLOW       10.137.0.0/16
7946                       ALLOW       10.137.0.0/16
4789/udp                   ALLOW       10.137.0.0/16
```

**✅ Checkpoint:** UFW active, ports open correctly

---

## Step 4: Join Worker-8 to Docker Swarm

### 4.1 Get Swarm Join Token

**From your local machine, SSH to manager:**

```bash
ssh dockeradmin@143.110.210.162  # Your manager public IP

# Get worker join token
docker swarm join-token worker
```

**Copy the entire output command.** It looks like:

```bash
docker swarm join --token SWMTKN-1-xxxxxx... 10.137.0.11:2377
```

**Important:** Use the **private IP** shown in the command (e.g., `10.137.0.11:2377`), NOT the public IP.

### 4.2 Join Worker-8 to Swarm

**SSH to worker-8 as dockeradmin:**

```bash
# From your local machine
ssh dockeradmin@143.110.212.253

# Paste the join command from manager
docker swarm join --token SWMTKN-1-xxxxxx... 10.137.0.11:2377
```

**Expected output:**

```
This node joined a swarm as a worker.
```

### 4.3 Verify Worker-8 Joined

**Back on manager:**

```bash
# List all nodes
docker node ls
```

**You should see:**

```
ID                            HOSTNAME                            STATUS    AVAILABILITY   MANAGER STATUS
abc123...                     mapleopentech-swarm-manager-1-prod      Ready     Active         Leader
def456...                     mapleopentech-swarm-worker-1-prod       Ready     Active
...
xyz789...                     mapleopentech-swarm-worker-8-prod       Ready     Active         <-- NEW!
```

**✅ Checkpoint:** Worker-8 appears in `docker node ls` with status `Ready Active`

### 4.4 Label Worker-8 for MapleFile Backend

**On manager:**

```bash
# Add maplefile-backend label (tells swarm to deploy maplefile backend here)
docker node update --label-add maplefile-backend=true mapleopentech-swarm-worker-8-prod

# Verify label
docker node inspect mapleopentech-swarm-worker-8-prod --format '{{.Spec.Labels}}'
# Should show: map[maplefile-backend:true]
```

**✅ Checkpoint:** Worker-8 has `maplefile-backend=true` label

---

## Step 5: Configure DNS

Before deploying, point your domain to worker-8.

### 5.1 Update DNS Records

1. Log into your domain registrar (where you bought maplefile.ca)
2. Find DNS settings / DNS records
3. Add/update these A records:

| Type | Name | Value | TTL |
|------|------|-------|-----|
| A | @ | `143.110.212.253` | 3600 |
| A | www | `143.110.212.253` | 3600 |

Replace `143.110.212.253` with your actual worker-8 public IP.

Example:
- **@ record**: Points `maplefile.ca` → 143.110.212.253
- **www record**: Points `www.maplefile.ca` → 143.110.212.253

### 5.2 Wait for DNS Propagation

DNS changes take 5-10 minutes (sometimes up to 1 hour).

**Test from your local machine:**

```bash
# Test root domain
dig maplefile.ca +short
# Should return: 143.110.212.253 (your worker-8 public IP)

# Test www subdomain
dig www.maplefile.ca +short
# Should return: 143.110.212.253 (your worker-8 public IP)
```

**⚠️ Don't proceed until the command returns the correct IP!**

**✅ Checkpoint:** DNS resolves to worker-8 public IP

---

## Step 6: Create Docker Networks

We need two overlay networks for our services:

1. **mapleopentech-private-prod** - Backend connects to databases (already exists from guides 02-04)
2. **mapleopentech-public-prod** - Caddy and Backend communicate (already exists from guide 06)

**On manager:**

```bash
ssh dockeradmin@143.110.210.162

# Check both networks exist
docker network ls | grep maple

# Should show:
# mapleopentech-private-prod   overlay   swarm
# mapleopentech-public-prod    overlay   swarm
```

**If mapleopentech-public-prod doesn't exist, create it:**

```bash
# Create public network
docker network create --driver overlay --attachable mapleopentech-public-prod

# Verify both exist
docker network ls | grep maple
```

**Why two networks?**
- **Private**: Backend talks to Cassandra, Redis (no internet access)
- **Public**: Caddy forwards requests to Backend (internet-facing)
- Backend joins BOTH networks to receive requests and access databases

**✅ Checkpoint:** Both `mapleopentech-private-prod` and `mapleopentech-public-prod` networks exist

---

## Step 7: Authenticate with DigitalOcean Registry

Worker-8 needs to authenticate with DO registry to pull your private backend image.

### 7.1 Install doctl on Worker-8

**SSH to worker-8 as dockeradmin:**

```bash
ssh dockeradmin@143.110.212.253

# Download doctl (latest version)
cd ~
wget https://github.com/digitalocean/doctl/releases/download/v1.147.0/doctl-1.147.0-linux-amd64.tar.gz

# Extract
tar xf doctl-*.tar.gz

# Move to system path
sudo mv doctl /usr/local/bin

# Clean up
rm doctl-*.tar.gz

# Verify installation
doctl version
# Should show: doctl version 1.147.0
```

### 7.2 Authenticate doctl

**You need your DigitalOcean API token.** Get it from:
1. DigitalOcean dashboard → API → Tokens/Keys
2. Use existing token or click "Generate New Token"
3. Copy the token (looks like: `dop_v1_xxxxx...`)

**On worker-8 (as dockeradmin):**

**⚠️ IMPORTANT:** Make sure you're logged in as `dockeradmin`, NOT as `root`. If you're root, exit and SSH back in as dockeradmin.

```bash
# Verify you're dockeradmin (not root)
whoami
# Should show: dockeradmin

# Initialize doctl with your API token
doctl auth init
# Paste your API token when prompted
```

**Expected output:**

```
Please authenticate doctl for use with your DigitalOcean account. You can generate a token in the control panel at https://cloud.digitalocean.com/account/api/tokens

Enter your access token: dop_v1_xxxxx...

Validating token... OK
```

**Alternative method (if interactive prompt fails):**

```bash
# Pass token directly (replace YOUR_API_TOKEN with your actual token)
doctl auth init --access-token dop_v1_xxxxx...
```

### 7.3 Login to Registry

```bash
# Login to DigitalOcean Container Registry
doctl registry login
```

**Expected output:**

```
Logging Docker in to registry.digitalocean.com
Login Succeeded
```

### 7.4 Test Image Pull

```bash
# Try pulling your backend image
docker pull registry.digitalocean.com/ssp/maplefile-backend:prod
```

**If you get "manifest not found" error:**

You need to build and push your image first. See section at end of this guide.

**✅ Checkpoint:** `docker images | grep maplefile` shows the backend image

---

## Step 8: Create Docker Secrets

**On manager:**

```bash
ssh dockeradmin@143.110.210.162
```

### 8.1 Generate and Create JWT Secret

```bash
# Generate JWT secret (base64 encoded, 64 characters)
JWT_SECRET=$(openssl rand -base64 64 | tr -d '\n')

# SAVE THIS! Print to screen
echo "JWT Secret: $JWT_SECRET"
# Copy this to your password manager!

# Create Docker secret (use -n to avoid adding newline)
echo -n "$JWT_SECRET" | docker secret create maplefile_jwt_secret -
```

### 8.2 Generate and Create JWT Secret

MapleFile backend needs multiple encryption keys:

```bash
# 1. IP Encryption Key (32 hex characters)
IP_ENCRYPTION_KEY=$(openssl rand -hex 16)
echo "IP Encryption Key: $IP_ENCRYPTION_KEY"  # SAVE THIS!
echo $IP_ENCRYPTION_KEY | docker secret create maplefile_ip_encryption_key -

# 2. File Encryption Key (32 bytes base64)
FILE_ENCRYPTION_KEY=$(openssl rand -base64 32 | tr -d '\n')
echo "File Encryption Key: $FILE_ENCRYPTION_KEY"  # SAVE THIS!
echo -n "$FILE_ENCRYPTION_KEY" | docker secret create maplefile_file_encryption_key -

# 3. Database Encryption Key (32 bytes base64)
DB_ENCRYPTION_KEY=$(openssl rand -base64 32 | tr -d '\n')
echo "DB Encryption Key: $DB_ENCRYPTION_KEY"  # SAVE THIS!
echo -n "$DB_ENCRYPTION_KEY" | docker secret create maplefile_db_encryption_key -
```

### 8.3 Generate and Create Mailgun API Key Secret

```bash
# Replace with your actual Mailgun API key
# Get this from: https://app.mailgun.com/app/account/security/api_keys
MAILGUN_API_KEY="your-mailgun-api-key-here"

echo "Mailgun API Key: $MAILGUN_API_KEY"  # SAVE THIS!
echo -n "$MAILGUN_API_KEY" | docker secret create maplefile_mailgun_api_key -
```

### 8.4 Verify All Required Secrets Exist

```bash
# List all secrets
docker secret ls
```

**You should see these secrets:**

```
ID                          NAME                              CREATED
abc123...                   maplefile_jwt_secret              1 second ago
def456...                   redis_password                    from guide 03
ghi789...                   maplefile_mailgun_api_key                   1 second ago
jkl012...                   spaces_access_key                 from guide 04.5
mno345...                   spaces_secret_key                 from guide 04.5
```

**If redis_password or Spaces secrets are missing:**

Create them now:

```bash
# Redis password (if missing)
REDIS_PASSWORD=$(openssl rand -base64 32 | tr -d "=+/" | cut -c1-32)
echo "Redis Password: $REDIS_PASSWORD"  # SAVE THIS!
echo $REDIS_PASSWORD | docker secret create redis_password -

# Spaces secrets (if missing)
# These should have been created in 04.5_spaces.md
# If not, create them now with your DigitalOcean Spaces credentials

# Replace with your actual Spaces access key from DigitalOcean dashboard
echo -n "DO00ABC123XYZ456" | docker secret create spaces_access_key -

# Replace with your actual Spaces secret key from DigitalOcean dashboard
echo -n "abc123def456ghi789jkl012mno345pqr678stu901" | docker secret create spaces_secret_key -

# Verify they were created
docker secret ls | grep -E "redis|spaces"
```

**✅ Checkpoint:** All required secrets exist: `maplefile_jwt_secret`, `redis_password`, `maplefile_mailgun_api_key`, `spaces_access_key`, `spaces_secret_key`

---

## Step 9: Deploy Backend Service

### 9.1 Create Stacks Directory (if not exists)

**On manager:**

```bash
# Create directory for stack files
mkdir -p ~/stacks
cd ~/stacks
```

### 9.2 Create Backend Stack File

```bash
# Create stacks directory if it doesn't exist
mkdir -p ~/stacks

# Create the stack file
vi ~/stacks/maplefile-stack.yml
```

**Paste this:**

```yaml
version: '3.8'

networks:
  mapleopentech-private-prod:
    external: true
  mapleopentech-public-prod:
    external: true

secrets:
  maplefile_jwt_secret:
    external: true
  redis_password:
    external: true
  maplefile_mailgun_api_key:
    external: true
  spaces_access_key:
    external: true
  spaces_secret_key:
    external: true

services:
  backend:
    image: registry.digitalocean.com/ssp/maplefile-backend:prod
    hostname: maplefile-backend
    networks:
      - mapleopentech-public-prod     # Receives requests from Caddy
      - mapleopentech-private-prod    # Accesses databases
    secrets:
      - maplefile_jwt_secret
      - redis_password
      - maplefile_mailgun_api_key
      - spaces_access_key
      - spaces_secret_key
    environment:
      # Application
      - APP_ENVIRONMENT=production
      - APP_VERSION=1.0.0
      - SERVER_HOST=0.0.0.0
      - SERVER_PORT=8000

      # Database (Cassandra - shared infrastructure)
      # Note: MapleFile uses Cassandra just like MaplePress
      # Both apps share the same 3-node Cassandra cluster
      # but use different keyspaces (maplefile vs maplepress)
      # IMPORTANT: Do NOT include :9042 port - gocql adds it automatically
      - DATABASE_HOSTS=cassandra-1,cassandra-2,cassandra-3
      - DATABASE_KEYSPACE=maplefile
      - DATABASE_CONSISTENCY=QUORUM
      - DATABASE_REPLICATION=3
      - DATABASE_MIGRATIONS_PATH=file://migrations

      # Cache (Redis - shared infrastructure)
      - CACHE_HOST=redis
      - CACHE_PORT=6379
      - CACHE_DB=1  # Use DB 1 (MaplePress uses DB 0)

      # Object Storage (DigitalOcean Spaces)
      # Replace with your actual Spaces configuration
      - S3_ENDPOINT=https://nyc3.digitaloceanspaces.com
      - S3_REGION=nyc3
      - S3_BUCKET=maplefile-prod
      - S3_USE_SSL=true

      # JWT Settings
      - JWT_ACCESS_TOKEN_DURATION=15m
      - JWT_REFRESH_TOKEN_DURATION=168h
      - JWT_SESSION_DURATION=24h

      # Security (CORS)
      # Replace with your actual frontend domain(s)
      - SECURITY_ALLOWED_ORIGINS=https://maplefile.com,https://www.maplefile.com

      # Logging
      - LOG_LEVEL=info
      - LOG_FORMAT=json

      # Mailgun (Email)
      - MAILGUN_DOMAIN=maplefile.ca
      - MAILGUN_API_BASE=https://api.mailgun.net/v3
      - MAILGUN_FROM_EMAIL=noreply@maplefile.ca
      - MAILGUN_FROM_NAME=MapleFile
      - MAILGUN_FRONTEND_URL=https://maplefile.com

      # Leader Election (for multi-instance deployments)
      - LEADER_ELECTION_ENABLED=true
      - LEADER_ELECTION_LOCK_TTL=10s
      - LEADER_ELECTION_HEARTBEAT_INTERVAL=3s

      # Invite Email Configuration
      # Maximum invitation emails a user can send per day to non-registered users
      - MAPLEFILE_INVITE_MAX_EMAILS_PER_DAY=3

      # Login Rate Limiting
      # Controls brute-force protection for login attempts
      - LOGIN_RATE_LIMIT_MAX_ATTEMPTS_PER_IP=50
      - LOGIN_RATE_LIMIT_IP_WINDOW=15m
      - LOGIN_RATE_LIMIT_MAX_FAILED_PER_ACCOUNT=10
      - LOGIN_RATE_LIMIT_LOCKOUT_DURATION=30m

    entrypoint: ["/bin/sh", "-c"]
    command:
      - |
        cd /app
        export JWT_SECRET=$$(cat /run/secrets/maplefile_jwt_secret)
        export CACHE_PASSWORD=$$(cat /run/secrets/redis_password)
        export MAILGUN_API_KEY=$$(cat /run/secrets/maplefile_mailgun_api_key)
        export S3_ACCESS_KEY=$$(cat /run/secrets/spaces_access_key)
        export S3_SECRET_KEY=$$(cat /run/secrets/spaces_secret_key)
        exec /app/maplefile-backend daemon
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.maplefile-backend == true  # Only deploy to worker-8
      restart_policy:
        condition: any  # Restart on ANY exit (including clean exits)
        delay: 10s
        max_attempts: 0  # Unlimited restart attempts
      update_config:
        parallelism: 1
        delay: 10s
        order: start-first  # Zero-downtime updates
      resources:
        limits:
          memory: 1G
        reservations:
          memory: 512M
    healthcheck:
      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 60s
```

Save: `Ctrl+O`, `Enter`, `Ctrl+X`

**Important customizations needed:**
1. **CORS Origins**: Update `https://maplefile.com` if using different frontend domain
2. **Mailgun Domain**: Update `MAILGUN_DOMAIN` and `MAILGUN_FROM_EMAIL` to match your verified Mailgun sending domain
3. **S3 Settings**: Verify endpoint, region, and bucket name match your Spaces setup

### 9.3 Initialize Cassandra Keyspace

**⚠️ CRITICAL: Run this BEFORE deploying backend (first-time setup only)**

The backend requires the `maplefile` keyspace to exist before it starts. This is a one-time infrastructure setup.

```bash
# SSH to any Cassandra node
ssh dockeradmin@<cassandra-node-ip>

# Find the Cassandra container (Swarm uses long service names)
export CASSANDRA_CONTAINER=$(docker ps --filter "name=cassandra" -q | head -1)

# Initialize the maplefile keyspace
docker exec -it $CASSANDRA_CONTAINER cqlsh -e "
CREATE KEYSPACE IF NOT EXISTS maplefile
WITH replication = {
    'class': 'SimpleStrategy',
    'replication_factor': 3
};"

# Verify keyspace created
docker exec -it $CASSANDRA_CONTAINER cqlsh -e "DESCRIBE KEYSPACE maplefile;"
```

**Expected output:**
```cql
CREATE KEYSPACE maplefile WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;
```

**✅ Checkpoint:** Keyspace `maplefile` exists in Cassandra

**When to run this:**
- ✅ First-time backend deployment
- ✅ After formatting/rebuilding Cassandra cluster
- ✅ When creating new environment (staging, QA, etc.)
- ❌ NOT needed for normal backend redeployments

**Note:** The backend will auto-migrate tables on startup, but expects the keyspace to already exist.

### 9.4 Deploy Backend

```bash
# Deploy the maplefile stack (backend service)
docker stack deploy -c ~/stacks/maplefile-stack.yml maplefile

# Check service created
docker service ls | grep maplefile
# Should show: maplefile_backend   0/1   registry.digitalocean.com/ssp/maplefile-backend:prod
```

### 9.5 Watch Backend Start

```bash
# Watch service come up
watch docker service ps maplefile_backend
# Press Ctrl+C when you see "Running" status

# Or check directly
docker service ps maplefile_backend
```

**Expected output after 1-2 minutes:**

```
ID             NAME                   NODE                                CURRENT STATE         ERROR
abc123...      maplefile_backend.1    mapleopentech-swarm-worker-8-prod       Running 1 minute ago
```

**If stuck in "Preparing" or "Starting":**

Check logs:

```bash
docker service logs -f maplefile_backend
# Look for errors
# Press Ctrl+C to exit
```

**Common issues:**
- **Image pull failed**: Worker-8 must authenticate with registry (see Step 7.3)
- **No suitable node**: Worker-8 missing `maplefile-backend=true` label (see Step 4.4)
- **Secrets missing**: Create all required secrets (see Step 8)
- **Keyspace missing**: Run Step 9.3 to initialize Cassandra keyspace

### 9.6 Verify Backend Health

**Important:** The backend container runs on worker-8, not the manager. You must SSH to worker-8 to test directly.

**From your local machine, SSH to worker-8:**

```bash
ssh dockeradmin@143.110.212.253

# Find backend container ID
docker ps | grep maplefile

# Test health endpoint
docker exec $(docker ps -q --filter "name=maplefile_backend") \
  wget --no-verbose --tries=1 --spider http://localhost:8000/api/v1/health

# Should return: HTTP/1.1 200 OK
```

**Alternative: Check from manager (without SSHing to worker-8):**

```bash
# On manager
docker service ps maplefile_backend
# Should show: Running X minutes ago

# Check logs for successful startup
docker service logs maplefile_backend --tail 30
# Should show: migrations completed, services initialized
```

**✅ Checkpoint:** Backend service running, health check passes

---

## Building and Pushing the Docker Image

If you haven't built and pushed your image yet, follow these steps:

### From Your Local Machine

```bash
# Navigate to maplefile-backend directory
cd ~/go/src/codeberg.org/mapleopentech/monorepo/cloud/maplefile-backend

# Build for production (linux/amd64 platform)
docker build -f Dockerfile --rm -t registry.digitalocean.com/ssp/maplefile-backend:prod --platform linux/amd64 .

# Tag the image
docker tag registry.digitalocean.com/ssp/maplefile-backend:prod registry.digitalocean.com/ssp/maplefile-backend:prod

# Authenticate with DigitalOcean registry (if not already)
doctl registry login

# Push to registry
docker push registry.digitalocean.com/ssp/maplefile-backend:prod
```

**Using Task (if configured in Taskfile.yml):**

Check if your `Taskfile.yml` has a deploy task:

```bash
# Check available tasks
task --list

# If deploy task exists and is configured for production:
task deploy
```

---

## Troubleshooting

### Problem: Backend Won't Start

**Symptom:** `docker service ps maplefile_backend` shows "Starting" for > 2 minutes

**Check:**

```bash
# View logs
docker service logs maplefile_backend --tail 100

# Common issues:
# 1. Can't reach databases
#    - Verify Cassandra running: docker service ls | grep cassandra
#    - Verify Redis running: docker service ls | grep redis
#    - Check backend is on mapleopentech-private-prod network

# 2. Secrets missing
docker secret ls
# Should show: maplefile_jwt_secret, redis_password, maplefile_mailgun_api_key, spaces_access_key, spaces_secret_key

# 3. Image pull failed
docker service ps maplefile_backend
# If you see "image not found", verify worker-8 authenticated with registry
ssh dockeradmin@143.110.212.253
docker pull registry.digitalocean.com/ssp/maplefile-backend:prod
```

### Problem: Keyspace Does Not Exist

**Symptom:** Backend logs show `keyspace maplefile does not exist` or `failed to create user`

**Cause:** Cassandra keyspace not initialized before backend startup

**Fix:**

```bash
# 1. SSH to any Cassandra node
ssh dockeradmin@<cassandra-node-ip>

# 2. Find the Cassandra container
export CASSANDRA_CONTAINER=$(docker ps --filter "name=cassandra" -q | head -1)

# 3. Create the keyspace
docker exec -it $CASSANDRA_CONTAINER cqlsh -e "
CREATE KEYSPACE IF NOT EXISTS maplefile
WITH replication = {
    'class': 'SimpleStrategy',
    'replication_factor': 3
};"

# 4. Verify keyspace exists
docker exec -it $CASSANDRA_CONTAINER cqlsh -e "DESCRIBE KEYSPACE maplefile;"

# 5. Restart backend to retry migrations
docker service update --force maplefile_backend
```

**Prevention:** Always run Step 9.3 (Initialize Cassandra Keyspace) before first backend deployment.

### Problem: Migrations Error

**Symptom:** Backend logs show `failed to open source, "file://migrations": open .: no such file or directory`

**Cause:** Docker image was built without migrations directory or worker-8 has cached old image.

**Fix:**

```bash
# 1. Verify Dockerfile includes migrations
# On your local machine:
cd ~/monorepo/cloud/maplefile-backend
grep "COPY.*migrations" Dockerfile
# Should show: COPY --from=build-env /app/migrations ./migrations

# 2. Rebuild and push image with migrations
docker build -f Dockerfile --rm -t registry.digitalocean.com/ssp/maplefile-backend:prod --platform linux/amd64 .
docker push registry.digitalocean.com/ssp/maplefile-backend:prod

# 3. Clear cache on worker-8 and force fresh pull
ssh dockeradmin@143.110.212.253
docker system prune -af

# 4. Authenticate and pull fresh image
doctl registry login
docker pull registry.digitalocean.com/ssp/maplefile-backend:prod

# 5. Verify new image has /app and migrations
docker run --rm registry.digitalocean.com/ssp/maplefile-backend:prod ls -la /app
# Should show: migrations/ directory

# 6. Redeploy backend from manager
ssh dockeradmin@143.110.210.162
docker stack deploy -c ~/stacks/maplefile-stack.yml maplefile
```

### Problem: Can't Connect to Cassandra

**Symptom:** Backend logs show connection errors to Cassandra

**Check:**

```bash
# 1. Verify Cassandra services are running
docker service ls | grep cassandra
# Should show: cassandra_cassandra-1, cassandra_cassandra-2, cassandra_cassandra-3

# 2. Verify backend is on mapleopentech-private-prod network
docker service inspect maplefile_backend --format '{{range .Spec.TaskTemplate.Networks}}{{.Target}} {{end}}'
# Should include mapleopentech-private-prod

# 3. Test DNS resolution from backend container
ssh dockeradmin@143.110.212.253
docker exec $(docker ps -q --filter "name=maplefile_backend") nslookup cassandra-1
# Should resolve to an IP
```

### Problem: Can't Connect to Redis

**Symptom:** Backend logs show connection errors to Redis

**Check:**

```bash
# 1. Verify Redis service is running
docker service ls | grep redis
# Should show: redis_redis   1/1

# 2. Check Redis password secret exists
docker secret ls | grep redis_password

# 3. Test connection from backend container
ssh dockeradmin@143.110.212.253
docker exec $(docker ps -q --filter "name=maplefile_backend") ping redis -c 2
# Should get responses
```

### Problem: Worker-8 Pulling Old/Cached Image

**Symptom:** Changes not reflected after deploying, logs show old errors

**Cause:** Docker Swarm worker has cached old image and isn't pulling fresh one.

**Fix:**

```bash
# On worker-8
ssh dockeradmin@143.110.212.253

# 1. Remove all cached images and containers
docker system prune -af
# WARNING: This removes ALL unused images, be careful!

# 2. Verify authentication with registry
doctl registry login

# 3. Force pull fresh image
docker pull registry.digitalocean.com/ssp/maplefile-backend:prod

# 4. Verify it's the new image (check digest)
docker images registry.digitalocean.com/ssp/maplefile-backend:prod --digests

# 5. From manager, force service update
ssh dockeradmin@143.110.210.162
docker service update --force maplefile_backend

# Watch it redeploy
watch docker service ps maplefile_backend
```

---

## Next Steps

✅ **You now have:**
- Worker-8 droplet configured and joined to Docker Swarm
- Backend service running on worker-8
- Backend connected to both networks (private for databases, public for Caddy)
- Docker secrets configured for sensitive credentials
- Backend pulling from DigitalOcean Container Registry
- Health checks passing
- Backend ready to receive requests from Caddy reverse proxy

**Next guide:**
- **10_maplefile_caddy.md** - Deploy Caddy reverse proxy with automatic SSL/TLS
- Configure HTTPS with Let's Encrypt (automatic certificate management)
- Security headers and rate limiting
- Make your backend accessible via https://maplefile.ca

**Maintenance commands:**

```bash
# View backend logs
docker service logs -f maplefile_backend

# Update backend to new version (IMPORTANT: Follow this complete process)
# Step 1: Build and push new image (on local machine)
cd ~/go/src/codeberg.org/mapleopentech/monorepo/cloud/maplefile-backend
task deploy  # Or: docker build and push manually

# Step 2: Deploy using the deployment script (on manager)
ssh dockeradmin@<manager-public-ip>
./deploy-maplefile-prod.sh

# Scale backend (add more replicas)
vi ~/stacks/maplefile-stack.yml  # Change replicas: 1 to replicas: 3
docker stack deploy -c ~/stacks/maplefile-stack.yml maplefile

# Quick restart (without redeploying)
docker service update --force maplefile_backend
```

### Deployment Script

For convenient deployments, create this script on the manager node:

```bash
# On manager node
cat > ~/stacks/deploy-maplefile-prod.sh << 'EOF'
#!/bin/bash
# Deploy MapleFile Backend to Production
# This script removes the existing service, pulls the latest image, and redeploys

set -e

echo "🛑 Removing existing backend service..."
docker service rm maplefile_backend || true

echo "⏳ Waiting for service to be fully removed..."
sleep 10

echo "📦 Pulling latest production image..."
docker pull registry.digitalocean.com/ssp/maplefile-backend:prod

echo "🚀 Deploying stack..."
cd ~/stacks
docker stack deploy -c maplefile-stack.yml maplefile

echo "⏳ Waiting for service to start..."
sleep 10

echo "📋 Following logs (Ctrl+C to exit)..."
docker service logs maplefile_backend --tail 50 -f
EOF

# Make executable
chmod +x ~/stacks/deploy-maplefile-prod.sh
```

**Usage:**

```bash
# From manager node
cd ~/stacks
./deploy-maplefile-prod.sh
```

**What the script does:**
1. Removes existing backend service (allows clean redeploy)
2. Waits 10 seconds for service removal
3. Pulls latest production image from registry
4. Deploys the maplefile stack
5. Waits 10 seconds for service startup
6. Follows backend logs (Ctrl+C to exit)

---

## Network Architecture Notes

**Private Network (mapleopentech-private-prod):**
- Cassandra cluster (3 nodes) - Shared by ALL apps
- Redis - Shared by ALL apps (different DB numbers per app)
- Both MaplePress and MapleFile use same infrastructure
- MaplePress uses Cassandra keyspace: `maplepress`, Redis DB: 0
- MapleFile uses Cassandra keyspace: `maplefile`, Redis DB: 1

**Public Network (mapleopentech-public-prod):**
- Each app gets its own Caddy instance
- maplefile-backend-caddy (to be deployed in Part 2)
- maplepress-backend-caddy (already deployed)
- Both share the public network but serve different domains

**Resource Efficiency:**
- 5 infrastructure workers support unlimited apps
- Each app only needs 1 worker for its backend + Caddy
- Shared databases, caches, and infrastructure

---

**Last Updated**: January 2025
**Maintained By**: Infrastructure Team
