# Deploy Caddy Reverse Proxy with Automatic SSL: Part 2

**Audience**: Junior DevOps Engineers, Infrastructure Team
**Time to Complete**: 20-30 minutes
**Prerequisites**:
- ✅ Completed guide **05_backend.md** (Backend deployed and running)
- ✅ Backend service accessible on `mapleopentech-public-prod` network
- ✅ Domain name pointing to worker-6 public IP
- ✅ Email address for Let's Encrypt SSL certificate notifications

---

## Overview

This guide configures **Caddy** as a reverse proxy with automatic SSL/TLS certificate management for your MaplePress backend.

### What is a Reverse Proxy?

Think of a reverse proxy as a "receptionist" for your backend:

1. **Internet user** → Makes request to `https://yourdomain.com`
2. **Caddy (receptionist)** → Receives the request
   - Handles SSL/TLS (HTTPS encryption)
   - Checks rate limits
   - Adds security headers
3. **Caddy forwards** → Sends request to your backend at `http://maplepress-backend:8000`
4. **Backend** → Processes request, sends response back
5. **Caddy** → Returns response to user

**Why use a reverse proxy?**
- Your backend doesn't need to handle SSL certificates
- One place to manage security, rate limiting, and headers
- Can load balance across multiple backend instances
- Protects your backend from direct internet exposure

### Why Caddy Instead of NGINX?

**Caddy's killer feature: Automatic HTTPS**
- Caddy automatically gets SSL certificates from Let's Encrypt
- Automatically renews them before expiry (no cron jobs!)
- Zero manual certificate management
- Simpler configuration (10 lines vs 200+ for NGINX)

**What you'll build:**
- Caddy reverse proxy on worker-6
- Automatic SSL certificate from Let's Encrypt
- HTTP to HTTPS automatic redirection
- Security headers and rate limiting
- Zero-downtime certificate renewals (automatic)

**Architecture:**
```
Internet
  ↓ HTTPS (port 443)
Caddy (worker-6)
  ↓ HTTP (port 8000, internal network only)
Backend (worker-6)
  ↓ Private network
Databases (Cassandra, Redis, Meilisearch on other workers)
```

**Key concept:** Caddy and Backend are both on worker-6, connected via the `mapleopentech-public-prod` Docker overlay network. Caddy can reach Backend by the hostname `backend` - Docker's built-in DNS resolves this to the backend container's IP automatically.

---

## Step 1: Verify DNS Configuration

Before deploying Caddy, your domain must point to worker-6 (where Caddy will run).

### 1.1 Check Current DNS

**From your local machine:**

```bash
# Check where your domain currently points
dig yourdomain.com +short

# Should return worker-6's public IP
# If it returns nothing or wrong IP, continue to next step
```

### 1.2 Update DNS Records

**If DNS is not configured or points to wrong server:**

1. Log into your domain registrar (where you bought `yourdomain.com`)
2. Find DNS settings / DNS management / Manage DNS
3. Add or update these A records:

| Type | Name | Value | TTL |
|------|------|-------|-----|
| A | @ | `<worker-6-public-ip>` | 3600 |
| A | www | `<worker-6-public-ip>` | 3600 |

**Example with real IPs (do NOT use these - use YOUR worker-6 IP):**
- Type: `A`
- Name: `@` (root domain)
- Value: `157.230.45.67` ← Replace with your worker-6 public IP
- TTL: `3600` (1 hour)

**What this does:**
- `@` record: Makes `yourdomain.com` point to worker-6
- `www` record: Makes `www.yourdomain.com` point to worker-6
- Both domains will work with Caddy

### 1.3 Wait for DNS Propagation

DNS changes take 5-10 minutes (sometimes up to 1 hour).

**Test from your local machine:**

```bash
# Test root domain
dig yourdomain.com +short
# Should return: <worker-6-public-ip>

# Test www subdomain
dig www.yourdomain.com +short
# Should return: <worker-6-public-ip>

# Alternative test
nslookup yourdomain.com
# Should show: Address: <worker-6-public-ip>
```

**Keep testing every minute until both commands return worker-6's public IP.**

⚠️ **CRITICAL:** Do NOT proceed until DNS resolves correctly! Caddy cannot get SSL certificates if DNS doesn't point to the right server.

### 1.4 Verify Firewall Allows HTTP/HTTPS

**On worker-6, check firewall:**

```bash
# SSH to worker-6
ssh dockeradmin@<worker-6-public-ip>

# Check firewall rules
sudo ufw status | grep -E "80|443"

# Should show:
# 80/tcp                     ALLOW       Anywhere
# 443/tcp                    ALLOW       Anywhere
```

**If ports are NOT open:**

```bash
# Allow HTTP (needed for Let's Encrypt)
sudo ufw allow 80/tcp

# Allow HTTPS (needed for encrypted traffic)
sudo ufw allow 443/tcp

# Verify
sudo ufw status | grep -E "80|443"

# Exit back to local machine
exit
```

**✅ Checkpoint:** DNS resolves to worker-6, ports 80 and 443 are open

---

## Step 2: Prepare Caddy Configuration

### 2.1 Create Caddy Config Directory

**On manager node:**

```bash
# SSH to manager
ssh dockeradmin@<manager-public-ip>

# Create directory for Caddy config
cd ~/stacks
mkdir -p caddy-config
cd caddy-config
```

### 2.2 Create Caddyfile

The **Caddyfile** is Caddy's configuration file. It's much simpler than NGINX config.

```bash
vi Caddyfile
```

**Paste this configuration:**

```caddy
{
    # Global options
    email YOUR_EMAIL@example.com

    # Use Let's Encrypt production (not staging)
    # Staging is for testing - production is for real certificates
    acme_ca https://acme-v02.api.letsencrypt.org/directory
}

# Your domain configuration
yourdomain.com www.yourdomain.com {
    # Reverse proxy all requests to backend
    reverse_proxy maplepress-backend:8000 {
        # Forward real client IP to backend
        header_up X-Real-IP {remote_host}
        header_up X-Forwarded-For {remote_host}
        header_up X-Forwarded-Proto {scheme}
        header_up X-Forwarded-Host {host}

        # Preserve Origin header for CORS (required for frontend)
        header_up Origin {http.request.header.Origin}
    }

    # Logging
    log {
        output stdout
        format json
        level INFO
    }

    # Security headers (Caddy adds many by default)
    header {
        # Prevent clickjacking
        X-Frame-Options "SAMEORIGIN"
        # Prevent MIME type sniffing
        X-Content-Type-Options "nosniff"
        # Enable XSS protection
        X-XSS-Protection "1; mode=block"
        # HSTS - Force HTTPS for 1 year
        Strict-Transport-Security "max-age=31536000; includeSubDomains"
        # Control referrer information
        Referrer-Policy "strict-origin-when-cross-origin"
        # Remove Server header (security by obscurity)
        -Server
    }

    # Rate limiting (requires Caddy plugin - see note below)
    # For basic setups, you can skip this or add later
}
```

**Important replacements:**
1. Replace `YOUR_EMAIL@example.com` with your real email (Let's Encrypt sends expiry warnings here)
2. Replace `yourdomain.com` with your actual domain (e.g., `getmaplepress.ca`)
3. Replace `www.yourdomain.com` with your www subdomain

Save: `Esc`, then `:wq`, then `Enter`

**Understanding the config:**

- **`backend:8000`** - This is how Caddy reaches your backend
  - `backend` = hostname of your backend service (Docker DNS resolves this)
  - `8000` = port your backend listens on
  - No IP address needed - Docker overlay network handles it!

**Important: Service Name vs Hostname**

When you run `docker service ls`, you see:
```
maplepress_backend   1/1   registry.digitalocean.com/ssp/maplepress_backend:prod
```

But in the Caddyfile, we use `maplepress-backend:8000`, not `maplepress_backend:8000`. Why?

- **Service name** (`maplepress_backend`): How Docker Swarm identifies the service
  - Used in: `docker service ls`, `docker service logs maplepress_backend`
  - Format: `{stack-name}_{service-name}`

- **Hostname** (`maplepress-backend`): How containers reach each other on the network
  - Used in: Caddyfile, application configs, container-to-container communication
  - Defined in the stack file: `hostname: maplepress-backend`

**Think of it like this:**
- Service name = The employee's official HR name (full legal name)
- Hostname = The nickname everyone uses in the office

Other containers don't care about the service name - they use the hostname for DNS resolution.

- **`header_up`** - Passes information to your backend about the real client
  - Without this, backend would think all requests come from Caddy
  - Your backend can log real client IPs for security/debugging

- **Security headers** - Tell browsers how to handle your site securely
  - HSTS: Forces browsers to always use HTTPS
  - X-Frame-Options: Prevents your site being embedded in iframes (clickjacking protection)
  - X-Content-Type-Options: Prevents MIME confusion attacks

### 2.3 Understanding the Automatic SSL Magic

**What happens when Caddy starts:**

1. Caddy sees `yourdomain.com` in the Caddyfile
2. Caddy checks if domain points to this server (DNS check)
3. Caddy requests SSL certificate from Let's Encrypt
4. Let's Encrypt does a challenge (HTTP-01 via port 80)
5. Caddy receives certificate and stores it in `/data/caddy`
6. Caddy automatically serves HTTPS on port 443
7. Caddy automatically redirects HTTP → HTTPS

**You don't have to:**
- Manually run certbot commands
- Stop the server to renew certificates
- Set up cron jobs
- Mount certificate directories

**Caddy handles ALL of this automatically!**

---

## Step 3: Deploy Caddy Service

### 3.1 Update Stack File to Add Caddy

Since we're using **Option C** (backend + backend-caddy in one stack), we need to UPDATE the existing `maplepress-stack.yml` file to add the `backend-caddy` service.

**On manager node:**

```bash
cd ~/stacks
vi maplepress-stack.yml
```

**Replace the entire file with this complete stack (includes both backend AND backend-caddy):**

```yaml
version: '3.8'

networks:
  mapleopentech-private-prod:
    external: true
  mapleopentech-public-prod:
    external: true

volumes:
  caddy_data:
    # Caddy stores certificates here
  caddy_config:
    # Caddy stores config cache here

configs:
  caddyfile:
    file: ./caddy-config/Caddyfile

secrets:
  maplepress_jwt_secret:
    external: true
  maplepress_ip_encryption_key:
    external: true
  redis_password:
    external: true
  meilisearch_master_key:
    external: true
  spaces_access_key:
    external: true
  spaces_secret_key:
    external: true

services:
  backend:
    image: registry.digitalocean.com/ssp/maplepress_backend:prod
    hostname: maplepress-backend
    networks:
      - mapleopentech-public-prod
      - mapleopentech-private-prod
    secrets:
      - maplepress_jwt_secret
      - maplepress_ip_encryption_key
      - redis_password
      - meilisearch_master_key
      - spaces_access_key
      - spaces_secret_key
    environment:
      - APP_ENVIRONMENT=production
      - APP_VERSION=1.0.0
      - SERVER_HOST=0.0.0.0
      - SERVER_PORT=8000
      - DATABASE_HOSTS=cassandra-1:9042,cassandra-2:9042,cassandra-3:9042
      - DATABASE_KEYSPACE=maplepress
      - DATABASE_CONSISTENCY=QUORUM
      - DATABASE_REPLICATION=3
      - DATABASE_MIGRATIONS_PATH=file://migrations
      - MEILISEARCH_HOST=http://meilisearch:7700
      - AWS_ENDPOINT=https://nyc3.digitaloceanspaces.com
      - AWS_REGION=nyc3
      - AWS_BUCKET_NAME=maplepress-prod
      - SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com
      - LOGGER_LEVEL=info
      - LOGGER_FORMAT=json
    entrypoint: ["/bin/sh", "-c"]
    command:
      - |
        cd /app
        export APP_JWT_SECRET=$$(cat /run/secrets/maplepress_jwt_secret)
        export SECURITY_IP_ENCRYPTION_KEY=$$(cat /run/secrets/maplepress_ip_encryption_key)
        export CACHE_PASSWORD=$$(cat /run/secrets/redis_password)
        export MEILISEARCH_API_KEY=$$(cat /run/secrets/meilisearch_master_key)
        export AWS_ACCESS_KEY=$$(cat /run/secrets/spaces_access_key)
        export AWS_SECRET_KEY=$$(cat /run/secrets/spaces_secret_key)
        export CACHE_HOST=redis
        export CACHE_PORT=6379
        export CACHE_DB=0
        exec /app/maplepress-backend daemon
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.backend == true
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 3
      update_config:
        parallelism: 1
        delay: 10s
        order: start-first
      resources:
        limits:
          memory: 1G
        reservations:
          memory: 512M
    healthcheck:
      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "--header=X-Tenant-ID: healthcheck", "http://localhost:8000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 60s

  backend-caddy:
    image: caddy:2-alpine
    hostname: caddy
    networks:
      - mapleopentech-public-prod
    ports:
      # Port 80 - HTTP (for Let's Encrypt challenges and HTTP→HTTPS redirect)
      # Using mode: host to bind directly to worker-6's network interface
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      # Port 443 - HTTPS (encrypted traffic)
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      # Port 443 UDP - HTTP/3 support (optional, modern protocol)
      - target: 443
        published: 443
        protocol: udp
        mode: host
    configs:
      # Docker config - automatically distributed to worker-6
      - source: caddyfile
        target: /etc/caddy/Caddyfile
    volumes:
      # Persistent storage for certificates
      - caddy_data:/data
      # Persistent storage for config cache
      - caddy_config:/config
    deploy:
      replicas: 1
      placement:
        constraints:
          # Deploy on same node as backend (worker-6)
          - node.labels.backend == true
      restart_policy:
        condition: on-failure
        delay: 5s
        # Note: No max_attempts - Docker will keep trying indefinitely
        # This prevents the service from scaling to 0 after a few failures
      update_config:
        # Rolling updates (zero downtime)
        parallelism: 1
        delay: 10s
        order: start-first
      resources:
        limits:
          # Caddy is lightweight - 256MB is plenty
          memory: 256M
        reservations:
          memory: 128M
    # Note: No healthcheck - Caddy's built-in health monitoring is sufficient
    # Docker healthchecks can cause SIGTERM shutdowns during startup or cert renewal
```

Save: `Esc`, then `:wq`, then `Enter`

**Understanding the stack file:**

- **`mapleopentech-public-prod` network**: Shared network with backend
  - Both Caddy and Backend are connected here
  - Allows Caddy to reach Backend by hostname
  - `external: true` means we created this network earlier (in 05_backend.md)

- **Ports** (using `mode: host`):
  - Port 80 (HTTP) - Needed for Let's Encrypt certificate challenges
  - Port 443 (HTTPS TCP) - Encrypted traffic
  - Port 443 (HTTPS UDP) - HTTP/3 support
  - **Why `mode: host`?** Binds directly to worker-6's network interface
    - `mode: ingress` (default) uses Docker Swarm routing mesh (any node can accept traffic)
    - `mode: host` binds only on the specific node running Caddy
    - Since we're pinning Caddy to worker-6 anyway, `host` mode is more reliable
    - Prevents potential routing issues with Let's Encrypt challenges

- **Configs** (not volumes for Caddyfile):
  - `caddyfile` - Docker config that's automatically distributed to worker-6
  - Why not a volume mount? Because the file is on the manager, but Caddy runs on worker-6
  - Docker configs solve this: they're stored in the swarm and sent to the right node
  - Configs are immutable - to update, you must redeploy the stack

- **Volumes**:
  - `caddy_data` - Stores SSL certificates (persists across restarts)
  - `caddy_config` - Stores runtime config cache (persists across restarts)
  - Without these volumes, Caddy would re-request certificates on every restart!

- **Placement constraint**: `node.labels.backend == true`
  - Ensures Caddy deploys on worker-6 (same node as backend)
  - Why? Better performance (no network hops between Caddy → Backend)
  - Backend is only accessible via overlay network, not public internet

- **`order: start-first`**: Zero-downtime updates
  - When updating, new Caddy starts before old one stops
  - Traffic smoothly transitions to new container

- **No healthcheck** (intentionally removed):
  - Caddy has built-in health monitoring and graceful shutdown
  - Docker healthchecks can cause false positives during:
    - Certificate acquisition (ACME challenge response)
    - Certificate renewal (brief backend disconnection)
    - Container startup (DNS resolution delays)
  - Failed healthchecks send SIGTERM, causing restart loops
  - **If you see SIGTERM in logs every 60-90 seconds** - healthcheck is still present
  - Solution: Remove the entire `healthcheck:` section from the YAML

### 3.2 Verify Prerequisites

Before deploying, let's make sure everything is ready:

```bash
# 1. Check backend is running
docker service ls | grep backend
# Should show: maplepress_backend   1/1   registry.digitalocean.com/ssp/maplepress_backend:prod

# 2. Check mapleopentech-public-prod network exists
docker network ls | grep mapleopentech-public-prod
# Should show: mapleopentech-public-prod   overlay   swarm

# 3. Verify worker-6 has backend label
docker node inspect mapleopentech-swarm-worker-6-prod --format '{{.Spec.Labels}}'
# Should show: map[backend:true]

# 4. Test backend health from manager
docker service ps maplepress_backend
# Should show: Running
```

If any of these checks fail, go back to **05_backend.md** and complete those steps first.

### 3.3 Deploy Caddy Stack

**If this is a re-deployment** (you've deployed Caddy before), first remove the old stack and config:

```bash
# Check if Caddy stack exists
docker stack ls | grep caddy

# If it exists, remove it first
docker stack rm maplepress
sleep 5

# Remove old config (if exists)
docker config rm maplepress_caddyfile 2>/dev/null || true

# Now proceed with fresh deployment below
```

**For fresh deployment** (or after removing old stack above):

```bash
# Make sure you're in the stacks directory
cd ~/stacks

# Deploy Caddy
docker stack deploy -c maplepress-stack.yml maplepress

# Expected output:
# Creating service maplepress_backend-caddy
```

**If you get error: "only updates to Labels are allowed"** - this means an old config exists. Run the removal commands above first.

### 3.4 Watch Caddy Start and Get Certificate

This is the exciting part! Watch Caddy automatically get your SSL certificate:

```bash
# Watch service come up
docker service ps maplepress_backend-caddy

# Should show:
# ID        NAME            NODE        DESIRED STATE   CURRENT STATE
# abc123    maplepress_backend-caddy.1   worker-6    Running         Running 30 seconds ago
```

**Watch the logs to see certificate acquisition:**

```bash
# Follow logs in real-time
docker service logs -f maplepress_backend-caddy

# You'll see something like:
# {"level":"info","ts":1234567890,"msg":"using provided configuration"}
# {"level":"info","ts":1234567890,"msg":"tls: obtaining certificate"}
# {"level":"info","ts":1234567890,"msg":"attempting ACME challenge","domain":"yourdomain.com"}
# {"level":"info","ts":1234567890,"msg":"successfully obtained certificate","domain":"yourdomain.com"}
# {"level":"info","ts":1234567890,"msg":"serving initial configuration"}
```

**This takes 10-30 seconds.** Press `Ctrl+C` when you see "successfully obtained certificate".

**What just happened?**
1. Caddy read your Caddyfile
2. Caddy saw your domain name
3. Caddy checked DNS points to this server
4. Caddy contacted Let's Encrypt ACME servers
5. Let's Encrypt verified domain ownership (via HTTP challenge on port 80)
6. Let's Encrypt issued certificate
7. Caddy stored certificate in `/data/caddy` volume
8. Caddy configured HTTPS automatically

**No manual steps required!**

---

## Step 4: Verify Caddy is Working

### 4.1 Test HTTP to HTTPS Redirect

**From your local machine:**

```bash
# Test HTTP (should redirect to HTTPS)
curl -I http://yourdomain.com

# Expected output:
# HTTP/1.1 308 Permanent Redirect
# Location: https://yourdomain.com/
```

The `308` status code means "permanent redirect" - Caddy is automatically redirecting HTTP to HTTPS.

### 4.2 Test HTTPS (Encrypted Connection)

```bash
# Test HTTPS
curl https://yourdomain.com

# Should return your backend response
# Look for your backend's HTML/JSON response
```

**If you get certificate errors**, wait 30 more seconds - certificate might still be provisioning.

### 4.3 Test Backend Connection Through Caddy

```bash
# Test health endpoint (if your backend has one)
curl https://yourdomain.com/health

# Test API endpoint (adjust path to match your backend)
curl https://yourdomain.com/api/v1/version
```

### 4.4 Verify Certificate Details

Check the certificate is valid and from Let's Encrypt:

```bash
# Check certificate details
curl -vI https://yourdomain.com 2>&1 | grep -A 10 "Server certificate"

# Look for:
# - subject: CN=yourdomain.com
# - issuer: Let's Encrypt
# - expire date: (should be ~90 days from now)
```

### 4.5 Test in Browser

Open your browser and visit: `https://yourdomain.com`

**You should see:**
- 🔒 **Green padlock** in address bar
- No security warnings
- Valid certificate when clicking the padlock
- Your backend's response

**Click the padlock icon** to inspect the certificate:
- Issued to: yourdomain.com
- Issued by: Let's Encrypt Authority X3 (or similar)
- Valid from: [today's date]
- Valid to: [~90 days from now]

**✅ Checkpoint:** HTTPS working, valid certificate, backend responding through Caddy

---

## Step 5: Verify Complete Setup

### 5.1 Check All Services Running

**On manager:**

```bash
docker service ls

# Should show all services running:
# ID        NAME                        MODE       REPLICAS   IMAGE
# ...       cassandra_cassandra-1       replicated 1/1        cassandra:4.1
# ...       cassandra_cassandra-2       replicated 1/1        cassandra:4.1
# ...       cassandra_cassandra-3       replicated 1/1        cassandra:4.1
# ...       redis_redis                 replicated 1/1        redis:7-alpine
# ...       meilisearch_meilisearch     replicated 1/1        getmeili/meilisearch:v1.5
# ...       maplepress_backend          replicated 1/1        registry.digitalocean.com/ssp/maplepress_backend:prod
# ...       maplepress_backend-caddy                 replicated 1/1        caddy:2-alpine
```

All should show `1/1` in REPLICAS column.

### 5.2 Check Networks

```bash
docker network ls | grep maple

# Should show:
# abc123...   mapleopentech-private-prod    overlay   swarm
# def456...   mapleopentech-public-prod     overlay   swarm
```

### 5.3 Verify Service Connectivity

**Test that Caddy can reach backend:**

```bash
# SSH to worker-6
ssh dockeradmin@<worker-6-public-ip>

# Get Caddy container ID
CADDY_CONTAINER=$(docker ps -q --filter "name=maplepress_backend-caddy")

# Test DNS resolution (Caddy → Backend)
docker exec $CADDY_CONTAINER nslookup backend
# Should show: backend resolves to an IP (10.x.x.x)

# Test HTTP connection (Caddy → Backend)
docker exec $CADDY_CONTAINER wget -qO- http://maplepress-backend:8000/health
# Should return backend health response

# Exit back to local machine
exit
```

### 5.4 Check Logs for Any Errors

```bash
# Check Caddy logs
docker service logs maplepress_backend-caddy --tail 50
# Look for any errors (should be clean)

# Check backend logs
docker service logs maplepress_backend --tail 50
# Verify backend is receiving requests through Caddy
```

**✅ Final Checkpoint:** All services running, HTTPS working, no errors in logs

---

## Understanding Certificate Auto-Renewal

### How It Works

**Caddy automatically handles certificate renewal - you don't need to do anything!**

**The renewal process:**
1. Caddy checks certificate expiry daily
2. When certificate has ~30 days left, Caddy requests renewal
3. Caddy performs ACME challenge with Let's Encrypt
4. New certificate obtained and hot-swapped (zero downtime)
5. Old certificate safely discarded

**Comparison to NGINX:**
- **NGINX**: Set up cron job, stop service, run certbot, restart service
- **Caddy**: Does everything automatically in the background

### Viewing Certificate Status

```bash
# SSH to worker-6
ssh dockeradmin@<worker-6-public-ip>

# Get Caddy container ID
CADDY_CONTAINER=$(docker ps -q --filter "name=maplepress_backend-caddy")

# List certificates
docker exec $CADDY_CONTAINER caddy list-modules

# View certificate storage
docker exec $CADDY_CONTAINER ls -la /data/caddy/certificates/

# Exit
exit
```

### What If Renewal Fails?

**Caddy will retry automatically.** It has built-in retry logic:
- Retries every 6 hours if renewal fails
- Sends error logs (check with `docker service logs maplepress_backend-caddy`)
- Certificate is valid for 90 days, renewal starts at 60 days - plenty of time

**Common renewal failure causes:**
1. **Port 80 blocked** - Let's Encrypt needs HTTP access for challenges
2. **DNS changed** - Domain no longer points to this server
3. **Rate limited** - Too many certificate requests (5 per domain per week limit)

**You'll get an email** if renewal fails (sent to the email in your Caddyfile).

---

## Maintenance Tasks

### View Caddy Logs

```bash
# On manager node

# Recent logs
docker service logs maplepress_backend-caddy --tail 50

# Follow logs in real-time
docker service logs -f maplepress_backend-caddy

# Press Ctrl+C to exit
```

### Update Caddyfile Configuration

If you need to change Caddy's configuration:

```bash
# On manager node
cd ~/stacks/caddy-config
vi Caddyfile

# Make your changes, then save

# IMPORTANT: Docker configs are immutable - must remove old stack first!
cd ~/stacks
docker stack rm maplepress

# Wait for stack to fully stop
sleep 5
docker service ls | grep caddy
# Should show nothing

# Remove old config
docker config rm maplepress_caddyfile

# Redeploy with new config
docker stack deploy -c maplepress-stack.yml maplepress

# Watch the update happen
docker service ps maplepress_backend-caddy

# Check logs to verify new config loaded
docker service logs maplepress_backend-caddy --tail 20
```

**Important:** Docker configs are immutable. You can't just edit the file and reload - you must:
1. Remove the stack (`docker stack rm`)
2. Remove the old config (`docker config rm`)
3. Redeploy (`docker stack deploy`)

**Why not just `docker service update --force`?**
- `update --force` restarts the service but doesn't update the config
- Docker configs cannot be modified once created
- You'll get error: "only updates to Labels are allowed"
- Solution: Remove and recreate the stack

**Caddy validates config before applying** - if there's an error in your Caddyfile, the deploy will fail and you can fix it and try again.

### Restart Caddy

```bash
# Force restart (may cause brief downtime)
docker service update --force maplepress_backend-caddy

# Watch it restart
watch docker service ps maplepress_backend-caddy
# Press Ctrl+C when done
```

### Update Caddy to Newer Version

```bash
# On manager
cd ~/stacks

# Edit stack file to use newer version
vi maplepress-stack.yml
# Change: image: caddy:2-alpine
# To: image: caddy:2.7-alpine (specific version)

# Redeploy (zero downtime due to order: start-first)
docker stack deploy -c maplepress-stack.yml maplepress

# Watch rolling update
watch docker service ps maplepress_backend-caddy
```

### Scale Backend (Multiple Replicas)

If you scale your backend to multiple replicas, Caddy automatically load balances:

```bash
# Scale backend to 3 replicas
docker service scale maplepress_backend=3

# Caddy automatically detects all backend instances
# Distributes traffic using round-robin
# No Caddy config changes needed!
```

### Add Another Domain

To serve multiple domains from the same backend:

```bash
# Edit Caddyfile
cd ~/stacks/caddy-config
vi Caddyfile

# Add another domain block:
# anotherdomain.com {
#     reverse_proxy maplepress-backend:8000
# }

# Reload Caddy
docker service update --force maplepress_backend-caddy

# Caddy automatically gets a certificate for the new domain too!
```

---

## Troubleshooting

### Problem: Caddy Won't Start

**Symptom:** `docker service ps maplepress_backend-caddy` shows service restarting

**Check:**

```bash
# View logs for errors
docker service logs maplepress_backend-caddy --tail 100

# Common issues:

# 1. Invalid Caddyfile syntax
#    - Fix syntax in ~/stacks/caddy-config/Caddyfile
#    - Redeploy: docker stack deploy -c maplepress-stack.yml maplepress

# 2. Port 80 or 443 already in use
#    - Check: ssh dockeradmin@<worker-6-public-ip> && sudo ss -tlnp | grep ":80\|:443"
#    - Stop conflicting service

# 3. Network not found
docker network ls | grep mapleopentech-public-prod
# If missing, create it: docker network create --driver overlay --attachable mapleopentech-public-prod
```

### Problem: Deployment Fails with "only updates to Labels are allowed"

**Symptom:** When running `docker stack deploy`, you get:

```
failed to update config maplepress_backend-caddyfile: Error response from daemon:
rpc error: code = InvalidArgument desc = only updates to Labels are allowed
```

**Cause:** Docker config already exists from a previous deployment, and configs are immutable.

**Solution:**

```bash
# 1. Remove the existing stack
docker stack rm maplepress

# 2. Wait for it to fully stop
sleep 10
docker service ls | grep caddy
# Should show nothing

# 3. Remove the old config
docker config ls | grep caddyfile
# Note the config ID/name

docker config rm maplepress_caddyfile

# 4. Verify config is gone
docker config ls | grep caddyfile
# Should show nothing

# 5. Now redeploy fresh
cd ~/stacks
docker stack deploy -c maplepress-stack.yml maplepress

# 6. Verify it's working
docker service ps maplepress_backend-caddy
```

**Why this happens:**
- Docker configs are immutable (cannot be updated once created)
- When you run `docker stack deploy` again, it tries to update the existing config
- Docker only allows label updates on configs, nothing else
- Solution: Remove old config, then redeploy (creates new config)

**Prevention:**
- Always remove stack before redeploying: `docker stack rm maplepress && sleep 5 && docker config rm maplepress_caddyfile`
- Or use versioned config names: `caddyfile_v1`, `caddyfile_v2`, etc.

### Problem: Service Scales to 0/1 (Keeps Shutting Down)

**Symptom:** `docker service ls` shows `maplepress_backend-caddy 0/1` - service completely stopped

**Cause:** Docker Swarm hit `max_attempts` in restart policy and gave up retrying

**Solution:**

```bash
# 1. Check if service is scaled to 0
docker service ls | grep caddy
# Shows: maplepress_backend-caddy  0/1

# 2. Remove the old stack completely
docker stack rm maplepress
sleep 5

# 3. Verify maplepress-stack.yml does NOT have max_attempts
cat ~/stacks/maplepress-stack.yml | grep -A3 "restart_policy"
# Should show:
#   restart_policy:
#     condition: on-failure
#     delay: 5s
# (NO max_attempts line!)

# 4. If max_attempts exists, remove it:
vi ~/stacks/maplepress-stack.yml
# Delete the line: max_attempts: 3

# 5. Redeploy fresh
docker stack deploy -c maplepress-stack.yml maplepress

# 6. Watch it start
docker service ps maplepress_backend-caddy
```

**Why this happens:**
- During initial setup, if something fails 3 times (e.g., backend not ready)
- Docker Swarm stops trying and scales service to 0
- Even after you fix the issue, service stays at 0
- Solution: Remove `max_attempts` so Docker keeps retrying indefinitely

### Problem: Caddy Keeps Restarting with SIGTERM (Every 60-90 Seconds)

**Symptom:**
- Service shows `1/1` but keeps restarting
- Logs show: `"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}`
- Service runs for 60-90 seconds, then gets SIGTERM and restarts
- Pattern repeats indefinitely

**Cause:** Docker healthcheck in the stack YAML is failing and sending SIGTERM

**Check logs for SIGTERM:**

```bash
docker service logs maplepress_backend-caddy --tail 30 | grep SIGTERM
# If you see: "signal":"SIGTERM" - this is the issue
```

**Solution - Remove Docker Healthcheck:**

```bash
# 1. Remove the Caddy stack
docker stack rm maplepress
sleep 10

# 2. Remove the old Caddyfile config
docker config rm maplepress_caddyfile

# 3. Recreate the Caddyfile config (use your actual domain and email)
cat > /tmp/Caddyfile << 'EOF'
{
    # Global options
    email YOUR_EMAIL@example.com

    # Use Let's Encrypt production (not staging)
    acme_ca https://acme-v02.api.letsencrypt.org/directory
}

# Your domain configuration
yourdomain.com www.yourdomain.com {
    # Reverse proxy all requests to backend
    reverse_proxy maplepress-backend:8000 {
        # Forward real client IP to backend
        header_up X-Real-IP {remote_host}
        header_up X-Forwarded-For {remote_host}
        header_up X-Forwarded-Proto {scheme}
        header_up X-Forwarded-Host {host}

        # Preserve Origin header for CORS (required for frontend)
        header_up Origin {http.request.header.Origin}
    }

    # Logging
    log {
        output stdout
        format json
        level INFO
    }

    # Security headers (Caddy adds many by default)
    header {
        # Prevent clickjacking
        X-Frame-Options "SAMEORIGIN"
        # Prevent MIME type sniffing
        X-Content-Type-Options "nosniff"
        # Enable XSS protection
        X-XSS-Protection "1; mode=block"
        # HSTS - Force HTTPS for 1 year
        Strict-Transport-Security "max-age=31536000; includeSubDomains"
        # Control referrer information
        Referrer-Policy "strict-origin-when-cross-origin"
        # Remove Server header (security by obscurity)
        -Server
    }
}
EOF

docker config create caddyfile /tmp/Caddyfile

# 4. Edit the stack YAML to remove healthcheck section
vi ~/stacks/maplepress-stack.yml

# Remove these lines (if they exist):
#    healthcheck:
#      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:80"]
#      interval: 30s
#      timeout: 5s
#      retries: 3
#      start_period: 30s

# Replace with:
#    # Note: No healthcheck - Caddy's built-in health monitoring is sufficient
#    # Docker healthchecks can cause SIGTERM shutdowns during startup or cert renewal

# 5. Redeploy with updated stack file (no healthcheck)
docker stack deploy -c ~/stacks/maplepress-stack.yml maplepress

# 6. Verify it stays running (no more SIGTERM)
docker service logs -f maplepress_backend-caddy
# Should NOT see repeated SIGTERM messages
```

**Why this happens:**
- Docker healthcheck tests `http://localhost:80` which redirects to HTTPS
- The healthcheck doesn't handle HTTPS redirects properly or has SSL verification issues
- After 3 failed healthchecks (30s interval × 3 = 90 seconds), Docker sends SIGTERM
- Caddy has built-in health monitoring, so Docker healthchecks are unnecessary
- **Important**: Both Caddyfile health checks AND Docker YAML healthchecks can cause this issue

**Verification - Service Should Stay Up:**

```bash
# Check service is running continuously
docker service ps maplepress_backend-caddy
# Should show ONE task in "Running" state (no recent restarts)

# Check uptime is increasing (not resetting every 90 seconds)
watch -n 5 'docker service ps maplepress_backend-caddy'
# Uptime should keep increasing: "Running 2 minutes", "Running 5 minutes", etc.
```

### Problem: Certificate Acquisition Fails

**Symptom:** Logs show "failed to obtain certificate" or "ACME challenge failed"

**Check:**

```bash
# 1. Verify DNS points to worker-6
dig yourdomain.com +short
# Should return: <worker-6-public-ip>

# 2. Verify port 80 is accessible from internet
curl -I http://yourdomain.com
# Should connect (even if it redirects)

# 3. Check firewall allows port 80
ssh dockeradmin@<worker-6-public-ip>
sudo ufw status | grep 80
# Should show: 80/tcp ALLOW Anywhere

# 4. Check Let's Encrypt rate limits
# Let's Encrypt allows 5 certificates per domain per week
# If you hit limit, wait 7 days or use staging for testing
```

**To use Let's Encrypt staging (for testing):**

```caddy
{
    email YOUR_EMAIL@example.com
    # Change to staging server (issues test certificates)
    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}
```

### Problem: 502 Bad Gateway

**Symptom:** `curl https://yourdomain.com` returns "502 Bad Gateway"

**Cause:** Caddy can't reach backend

**Check:**

```bash
# 1. Is backend running?
docker service ls | grep backend
# Must show: maplepress_backend 1/1

# 2. Are both on same network?
docker service inspect maplepress_backend --format '{{range .Spec.TaskTemplate.Networks}}{{.Target}} {{end}}'
docker service inspect maplepress_backend-caddy --format '{{range .Spec.TaskTemplate.Networks}}{{.Target}} {{end}}'
# Both should include: mapleopentech-public-prod

# 3. Test from Caddy container
ssh dockeradmin@<worker-6-public-ip>
CADDY_CONTAINER=$(docker ps -q --filter "name=maplepress_backend-caddy")
docker exec $CADDY_CONTAINER wget -qO- http://maplepress-backend:8000/health
# Should return backend health response (not connection refused)

# 4. Check backend logs
docker service logs maplepress_backend --tail 50
# Look for errors
```

### Problem: CORS Error When Accessing Backend from Frontend

**Symptom:** Browser console shows:
```
Access to fetch at 'https://getmaplepress.ca/api/...' from origin 'https://getmaplepress.com'
has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present
```

**Root Cause:** Caddy is not forwarding the `Origin` header to the backend, so the backend's CORS middleware can't reflect it back.

**Check:**

```bash
# Test CORS from command line
curl -H "Origin: https://getmaplepress.com" \
     -I https://getmaplepress.ca/health

# Look for this header in response:
# Access-Control-Allow-Origin: https://getmaplepress.com

# If missing, Caddy is not forwarding Origin header
```

**Solution:**

Update the Caddyfile to preserve the Origin header:

```bash
# On manager node
vi ~/stacks/Caddyfile
```

Find the `reverse_proxy maplepress-backend:8000` section and ensure it has:

```caddy
reverse_proxy maplepress-backend:8000 {
    header_up X-Real-IP {remote_host}
    header_up X-Forwarded-For {remote_host}
    header_up X-Forwarded-Proto {scheme}
    header_up X-Forwarded-Host {host}

    # IMPORTANT: Preserve Origin header for CORS
    header_up Origin {http.request.header.Origin}
}
```

Then redeploy:

```bash
# Remove stack and config (configs are immutable)
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile

# Redeploy with updated config
cd ~/stacks
docker stack deploy -c maplepress-stack.yml maplepress

# Verify
docker service ls | grep caddy
```

**Verify the fix:**

```bash
# Test CORS again
curl -H "Origin: https://getmaplepress.com" \
     -I https://getmaplepress.ca/health

# Should now include:
# Access-Control-Allow-Origin: https://getmaplepress.com
# Access-Control-Allow-Credentials: true
```

**Why this works:**
- Backend CORS middleware reflects whatever origin it receives
- Without forwarding, backend gets no Origin header → no CORS headers in response
- With forwarding, backend gets Origin header → reflects it back → CORS works ✅

### Problem: "DNS Not Found" or "Connection Refused"

**Symptom:** Can't access https://yourdomain.com - connection times out

**Check:**

```bash
# 1. Test DNS resolution
dig yourdomain.com +short
# Should return: <worker-6-public-ip>

# If wrong:
# - Update DNS A record at your domain registrar
# - Wait 5-10 minutes for propagation
# - Use: dig @8.8.8.8 yourdomain.com (test with Google DNS)

# 2. Test direct IP access
curl http://<worker-6-public-ip>
# Should return Caddy response or redirect

# If times out:
# - Check firewall on worker-6
# - Verify ports 80 and 443 are open
ssh dockeradmin@<worker-6-public-ip>
sudo ufw status | grep -E "80|443"
```

### Problem: Certificate Shows as Invalid

**Symptom:** Browser shows "Your connection is not private" warning

**Possible causes:**

1. **Using staging certificate** (for testing)
   - Change Caddyfile to use production Let's Encrypt
   - Remove staging certificates: `docker volume rm maplepress_backend-caddy_data`
   - Redeploy Caddy

2. **DNS mismatch** - Domain in Caddyfile doesn't match accessed domain
   - Verify Caddyfile domain matches what you're accessing
   - Example: Caddyfile says `example.com` but you access `www.example.com`

3. **Certificate not fully provisioned** - Wait 30 more seconds

4. **Check logs:**
   ```bash
   docker service logs maplepress_backend-caddy | grep -i certificate
   ```

### Problem: HTTP Not Redirecting to HTTPS

**Symptom:** Accessing `http://yourdomain.com` doesn't redirect to `https://`

**This is unusual** - Caddy redirects automatically. Check:

```bash
# Test redirect explicitly
curl -I http://yourdomain.com

# Should show:
# HTTP/1.1 308 Permanent Redirect
# Location: https://yourdomain.com/

# If not redirecting:
# 1. Check Caddyfile syntax (HTTPS should be automatic for domain names)
# 2. Restart Caddy: docker service update --force maplepress_backend-caddy
# 3. Check logs: docker service logs maplepress_backend-caddy
```

---

## Security Checklist

✅ **Firewall configured** - Ports 22, 80, 443 open; all others closed
✅ **SSH keys** - Password-less SSH authentication
✅ **dockeradmin user** - Not running as root
✅ **Automatic HTTPS** - SSL/TLS via Let's Encrypt
✅ **Auto-renewal** - Certificates renew automatically (no manual intervention)
✅ **Security headers** - HSTS, X-Frame-Options, X-Content-Type-Options configured
✅ **Rate limiting** - Optional, can add via Caddy plugins
✅ **Private network** - Databases not exposed to internet
✅ **Backend not exposed** - Only accessible via Caddy, not directly from internet

---

## Next Steps

✅ **You now have:**
- Caddy reverse proxy with automatic SSL/TLS
- HTTPS-secured backend at https://yourdomain.com
- Automatic HTTP to HTTPS redirection
- Zero-maintenance certificate renewals
- Security headers configured
- Complete production infrastructure

**Recommended next actions:**

1. **Test your application** - Verify all endpoints work through Caddy
2. **Set up monitoring** - Monitor Caddy and backend logs for errors
3. **Configure backups** - Back up your backend data (databases)
4. **Load testing** - Test your infrastructure under load
5. **CI/CD pipeline** - Automate backend deployments

**Key advantages over NGINX:**
- ✅ No manual certificate management (saves hours per year)
- ✅ Simpler configuration (10 lines vs 200+)
- ✅ Automatic renewals (no cron jobs, no service restarts)
- ✅ Same performance for your use case
- ✅ Built-in HTTP/2 and HTTP/3 support

**Maintenance tasks:**

```bash
# View Caddy logs
docker service logs -f maplepress_backend-caddy

# View backend logs
docker service logs -f maplepress_backend

# Update backend to new version (zero downtime)
cd ~/stacks
docker stack deploy -c maplepress-stack.yml maplepress

# Update Caddyfile (zero downtime config reload)
cd ~/stacks/caddy-config
vi Caddyfile
cd ~/stacks
docker stack deploy -c maplepress-stack.yml maplepress

# Check certificate status (automatic, just for info)
docker service logs maplepress_backend-caddy | grep -i certificate
```

**Your complete infrastructure:**

```
Internet (HTTPS)
  ↓
Caddy (worker-6) - Automatic SSL, reverse proxy
  ↓
Backend (worker-6) - Your Go application
  ↓ mapleopentech-private-prod network
Cassandra (workers 1-3) - Database cluster
Redis (worker-4) - Cache
Meilisearch (worker-5) - Search engine
```

---

**Last Updated**: November 2025
**Maintained By**: Infrastructure Team

**Changelog:**
- November 2025: Removed health checks from Caddyfile AND Docker healthcheck from stack YAML (both caused SIGTERM shutdown issues), removed `max_attempts` from restart policy (prevents service scaling to 0), changed port mode to `host` for better reliability
