# Extra Operations and Domain Changes

**Audience**: DevOps Engineers, Infrastructure Team
**Time to Complete**: Varies by operation
**Prerequisites**: Completed guides 01-07 (full MaplePress deployment)

---

## Overview

This guide covers additional operations and changes that you might need to perform on your production infrastructure:

1. **Domain Changes**
   - Changing backend domain (e.g., `getmaplepress.ca` → `getmaplepress.net`)
   - Changing frontend domain (e.g., `getmaplepress.com` → `getmaplepress.app`)
2. **SSL Certificate Management**
3. **Scaling Operations**
4. **Backup and Recovery**

---

## Table of Contents

1. [Change Backend Domain](#operation-1-change-backend-domain)
2. [Change Frontend Domain](#operation-2-change-frontend-domain)
3. [Change Both Domains](#operation-3-change-both-domains-at-once)
4. [Force SSL Certificate Renewal](#operation-4-force-ssl-certificate-renewal)
5. [Scale Backend Horizontally](#operation-5-scale-backend-horizontally)

---

## Operation 1: Change Backend Domain

**Scenario:** Changing backend API domain from `getmaplepress.ca` → `getmaplepress.net`

**Impact:**
- ✅ Backend becomes available at new domain
- ❌ Old domain stops working
- ⚠️ Frontend needs CORS update to allow new backend domain
- ⚠️ SSL certificate automatically obtained for new domain
- ⚠️ Downtime: ~2-5 minutes during redeployment

### Step 1: DNS Configuration

**First, point the new domain to worker-6:**

1. Log into your DNS provider (DigitalOcean, Cloudflare, etc.)
2. Create DNS A records for new domain:

```
Type: A Record
Name: getmaplepress.net
Value: <worker-6-public-ip>
TTL: 300 (5 minutes)

Type: A Record
Name: www.getmaplepress.net
Value: <worker-6-public-ip>
TTL: 300
```

3. Wait for DNS propagation (5-60 minutes):

```bash
# Test DNS from your local machine
dig getmaplepress.net +short
# Should show: <worker-6-public-ip>

dig www.getmaplepress.net +short
# Should show: <worker-6-public-ip>

# Alternative test
nslookup getmaplepress.net
```

### Step 2: Update Backend Caddyfile

**On manager node:**

```bash
ssh dockeradmin@<manager-public-ip>
cd ~/stacks/caddy-config

# Backup old Caddyfile
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)

# Edit Caddyfile
vi Caddyfile
```

**Change this:**

```caddy
# OLD DOMAIN
getmaplepress.ca www.getmaplepress.ca {
    reverse_proxy maplepress-backend:8000 {
        # ... config ...
    }
}
```

**To this:**

```caddy
# NEW DOMAIN
getmaplepress.net www.getmaplepress.net {
    reverse_proxy maplepress-backend:8000 {
        header_up X-Real-IP {remote_host}
        header_up X-Forwarded-For {remote_host}
        header_up X-Forwarded-Proto {scheme}
        header_up X-Forwarded-Host {host}

        # IMPORTANT: Preserve Origin header for CORS
        header_up Origin {http.request.header.Origin}
    }

    log {
        output stdout
        format json
        level INFO
    }

    header {
        X-Frame-Options "SAMEORIGIN"
        X-Content-Type-Options "nosniff"
        X-XSS-Protection "1; mode=block"
        Strict-Transport-Security "max-age=31536000; includeSubDomains"
        Referrer-Policy "strict-origin-when-cross-origin"
        -Server
    }
}
```

Save: `Esc`, `:wq`, `Enter`

### Step 3: Update CORS Configuration

**Update the stack file to allow the frontend to call the new backend domain:**

```bash
# Still on manager node
cd ~/stacks
vi maplepress-stack.yml
```

**Find this line:**

```yaml
- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com
```

**No change needed** - The CORS config is for what origins can call the backend, not the backend's domain itself. The frontend (`getmaplepress.com`) will now call `getmaplepress.net` instead of `getmaplepress.ca`.

### Step 4: Redeploy Backend Stack

```bash
# Remove old stack
docker stack rm maplepress
sleep 10

# Remove old config (contains old domain)
docker config rm maplepress_caddyfile

# Deploy with new domain
docker stack deploy -c maplepress-stack.yml maplepress

# Watch services come up
docker service ps maplepress_backend
docker service ps maplepress_backend-caddy
```

### Step 5: Verify SSL Certificate

**Caddy will automatically obtain SSL certificates for the new domain:**

```bash
# Watch Caddy logs for certificate acquisition
docker service logs -f maplepress_backend-caddy

# You should see logs like:
# "certificate obtained successfully"
# "serving https://getmaplepress.net"
```

**Test from local machine:**

```bash
# Test new domain with HTTPS
curl -I https://getmaplepress.net/health
# Should return: HTTP/2 200

# Verify SSL certificate
curl -vI https://getmaplepress.net/health 2>&1 | grep "subject:"
# Should show: subject: CN=getmaplepress.net

# Test CORS
curl -v -H "Origin: https://getmaplepress.com" https://getmaplepress.net/health 2>&1 | grep "access-control-allow-origin"
# Should show: access-control-allow-origin: https://getmaplepress.com
```

### Step 6: Update Frontend to Use New Backend Domain

**On your local machine:**

```bash
cd ~/go/src/codeberg.org/mapleopentech/monorepo/web/maplepress-frontend

# Update production environment file
vi .env.production
```

**Change:**

```bash
# OLD
VITE_API_BASE_URL=https://getmaplepress.ca

# NEW
VITE_API_BASE_URL=https://getmaplepress.net
```

**Rebuild and redeploy frontend:**

```bash
# Build with new backend URL
npm run build

# Verify the new URL is in the build
grep -r "getmaplepress.net" dist/assets/*.js | head -2
# Should show: getmaplepress.net

# SSH to worker-7 and update the frontend build
ssh dockeradmin@<worker-7-public-ip>
cd /var/www/monorepo/web/maplepress-frontend

# Pull latest code
git pull origin main

# Rebuild
npm run build

# Verify symlink
ls -la /var/www/maplepress-frontend
# Should point to: /var/www/monorepo/web/maplepress-frontend/dist

exit
```

### Step 7: Test End-to-End

```bash
# Visit frontend in browser
open https://getmaplepress.com

# Open DevTools (F12) → Network tab
# Verify API calls now go to: https://getmaplepress.net
# Verify status: 200 (not 0 or CORS errors)
```

### Step 8: (Optional) Keep Old Domain Working

If you want both domains to work temporarily:

```bash
# Edit Caddyfile to include BOTH domains
vi ~/stacks/caddy-config/Caddyfile
```

```caddy
# Support both old and new domains
getmaplepress.ca www.getmaplepress.ca, getmaplepress.net www.getmaplepress.net {
    reverse_proxy maplepress-backend:8000 {
        # ... same config ...
    }
}
```

Then redeploy as in Step 4.

### Rollback Procedure

If something goes wrong:

```bash
# 1. Restore old Caddyfile
cd ~/stacks/caddy-config
cp Caddyfile.backup.YYYYMMDD Caddyfile

# 2. Redeploy
cd ~/stacks
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress

# 3. Restore frontend .env.production
cd ~/go/src/codeberg.org/mapleopentech/monorepo/web/maplepress-frontend
# Change back to: VITE_API_BASE_URL=https://getmaplepress.ca
# Rebuild and redeploy
```

**✅ Backend domain change complete!**

---

## Operation 2: Change Frontend Domain

**Scenario:** Changing frontend domain from `getmaplepress.com` → `getmaplepress.app`

**Impact:**
- ✅ Frontend becomes available at new domain
- ❌ Old domain stops working
- ⚠️ Backend CORS needs update to allow new frontend domain
- ⚠️ SSL certificate automatically obtained for new domain
- ⚠️ Downtime: ~2-5 minutes during redeployment

### Step 1: DNS Configuration

**Point the new domain to worker-7:**

```
Type: A Record
Name: getmaplepress.app
Value: <worker-7-public-ip>
TTL: 300

Type: A Record
Name: www.getmaplepress.app
Value: <worker-7-public-ip>
TTL: 300
```

**Test DNS propagation:**

```bash
dig getmaplepress.app +short
# Should show: <worker-7-public-ip>

nslookup getmaplepress.app
```

### Step 2: Update Frontend Caddyfile

**On manager node:**

```bash
ssh dockeradmin@<manager-public-ip>
cd ~/stacks/maplepress-frontend-caddy-config

# Backup
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)

# Edit
vi Caddyfile
```

**Change this:**

```caddy
# OLD DOMAIN
getmaplepress.com www.getmaplepress.com {
    root * /var/www/maplepress-frontend
    # ... config ...
}
```

**To this:**

```caddy
# NEW DOMAIN
getmaplepress.app www.getmaplepress.app {
    root * /var/www/maplepress-frontend
    file_server
    try_files {path} /index.html
    encode gzip

    log {
        output stdout
        format json
        level INFO
    }

    header {
        X-Frame-Options "SAMEORIGIN"
        X-Content-Type-Options "nosniff"
        X-XSS-Protection "1; mode=block"
        Strict-Transport-Security "max-age=31536000; includeSubDomains"
        Referrer-Policy "strict-origin-when-cross-origin"
        -Server
    }

    @static {
        path *.js *.css *.png *.jpg *.jpeg *.gif *.svg *.woff *.woff2 *.ttf *.eot *.ico
    }
    header @static Cache-Control "public, max-age=31536000, immutable"
}
```

Save: `Esc`, `:wq`, `Enter`

### Step 3: Update Backend CORS Configuration

**CRITICAL:** The backend needs to allow the new frontend domain:

```bash
cd ~/stacks
vi maplepress-stack.yml
```

**Find this line:**

```yaml
- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com
```

**Change to:**

```yaml
- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.app,https://www.getmaplepress.app
```

**If you want to support BOTH old and new domains temporarily:**

```yaml
- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com,https://getmaplepress.app,https://www.getmaplepress.app
```

### Step 4: Redeploy Backend (for CORS update)

```bash
# Backend CORS config changed, must redeploy
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress

# Verify backend running
docker service ps maplepress_backend
```

### Step 5: Redeploy Frontend

```bash
# Remove frontend stack
docker stack rm maplepress-frontend
sleep 10
docker config rm maplepress-frontend_caddyfile

# Deploy with new domain
docker stack deploy -c maplepress-frontend-stack.yml maplepress-frontend

# Watch it come up
docker service ps maplepress-frontend_caddy
```

### Step 6: Verify SSL Certificate

**Test from local machine:**

```bash
# Test new frontend domain
curl -I https://getmaplepress.app
# Should return: HTTP/2 200

# Verify SSL certificate
curl -vI https://getmaplepress.app 2>&1 | grep "subject:"
# Should show: subject: CN=getmaplepress.app
```

### Step 7: Test CORS from New Frontend

```bash
# Visit new frontend in browser
open https://getmaplepress.app

# Open DevTools (F12)
# Network tab: Verify API calls succeed
# Console tab: Should be NO CORS errors
```

### Step 8: Verify Backend Accepts New Origin

```bash
# Test CORS from backend perspective
curl -v -H "Origin: https://getmaplepress.app" https://getmaplepress.ca/health 2>&1 | grep "access-control-allow-origin"
# Should show: access-control-allow-origin: https://getmaplepress.app
```

### Rollback Procedure

```bash
# 1. Restore old frontend Caddyfile
cd ~/stacks/maplepress-frontend-caddy-config
cp Caddyfile.backup.YYYYMMDD Caddyfile

# 2. Restore old backend CORS config
cd ~/stacks
vi maplepress-stack.yml
# Change back to: https://getmaplepress.com,https://www.getmaplepress.com

# 3. Redeploy both
docker stack rm maplepress
docker stack rm maplepress-frontend
sleep 10
docker config rm maplepress_caddyfile
docker config rm maplepress-frontend_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress
docker stack deploy -c maplepress-frontend-stack.yml maplepress-frontend
```

**✅ Frontend domain change complete!**

---

## Operation 3: Change Both Domains at Once

**Scenario:** Changing both domains simultaneously:
- Backend: `getmaplepress.ca` → `api.maplepress.io`
- Frontend: `getmaplepress.com` → `app.maplepress.io`

**Benefits:**
- Single maintenance window
- Coordinated cutover
- Clean brand migration

**Downtime:** ~5-10 minutes

### Complete Process

```bash
# ==============================================================================
# STEP 1: DNS Configuration (Do this first, wait for propagation)
# ==============================================================================

# Backend DNS:
# A Record: api.maplepress.io → <worker-6-public-ip>
# A Record: www.api.maplepress.io → <worker-6-public-ip>

# Frontend DNS:
# A Record: app.maplepress.io → <worker-7-public-ip>
# A Record: www.app.maplepress.io → <worker-7-public-ip>

# Test DNS
dig api.maplepress.io +short  # Should show worker-6 IP
dig app.maplepress.io +short  # Should show worker-7 IP

# ==============================================================================
# STEP 2: Update Backend Caddyfile
# ==============================================================================
ssh dockeradmin@<manager-public-ip>
cd ~/stacks/caddy-config
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)
vi Caddyfile

# Change domain from getmaplepress.ca to api.maplepress.io
# (Keep all other config the same)

# ==============================================================================
# STEP 3: Update Frontend Caddyfile
# ==============================================================================
cd ~/stacks/maplepress-frontend-caddy-config
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)
vi Caddyfile

# Change domain from getmaplepress.com to app.maplepress.io
# (Keep all other config the same)

# ==============================================================================
# STEP 4: Update Backend CORS for New Frontend Domain
# ==============================================================================
cd ~/stacks
vi maplepress-stack.yml

# Change:
# - SECURITY_CORS_ALLOWED_ORIGINS=https://app.maplepress.io,https://www.app.maplepress.io

# ==============================================================================
# STEP 5: Update Frontend .env.production for New Backend
# ==============================================================================
ssh dockeradmin@<worker-7-public-ip>
cd /var/www/monorepo/web/maplepress-frontend
vi .env.production

# Change:
# VITE_API_BASE_URL=https://api.maplepress.io

# Rebuild
npm run build

# Verify new URL in build
grep -r "api.maplepress.io" dist/assets/*.js | head -2

exit

# ==============================================================================
# STEP 6: Coordinated Deployment (Back on Manager)
# ==============================================================================
ssh dockeradmin@<manager-public-ip>
cd ~/stacks

# Remove both stacks
docker stack rm maplepress
docker stack rm maplepress-frontend
sleep 10

# Remove configs
docker config rm maplepress_caddyfile
docker config rm maplepress-frontend_caddyfile

# Deploy both stacks
docker stack deploy -c maplepress-stack.yml maplepress
docker stack deploy -c maplepress-frontend-stack.yml maplepress-frontend

# ==============================================================================
# STEP 7: Verify Both Services
# ==============================================================================
docker service ls | grep maplepress
# Should show 3 services all 1/1:
#   maplepress_backend
#   maplepress_backend-caddy
#   maplepress-frontend_caddy

# ==============================================================================
# STEP 8: Test End-to-End (Local Machine)
# ==============================================================================
# Test backend
curl -I https://api.maplepress.io/health
# Should return: HTTP/2 200

# Test frontend
curl -I https://app.maplepress.io
# Should return: HTTP/2 200

# Test CORS
curl -v -H "Origin: https://app.maplepress.io" https://api.maplepress.io/health 2>&1 | grep "access-control"
# Should show: access-control-allow-origin: https://app.maplepress.io

# Test in browser
open https://app.maplepress.io
# DevTools → Network: Verify calls to api.maplepress.io succeed
```

**✅ Both domain changes complete!**

---

## Operation 4: Force SSL Certificate Renewal

**Scenario:** You need to manually renew SSL certificates (rarely needed - Caddy auto-renews)

### When You Might Need This

- Testing certificate renewal process
- Certificate was revoked
- Manual intervention required after failed auto-renewal

### Backend Certificate Renewal

```bash
# SSH to worker-6
ssh dockeradmin@<worker-6-public-ip>

# Get Caddy container ID
docker ps | grep maplepress_backend-caddy

# Access Caddy container
docker exec -it <container-id> sh

# Inside container - force certificate renewal
caddy reload --config /etc/caddy/Caddyfile --force

# Or restart Caddy to trigger renewal
exit

# Back on worker-6
docker service update --force maplepress_backend-caddy

# Watch logs for certificate acquisition
docker service logs -f maplepress_backend-caddy | grep -i certificate
```

### Frontend Certificate Renewal

```bash
# SSH to worker-7
ssh dockeradmin@<worker-7-public-ip>

# Get Caddy container ID
docker ps | grep maplepress-frontend

# Force reload
docker exec <container-id> caddy reload --config /etc/caddy/Caddyfile --force

# Or force restart
exit
docker service update --force maplepress-frontend_caddy

# Watch logs
docker service logs -f maplepress-frontend_caddy | grep -i certificate
```

### Verify New Certificate

```bash
# From local machine
openssl s_client -connect getmaplepress.ca:443 -servername getmaplepress.ca < /dev/null 2>/dev/null | openssl x509 -noout -dates

# Should show:
# notBefore=Nov  5 12:00:00 2025 GMT
# notAfter=Feb   3 12:00:00 2026 GMT
```

---

## Operation 5: Scale Backend Horizontally

**Scenario:** Your backend needs to handle more traffic - add more replicas

### Considerations

- Each replica needs database connections
- Cassandra can handle the load (QUORUM with 3 nodes)
- Redis connections are pooled
- Stateless design allows easy horizontal scaling

### Scale to 3 Replicas

```bash
# On manager node
cd ~/stacks
vi maplepress-stack.yml

# Find backend service, change replicas
# FROM:
#   deploy:
#     replicas: 1

# TO:
#   deploy:
#     replicas: 3

# Redeploy
docker stack deploy -c maplepress-stack.yml maplepress

# Watch replicas come up
watch docker service ps maplepress_backend
# Press Ctrl+C when all show Running

# Verify all healthy
docker service ps maplepress_backend --filter "desired-state=running"
# Should show 3 replicas
```

### Load Balancing

Caddy automatically load balances between replicas:

```bash
# Test load balancing
for i in {1..10}; do
  curl -s https://getmaplepress.ca/health
  sleep 1
done

# Check which replicas handled requests
docker service logs maplepress_backend | grep "GET /health" | tail -20
# You should see different container IDs handling requests
```

### Scale Back Down

```bash
# Edit stack file
vi ~/stacks/maplepress-stack.yml

# Change back to replicas: 1
# Redeploy
docker stack deploy -c maplepress-stack.yml maplepress

# Verify
docker service ps maplepress_backend
# Should show only 1 replica running, others Shutdown
```

---

## Quick Reference: Domain Change Checklist

### Backend Domain Change

- [ ] Update DNS A records (point new domain to worker-6)
- [ ] Wait for DNS propagation (5-60 minutes)
- [ ] Backup Caddyfile: `cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)`
- [ ] Update backend Caddyfile with new domain
- [ ] Redeploy backend stack
- [ ] Verify SSL certificate obtained for new domain
- [ ] Update frontend `.env.production` with new backend URL
- [ ] Rebuild and redeploy frontend
- [ ] Test CORS end-to-end

### Frontend Domain Change

- [ ] Update DNS A records (point new domain to worker-7)
- [ ] Wait for DNS propagation
- [ ] Backup frontend Caddyfile
- [ ] Update frontend Caddyfile with new domain
- [ ] **Update backend CORS** in `maplepress-stack.yml`
- [ ] Redeploy backend (for CORS)
- [ ] Redeploy frontend stack
- [ ] Verify SSL certificate
- [ ] Test in browser (no CORS errors)

---

## Troubleshooting Domain Changes

### Problem: SSL Certificate Not Obtained

**Symptom:** After domain change, HTTPS doesn't work

```bash
# Check Caddy logs
docker service logs maplepress_backend-caddy --tail 100 | grep -i "acme\|certificate"

# Common issues:
# 1. DNS not propagated - wait longer
# 2. Port 80 not accessible - check firewall
# 3. Let's Encrypt rate limit - wait 1 hour
```

**Fix:**

```bash
# Verify DNS resolves
dig <new-domain> +short
# Must show correct worker IP

# Verify port 80 accessible
curl http://<new-domain>
# Should redirect to HTTPS

# If rate limited, wait and retry
# Let's Encrypt limit: 5 certificates per domain per week
```

### Problem: CORS Errors After Domain Change

**Symptom:** Frontend shows CORS errors in browser console

**Cause:** Forgot to update backend CORS configuration

**Fix:**

```bash
# Check backend CORS config
cat ~/stacks/maplepress-stack.yml | grep CORS
# Should include NEW frontend domain

# Update if needed
vi ~/stacks/maplepress-stack.yml
# Add new frontend domain to SECURITY_CORS_ALLOWED_ORIGINS

# Redeploy backend
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress

# Test CORS
curl -v -H "Origin: https://<new-frontend-domain>" https://<backend-domain>/health 2>&1 | grep "access-control"
```

### Problem: Old Domain Still Works

**Symptom:** Both old and new domains work

**Cause:** Caddyfile includes both domains

**Expected Behavior:** This is fine during migration - you can support both

**To Remove Old Domain:**

```bash
# Edit Caddyfile and remove old domain
vi ~/stacks/caddy-config/Caddyfile
# Remove old domain from the domain list

# Redeploy
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress
```

---

**Last Updated**: November 2025
**Maintained By**: Infrastructure Team