Bartlomiej Mika 755d54a99d Initial commit: Open sourcing all of the Maple Open Technologies code.

2025-12-02 14:33:08 -05:00

21 KiB

Raw Blame History

Extra Operations and Domain Changes

Audience: DevOps Engineers, Infrastructure Team Time to Complete: Varies by operation Prerequisites: Completed guides 01-07 (full MaplePress deployment)

Overview

This guide covers additional operations and changes that you might need to perform on your production infrastructure:

Domain Changes
- Changing backend domain (e.g., getmaplepress.ca → getmaplepress.net)
- Changing frontend domain (e.g., getmaplepress.com → getmaplepress.app)
SSL Certificate Management
Scaling Operations
Backup and Recovery

Change Backend Domain
Change Frontend Domain
Change Both Domains
Force SSL Certificate Renewal
Scale Backend Horizontally

Operation 1: Change Backend Domain

Scenario: Changing backend API domain from getmaplepress.ca → getmaplepress.net

Impact:

✅ Backend becomes available at new domain
❌ Old domain stops working
⚠️ Frontend needs CORS update to allow new backend domain
⚠️ SSL certificate automatically obtained for new domain
⚠️ Downtime: ~2-5 minutes during redeployment

Step 1: DNS Configuration

First, point the new domain to worker-6:

Log into your DNS provider (DigitalOcean, Cloudflare, etc.)
Create DNS A records for new domain:

Type: A Record
Name: getmaplepress.net
Value: <worker-6-public-ip>
TTL: 300 (5 minutes)

Type: A Record
Name: www.getmaplepress.net
Value: <worker-6-public-ip>
TTL: 300

Wait for DNS propagation (5-60 minutes):

# Test DNS from your local machine
dig getmaplepress.net +short
# Should show: <worker-6-public-ip>

dig www.getmaplepress.net +short
# Should show: <worker-6-public-ip>

# Alternative test
nslookup getmaplepress.net

Step 2: Update Backend Caddyfile

On manager node:

ssh dockeradmin@<manager-public-ip>
cd ~/stacks/caddy-config

# Backup old Caddyfile
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)

# Edit Caddyfile
vi Caddyfile

Change this:

# OLD DOMAIN
getmaplepress.ca www.getmaplepress.ca {
    reverse_proxy maplepress-backend:8000 {
        # ... config ...
    }
}

To this:

# NEW DOMAIN
getmaplepress.net www.getmaplepress.net {
    reverse_proxy maplepress-backend:8000 {
        header_up X-Real-IP {remote_host}
        header_up X-Forwarded-For {remote_host}
        header_up X-Forwarded-Proto {scheme}
        header_up X-Forwarded-Host {host}

        # IMPORTANT: Preserve Origin header for CORS
        header_up Origin {http.request.header.Origin}
    }

    log {
        output stdout
        format json
        level INFO
    }

    header {
        X-Frame-Options "SAMEORIGIN"
        X-Content-Type-Options "nosniff"
        X-XSS-Protection "1; mode=block"
        Strict-Transport-Security "max-age=31536000; includeSubDomains"
        Referrer-Policy "strict-origin-when-cross-origin"
        -Server
    }
}

Save: Esc, :wq, Enter

Step 3: Update CORS Configuration

Update the stack file to allow the frontend to call the new backend domain:

# Still on manager node
cd ~/stacks
vi maplepress-stack.yml

Find this line:

- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com

No change needed - The CORS config is for what origins can call the backend, not the backend's domain itself. The frontend (getmaplepress.com) will now call getmaplepress.net instead of getmaplepress.ca.

Step 4: Redeploy Backend Stack

# Remove old stack
docker stack rm maplepress
sleep 10

# Remove old config (contains old domain)
docker config rm maplepress_caddyfile

# Deploy with new domain
docker stack deploy -c maplepress-stack.yml maplepress

# Watch services come up
docker service ps maplepress_backend
docker service ps maplepress_backend-caddy

Step 5: Verify SSL Certificate

Caddy will automatically obtain SSL certificates for the new domain:

# Watch Caddy logs for certificate acquisition
docker service logs -f maplepress_backend-caddy

# You should see logs like:
# "certificate obtained successfully"
# "serving https://getmaplepress.net"

Test from local machine:

# Test new domain with HTTPS
curl -I https://getmaplepress.net/health
# Should return: HTTP/2 200

# Verify SSL certificate
curl -vI https://getmaplepress.net/health 2>&1 | grep "subject:"
# Should show: subject: CN=getmaplepress.net

# Test CORS
curl -v -H "Origin: https://getmaplepress.com" https://getmaplepress.net/health 2>&1 | grep "access-control-allow-origin"
# Should show: access-control-allow-origin: https://getmaplepress.com

Step 6: Update Frontend to Use New Backend Domain

On your local machine:

cd ~/go/src/codeberg.org/mapleopentech/monorepo/web/maplepress-frontend

# Update production environment file
vi .env.production

Change:

# OLD
VITE_API_BASE_URL=https://getmaplepress.ca

# NEW
VITE_API_BASE_URL=https://getmaplepress.net

Rebuild and redeploy frontend:

# Build with new backend URL
npm run build

# Verify the new URL is in the build
grep -r "getmaplepress.net" dist/assets/*.js | head -2
# Should show: getmaplepress.net

# SSH to worker-7 and update the frontend build
ssh dockeradmin@<worker-7-public-ip>
cd /var/www/monorepo/web/maplepress-frontend

# Pull latest code
git pull origin main

# Rebuild
npm run build

# Verify symlink
ls -la /var/www/maplepress-frontend
# Should point to: /var/www/monorepo/web/maplepress-frontend/dist

exit

Step 7: Test End-to-End

# Visit frontend in browser
open https://getmaplepress.com

# Open DevTools (F12) → Network tab
# Verify API calls now go to: https://getmaplepress.net
# Verify status: 200 (not 0 or CORS errors)

Step 8: (Optional) Keep Old Domain Working

If you want both domains to work temporarily:

# Edit Caddyfile to include BOTH domains
vi ~/stacks/caddy-config/Caddyfile

# Support both old and new domains
getmaplepress.ca www.getmaplepress.ca, getmaplepress.net www.getmaplepress.net {
    reverse_proxy maplepress-backend:8000 {
        # ... same config ...
    }
}

Then redeploy as in Step 4.

Rollback Procedure

If something goes wrong:

# 1. Restore old Caddyfile
cd ~/stacks/caddy-config
cp Caddyfile.backup.YYYYMMDD Caddyfile

# 2. Redeploy
cd ~/stacks
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress

# 3. Restore frontend .env.production
cd ~/go/src/codeberg.org/mapleopentech/monorepo/web/maplepress-frontend
# Change back to: VITE_API_BASE_URL=https://getmaplepress.ca
# Rebuild and redeploy

✅ Backend domain change complete!

Operation 2: Change Frontend Domain

Scenario: Changing frontend domain from getmaplepress.com → getmaplepress.app

Impact:

✅ Frontend becomes available at new domain
❌ Old domain stops working
⚠️ Backend CORS needs update to allow new frontend domain
⚠️ SSL certificate automatically obtained for new domain
⚠️ Downtime: ~2-5 minutes during redeployment

Step 1: DNS Configuration

Point the new domain to worker-7:

Type: A Record
Name: getmaplepress.app
Value: <worker-7-public-ip>
TTL: 300

Type: A Record
Name: www.getmaplepress.app
Value: <worker-7-public-ip>
TTL: 300

Test DNS propagation:

dig getmaplepress.app +short
# Should show: <worker-7-public-ip>

nslookup getmaplepress.app

Step 2: Update Frontend Caddyfile

On manager node:

ssh dockeradmin@<manager-public-ip>
cd ~/stacks/maplepress-frontend-caddy-config

# Backup
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)

# Edit
vi Caddyfile

Change this:

# OLD DOMAIN
getmaplepress.com www.getmaplepress.com {
    root * /var/www/maplepress-frontend
    # ... config ...
}

To this:

# NEW DOMAIN
getmaplepress.app www.getmaplepress.app {
    root * /var/www/maplepress-frontend
    file_server
    try_files {path} /index.html
    encode gzip

    log {
        output stdout
        format json
        level INFO
    }

    header {
        X-Frame-Options "SAMEORIGIN"
        X-Content-Type-Options "nosniff"
        X-XSS-Protection "1; mode=block"
        Strict-Transport-Security "max-age=31536000; includeSubDomains"
        Referrer-Policy "strict-origin-when-cross-origin"
        -Server
    }

    @static {
        path *.js *.css *.png *.jpg *.jpeg *.gif *.svg *.woff *.woff2 *.ttf *.eot *.ico
    }
    header @static Cache-Control "public, max-age=31536000, immutable"
}

Save: Esc, :wq, Enter

Step 3: Update Backend CORS Configuration

CRITICAL: The backend needs to allow the new frontend domain:

cd ~/stacks
vi maplepress-stack.yml

Find this line:

- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com

Change to:

- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.app,https://www.getmaplepress.app

If you want to support BOTH old and new domains temporarily:

- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com,https://getmaplepress.app,https://www.getmaplepress.app

Step 4: Redeploy Backend (for CORS update)

# Backend CORS config changed, must redeploy
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress

# Verify backend running
docker service ps maplepress_backend

Step 5: Redeploy Frontend

# Remove frontend stack
docker stack rm maplepress-frontend
sleep 10
docker config rm maplepress-frontend_caddyfile

# Deploy with new domain
docker stack deploy -c maplepress-frontend-stack.yml maplepress-frontend

# Watch it come up
docker service ps maplepress-frontend_caddy

Step 6: Verify SSL Certificate

Test from local machine:

# Test new frontend domain
curl -I https://getmaplepress.app
# Should return: HTTP/2 200

# Verify SSL certificate
curl -vI https://getmaplepress.app 2>&1 | grep "subject:"
# Should show: subject: CN=getmaplepress.app

Step 7: Test CORS from New Frontend

# Visit new frontend in browser
open https://getmaplepress.app

# Open DevTools (F12)
# Network tab: Verify API calls succeed
# Console tab: Should be NO CORS errors

Step 8: Verify Backend Accepts New Origin

# Test CORS from backend perspective
curl -v -H "Origin: https://getmaplepress.app" https://getmaplepress.ca/health 2>&1 | grep "access-control-allow-origin"
# Should show: access-control-allow-origin: https://getmaplepress.app

Rollback Procedure

# 1. Restore old frontend Caddyfile
cd ~/stacks/maplepress-frontend-caddy-config
cp Caddyfile.backup.YYYYMMDD Caddyfile

# 2. Restore old backend CORS config
cd ~/stacks
vi maplepress-stack.yml
# Change back to: https://getmaplepress.com,https://www.getmaplepress.com

# 3. Redeploy both
docker stack rm maplepress
docker stack rm maplepress-frontend
sleep 10
docker config rm maplepress_caddyfile
docker config rm maplepress-frontend_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress
docker stack deploy -c maplepress-frontend-stack.yml maplepress-frontend

✅ Frontend domain change complete!

Operation 3: Change Both Domains at Once

Scenario: Changing both domains simultaneously:

Backend: getmaplepress.ca → api.maplepress.io
Frontend: getmaplepress.com → app.maplepress.io

Benefits:

Single maintenance window
Coordinated cutover
Clean brand migration

Downtime: ~5-10 minutes

Complete Process

# ==============================================================================
# STEP 1: DNS Configuration (Do this first, wait for propagation)
# ==============================================================================

# Backend DNS:
# A Record: api.maplepress.io → <worker-6-public-ip>
# A Record: www.api.maplepress.io → <worker-6-public-ip>

# Frontend DNS:
# A Record: app.maplepress.io → <worker-7-public-ip>
# A Record: www.app.maplepress.io → <worker-7-public-ip>

# Test DNS
dig api.maplepress.io +short  # Should show worker-6 IP
dig app.maplepress.io +short  # Should show worker-7 IP

# ==============================================================================
# STEP 2: Update Backend Caddyfile
# ==============================================================================
ssh dockeradmin@<manager-public-ip>
cd ~/stacks/caddy-config
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)
vi Caddyfile

# Change domain from getmaplepress.ca to api.maplepress.io
# (Keep all other config the same)

# ==============================================================================
# STEP 3: Update Frontend Caddyfile
# ==============================================================================
cd ~/stacks/maplepress-frontend-caddy-config
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)
vi Caddyfile

# Change domain from getmaplepress.com to app.maplepress.io
# (Keep all other config the same)

# ==============================================================================
# STEP 4: Update Backend CORS for New Frontend Domain
# ==============================================================================
cd ~/stacks
vi maplepress-stack.yml

# Change:
# - SECURITY_CORS_ALLOWED_ORIGINS=https://app.maplepress.io,https://www.app.maplepress.io

# ==============================================================================
# STEP 5: Update Frontend .env.production for New Backend
# ==============================================================================
ssh dockeradmin@<worker-7-public-ip>
cd /var/www/monorepo/web/maplepress-frontend
vi .env.production

# Change:
# VITE_API_BASE_URL=https://api.maplepress.io

# Rebuild
npm run build

# Verify new URL in build
grep -r "api.maplepress.io" dist/assets/*.js | head -2

exit

# ==============================================================================
# STEP 6: Coordinated Deployment (Back on Manager)
# ==============================================================================
ssh dockeradmin@<manager-public-ip>
cd ~/stacks

# Remove both stacks
docker stack rm maplepress
docker stack rm maplepress-frontend
sleep 10

# Remove configs
docker config rm maplepress_caddyfile
docker config rm maplepress-frontend_caddyfile

# Deploy both stacks
docker stack deploy -c maplepress-stack.yml maplepress
docker stack deploy -c maplepress-frontend-stack.yml maplepress-frontend

# ==============================================================================
# STEP 7: Verify Both Services
# ==============================================================================
docker service ls | grep maplepress
# Should show 3 services all 1/1:
#   maplepress_backend
#   maplepress_backend-caddy
#   maplepress-frontend_caddy

# ==============================================================================
# STEP 8: Test End-to-End (Local Machine)
# ==============================================================================
# Test backend
curl -I https://api.maplepress.io/health
# Should return: HTTP/2 200

# Test frontend
curl -I https://app.maplepress.io
# Should return: HTTP/2 200

# Test CORS
curl -v -H "Origin: https://app.maplepress.io" https://api.maplepress.io/health 2>&1 | grep "access-control"
# Should show: access-control-allow-origin: https://app.maplepress.io

# Test in browser
open https://app.maplepress.io
# DevTools → Network: Verify calls to api.maplepress.io succeed

✅ Both domain changes complete!

Operation 4: Force SSL Certificate Renewal

Scenario: You need to manually renew SSL certificates (rarely needed - Caddy auto-renews)

When You Might Need This

Testing certificate renewal process
Certificate was revoked
Manual intervention required after failed auto-renewal

Backend Certificate Renewal

# SSH to worker-6
ssh dockeradmin@<worker-6-public-ip>

# Get Caddy container ID
docker ps | grep maplepress_backend-caddy

# Access Caddy container
docker exec -it <container-id> sh

# Inside container - force certificate renewal
caddy reload --config /etc/caddy/Caddyfile --force

# Or restart Caddy to trigger renewal
exit

# Back on worker-6
docker service update --force maplepress_backend-caddy

# Watch logs for certificate acquisition
docker service logs -f maplepress_backend-caddy | grep -i certificate

Frontend Certificate Renewal

# SSH to worker-7
ssh dockeradmin@<worker-7-public-ip>

# Get Caddy container ID
docker ps | grep maplepress-frontend

# Force reload
docker exec <container-id> caddy reload --config /etc/caddy/Caddyfile --force

# Or force restart
exit
docker service update --force maplepress-frontend_caddy

# Watch logs
docker service logs -f maplepress-frontend_caddy | grep -i certificate

Verify New Certificate

# From local machine
openssl s_client -connect getmaplepress.ca:443 -servername getmaplepress.ca < /dev/null 2>/dev/null | openssl x509 -noout -dates

# Should show:
# notBefore=Nov  5 12:00:00 2025 GMT
# notAfter=Feb   3 12:00:00 2026 GMT

Operation 5: Scale Backend Horizontally

Scenario: Your backend needs to handle more traffic - add more replicas

Considerations

Each replica needs database connections
Cassandra can handle the load (QUORUM with 3 nodes)
Redis connections are pooled
Stateless design allows easy horizontal scaling

Scale to 3 Replicas

# On manager node
cd ~/stacks
vi maplepress-stack.yml

# Find backend service, change replicas
# FROM:
#   deploy:
#     replicas: 1

# TO:
#   deploy:
#     replicas: 3

# Redeploy
docker stack deploy -c maplepress-stack.yml maplepress

# Watch replicas come up
watch docker service ps maplepress_backend
# Press Ctrl+C when all show Running

# Verify all healthy
docker service ps maplepress_backend --filter "desired-state=running"
# Should show 3 replicas

Load Balancing

Caddy automatically load balances between replicas:

# Test load balancing
for i in {1..10}; do
  curl -s https://getmaplepress.ca/health
  sleep 1
done

# Check which replicas handled requests
docker service logs maplepress_backend | grep "GET /health" | tail -20
# You should see different container IDs handling requests

Scale Back Down

# Edit stack file
vi ~/stacks/maplepress-stack.yml

# Change back to replicas: 1
# Redeploy
docker stack deploy -c maplepress-stack.yml maplepress

# Verify
docker service ps maplepress_backend
# Should show only 1 replica running, others Shutdown

Quick Reference: Domain Change Checklist

Backend Domain Change

Update DNS A records (point new domain to worker-6)
Wait for DNS propagation (5-60 minutes)
Backup Caddyfile: cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)
Update backend Caddyfile with new domain
Redeploy backend stack
Verify SSL certificate obtained for new domain
Update frontend .env.production with new backend URL
Rebuild and redeploy frontend
Test CORS end-to-end

Frontend Domain Change

Update DNS A records (point new domain to worker-7)
Wait for DNS propagation
Backup frontend Caddyfile
Update frontend Caddyfile with new domain
Update backend CORS in maplepress-stack.yml
Redeploy backend (for CORS)
Redeploy frontend stack
Verify SSL certificate
Test in browser (no CORS errors)

Troubleshooting Domain Changes

Problem: SSL Certificate Not Obtained

Symptom: After domain change, HTTPS doesn't work

# Check Caddy logs
docker service logs maplepress_backend-caddy --tail 100 | grep -i "acme\|certificate"

# Common issues:
# 1. DNS not propagated - wait longer
# 2. Port 80 not accessible - check firewall
# 3. Let's Encrypt rate limit - wait 1 hour

Fix:

# Verify DNS resolves
dig <new-domain> +short
# Must show correct worker IP

# Verify port 80 accessible
curl http://<new-domain>
# Should redirect to HTTPS

# If rate limited, wait and retry
# Let's Encrypt limit: 5 certificates per domain per week

Problem: CORS Errors After Domain Change

Symptom: Frontend shows CORS errors in browser console

Cause: Forgot to update backend CORS configuration

Fix:

# Check backend CORS config
cat ~/stacks/maplepress-stack.yml | grep CORS
# Should include NEW frontend domain

# Update if needed
vi ~/stacks/maplepress-stack.yml
# Add new frontend domain to SECURITY_CORS_ALLOWED_ORIGINS

# Redeploy backend
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress

# Test CORS
curl -v -H "Origin: https://<new-frontend-domain>" https://<backend-domain>/health 2>&1 | grep "access-control"

Problem: Old Domain Still Works

Symptom: Both old and new domains work

Cause: Caddyfile includes both domains

Expected Behavior: This is fine during migration - you can support both

To Remove Old Domain:

# Edit Caddyfile and remove old domain
vi ~/stacks/caddy-config/Caddyfile
# Remove old domain from the domain list

# Redeploy
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress

Last Updated: November 2025 Maintained By: Infrastructure Team

21 KiB Raw Blame History

Extra Operations and Domain Changes

Overview

Table of Contents

Operation 1: Change Backend Domain

Step 1: DNS Configuration

Step 2: Update Backend Caddyfile

Step 3: Update CORS Configuration

Step 4: Redeploy Backend Stack

Step 5: Verify SSL Certificate

Step 6: Update Frontend to Use New Backend Domain

Step 7: Test End-to-End

Step 8: (Optional) Keep Old Domain Working

Rollback Procedure

Operation 2: Change Frontend Domain

Step 1: DNS Configuration

Step 2: Update Frontend Caddyfile

Step 3: Update Backend CORS Configuration

Step 4: Redeploy Backend (for CORS update)

Step 5: Redeploy Frontend

Step 6: Verify SSL Certificate

Step 7: Test CORS from New Frontend

Step 8: Verify Backend Accepts New Origin

Rollback Procedure

Operation 3: Change Both Domains at Once

Complete Process

Operation 4: Force SSL Certificate Renewal

When You Might Need This

Backend Certificate Renewal

Frontend Certificate Renewal

Verify New Certificate

Operation 5: Scale Backend Horizontally

Considerations

Scale to 3 Replicas

Load Balancing

Scale Back Down

Quick Reference: Domain Change Checklist

Backend Domain Change

Frontend Domain Change

Troubleshooting Domain Changes

Problem: SSL Certificate Not Obtained

Problem: CORS Errors After Domain Change

Problem: Old Domain Still Works

21 KiB

Raw Blame History