Initial commit: Open sourcing all of the Maple Open Technologies code.

This commit is contained in:
Bartlomiej Mika 2025-12-02 14:33:08 -05:00
commit 755d54a99d
2010 changed files with 448675 additions and 0 deletions

View file

@ -0,0 +1,898 @@
# Extra Operations and Domain Changes
**Audience**: DevOps Engineers, Infrastructure Team
**Time to Complete**: Varies by operation
**Prerequisites**: Completed guides 01-07 (full MaplePress deployment)
---
## Overview
This guide covers additional operations and changes that you might need to perform on your production infrastructure:
1. **Domain Changes**
- Changing backend domain (e.g., `getmaplepress.ca``getmaplepress.net`)
- Changing frontend domain (e.g., `getmaplepress.com``getmaplepress.app`)
2. **SSL Certificate Management**
3. **Scaling Operations**
4. **Backup and Recovery**
---
## Table of Contents
1. [Change Backend Domain](#operation-1-change-backend-domain)
2. [Change Frontend Domain](#operation-2-change-frontend-domain)
3. [Change Both Domains](#operation-3-change-both-domains-at-once)
4. [Force SSL Certificate Renewal](#operation-4-force-ssl-certificate-renewal)
5. [Scale Backend Horizontally](#operation-5-scale-backend-horizontally)
---
## Operation 1: Change Backend Domain
**Scenario:** Changing backend API domain from `getmaplepress.ca``getmaplepress.net`
**Impact:**
- ✅ Backend becomes available at new domain
- ❌ Old domain stops working
- ⚠️ Frontend needs CORS update to allow new backend domain
- ⚠️ SSL certificate automatically obtained for new domain
- ⚠️ Downtime: ~2-5 minutes during redeployment
### Step 1: DNS Configuration
**First, point the new domain to worker-6:**
1. Log into your DNS provider (DigitalOcean, Cloudflare, etc.)
2. Create DNS A records for new domain:
```
Type: A Record
Name: getmaplepress.net
Value: <worker-6-public-ip>
TTL: 300 (5 minutes)
Type: A Record
Name: www.getmaplepress.net
Value: <worker-6-public-ip>
TTL: 300
```
3. Wait for DNS propagation (5-60 minutes):
```bash
# Test DNS from your local machine
dig getmaplepress.net +short
# Should show: <worker-6-public-ip>
dig www.getmaplepress.net +short
# Should show: <worker-6-public-ip>
# Alternative test
nslookup getmaplepress.net
```
### Step 2: Update Backend Caddyfile
**On manager node:**
```bash
ssh dockeradmin@<manager-public-ip>
cd ~/stacks/caddy-config
# Backup old Caddyfile
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)
# Edit Caddyfile
vi Caddyfile
```
**Change this:**
```caddy
# OLD DOMAIN
getmaplepress.ca www.getmaplepress.ca {
reverse_proxy maplepress-backend:8000 {
# ... config ...
}
}
```
**To this:**
```caddy
# NEW DOMAIN
getmaplepress.net www.getmaplepress.net {
reverse_proxy maplepress-backend:8000 {
header_up X-Real-IP {remote_host}
header_up X-Forwarded-For {remote_host}
header_up X-Forwarded-Proto {scheme}
header_up X-Forwarded-Host {host}
# IMPORTANT: Preserve Origin header for CORS
header_up Origin {http.request.header.Origin}
}
log {
output stdout
format json
level INFO
}
header {
X-Frame-Options "SAMEORIGIN"
X-Content-Type-Options "nosniff"
X-XSS-Protection "1; mode=block"
Strict-Transport-Security "max-age=31536000; includeSubDomains"
Referrer-Policy "strict-origin-when-cross-origin"
-Server
}
}
```
Save: `Esc`, `:wq`, `Enter`
### Step 3: Update CORS Configuration
**Update the stack file to allow the frontend to call the new backend domain:**
```bash
# Still on manager node
cd ~/stacks
vi maplepress-stack.yml
```
**Find this line:**
```yaml
- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com
```
**No change needed** - The CORS config is for what origins can call the backend, not the backend's domain itself. The frontend (`getmaplepress.com`) will now call `getmaplepress.net` instead of `getmaplepress.ca`.
### Step 4: Redeploy Backend Stack
```bash
# Remove old stack
docker stack rm maplepress
sleep 10
# Remove old config (contains old domain)
docker config rm maplepress_caddyfile
# Deploy with new domain
docker stack deploy -c maplepress-stack.yml maplepress
# Watch services come up
docker service ps maplepress_backend
docker service ps maplepress_backend-caddy
```
### Step 5: Verify SSL Certificate
**Caddy will automatically obtain SSL certificates for the new domain:**
```bash
# Watch Caddy logs for certificate acquisition
docker service logs -f maplepress_backend-caddy
# You should see logs like:
# "certificate obtained successfully"
# "serving https://getmaplepress.net"
```
**Test from local machine:**
```bash
# Test new domain with HTTPS
curl -I https://getmaplepress.net/health
# Should return: HTTP/2 200
# Verify SSL certificate
curl -vI https://getmaplepress.net/health 2>&1 | grep "subject:"
# Should show: subject: CN=getmaplepress.net
# Test CORS
curl -v -H "Origin: https://getmaplepress.com" https://getmaplepress.net/health 2>&1 | grep "access-control-allow-origin"
# Should show: access-control-allow-origin: https://getmaplepress.com
```
### Step 6: Update Frontend to Use New Backend Domain
**On your local machine:**
```bash
cd ~/go/src/codeberg.org/mapleopentech/monorepo/web/maplepress-frontend
# Update production environment file
vi .env.production
```
**Change:**
```bash
# OLD
VITE_API_BASE_URL=https://getmaplepress.ca
# NEW
VITE_API_BASE_URL=https://getmaplepress.net
```
**Rebuild and redeploy frontend:**
```bash
# Build with new backend URL
npm run build
# Verify the new URL is in the build
grep -r "getmaplepress.net" dist/assets/*.js | head -2
# Should show: getmaplepress.net
# SSH to worker-7 and update the frontend build
ssh dockeradmin@<worker-7-public-ip>
cd /var/www/monorepo/web/maplepress-frontend
# Pull latest code
git pull origin main
# Rebuild
npm run build
# Verify symlink
ls -la /var/www/maplepress-frontend
# Should point to: /var/www/monorepo/web/maplepress-frontend/dist
exit
```
### Step 7: Test End-to-End
```bash
# Visit frontend in browser
open https://getmaplepress.com
# Open DevTools (F12) → Network tab
# Verify API calls now go to: https://getmaplepress.net
# Verify status: 200 (not 0 or CORS errors)
```
### Step 8: (Optional) Keep Old Domain Working
If you want both domains to work temporarily:
```bash
# Edit Caddyfile to include BOTH domains
vi ~/stacks/caddy-config/Caddyfile
```
```caddy
# Support both old and new domains
getmaplepress.ca www.getmaplepress.ca, getmaplepress.net www.getmaplepress.net {
reverse_proxy maplepress-backend:8000 {
# ... same config ...
}
}
```
Then redeploy as in Step 4.
### Rollback Procedure
If something goes wrong:
```bash
# 1. Restore old Caddyfile
cd ~/stacks/caddy-config
cp Caddyfile.backup.YYYYMMDD Caddyfile
# 2. Redeploy
cd ~/stacks
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress
# 3. Restore frontend .env.production
cd ~/go/src/codeberg.org/mapleopentech/monorepo/web/maplepress-frontend
# Change back to: VITE_API_BASE_URL=https://getmaplepress.ca
# Rebuild and redeploy
```
**✅ Backend domain change complete!**
---
## Operation 2: Change Frontend Domain
**Scenario:** Changing frontend domain from `getmaplepress.com``getmaplepress.app`
**Impact:**
- ✅ Frontend becomes available at new domain
- ❌ Old domain stops working
- ⚠️ Backend CORS needs update to allow new frontend domain
- ⚠️ SSL certificate automatically obtained for new domain
- ⚠️ Downtime: ~2-5 minutes during redeployment
### Step 1: DNS Configuration
**Point the new domain to worker-7:**
```
Type: A Record
Name: getmaplepress.app
Value: <worker-7-public-ip>
TTL: 300
Type: A Record
Name: www.getmaplepress.app
Value: <worker-7-public-ip>
TTL: 300
```
**Test DNS propagation:**
```bash
dig getmaplepress.app +short
# Should show: <worker-7-public-ip>
nslookup getmaplepress.app
```
### Step 2: Update Frontend Caddyfile
**On manager node:**
```bash
ssh dockeradmin@<manager-public-ip>
cd ~/stacks/maplepress-frontend-caddy-config
# Backup
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)
# Edit
vi Caddyfile
```
**Change this:**
```caddy
# OLD DOMAIN
getmaplepress.com www.getmaplepress.com {
root * /var/www/maplepress-frontend
# ... config ...
}
```
**To this:**
```caddy
# NEW DOMAIN
getmaplepress.app www.getmaplepress.app {
root * /var/www/maplepress-frontend
file_server
try_files {path} /index.html
encode gzip
log {
output stdout
format json
level INFO
}
header {
X-Frame-Options "SAMEORIGIN"
X-Content-Type-Options "nosniff"
X-XSS-Protection "1; mode=block"
Strict-Transport-Security "max-age=31536000; includeSubDomains"
Referrer-Policy "strict-origin-when-cross-origin"
-Server
}
@static {
path *.js *.css *.png *.jpg *.jpeg *.gif *.svg *.woff *.woff2 *.ttf *.eot *.ico
}
header @static Cache-Control "public, max-age=31536000, immutable"
}
```
Save: `Esc`, `:wq`, `Enter`
### Step 3: Update Backend CORS Configuration
**CRITICAL:** The backend needs to allow the new frontend domain:
```bash
cd ~/stacks
vi maplepress-stack.yml
```
**Find this line:**
```yaml
- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com
```
**Change to:**
```yaml
- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.app,https://www.getmaplepress.app
```
**If you want to support BOTH old and new domains temporarily:**
```yaml
- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com,https://getmaplepress.app,https://www.getmaplepress.app
```
### Step 4: Redeploy Backend (for CORS update)
```bash
# Backend CORS config changed, must redeploy
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress
# Verify backend running
docker service ps maplepress_backend
```
### Step 5: Redeploy Frontend
```bash
# Remove frontend stack
docker stack rm maplepress-frontend
sleep 10
docker config rm maplepress-frontend_caddyfile
# Deploy with new domain
docker stack deploy -c maplepress-frontend-stack.yml maplepress-frontend
# Watch it come up
docker service ps maplepress-frontend_caddy
```
### Step 6: Verify SSL Certificate
**Test from local machine:**
```bash
# Test new frontend domain
curl -I https://getmaplepress.app
# Should return: HTTP/2 200
# Verify SSL certificate
curl -vI https://getmaplepress.app 2>&1 | grep "subject:"
# Should show: subject: CN=getmaplepress.app
```
### Step 7: Test CORS from New Frontend
```bash
# Visit new frontend in browser
open https://getmaplepress.app
# Open DevTools (F12)
# Network tab: Verify API calls succeed
# Console tab: Should be NO CORS errors
```
### Step 8: Verify Backend Accepts New Origin
```bash
# Test CORS from backend perspective
curl -v -H "Origin: https://getmaplepress.app" https://getmaplepress.ca/health 2>&1 | grep "access-control-allow-origin"
# Should show: access-control-allow-origin: https://getmaplepress.app
```
### Rollback Procedure
```bash
# 1. Restore old frontend Caddyfile
cd ~/stacks/maplepress-frontend-caddy-config
cp Caddyfile.backup.YYYYMMDD Caddyfile
# 2. Restore old backend CORS config
cd ~/stacks
vi maplepress-stack.yml
# Change back to: https://getmaplepress.com,https://www.getmaplepress.com
# 3. Redeploy both
docker stack rm maplepress
docker stack rm maplepress-frontend
sleep 10
docker config rm maplepress_caddyfile
docker config rm maplepress-frontend_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress
docker stack deploy -c maplepress-frontend-stack.yml maplepress-frontend
```
**✅ Frontend domain change complete!**
---
## Operation 3: Change Both Domains at Once
**Scenario:** Changing both domains simultaneously:
- Backend: `getmaplepress.ca``api.maplepress.io`
- Frontend: `getmaplepress.com``app.maplepress.io`
**Benefits:**
- Single maintenance window
- Coordinated cutover
- Clean brand migration
**Downtime:** ~5-10 minutes
### Complete Process
```bash
# ==============================================================================
# STEP 1: DNS Configuration (Do this first, wait for propagation)
# ==============================================================================
# Backend DNS:
# A Record: api.maplepress.io → <worker-6-public-ip>
# A Record: www.api.maplepress.io → <worker-6-public-ip>
# Frontend DNS:
# A Record: app.maplepress.io → <worker-7-public-ip>
# A Record: www.app.maplepress.io → <worker-7-public-ip>
# Test DNS
dig api.maplepress.io +short # Should show worker-6 IP
dig app.maplepress.io +short # Should show worker-7 IP
# ==============================================================================
# STEP 2: Update Backend Caddyfile
# ==============================================================================
ssh dockeradmin@<manager-public-ip>
cd ~/stacks/caddy-config
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)
vi Caddyfile
# Change domain from getmaplepress.ca to api.maplepress.io
# (Keep all other config the same)
# ==============================================================================
# STEP 3: Update Frontend Caddyfile
# ==============================================================================
cd ~/stacks/maplepress-frontend-caddy-config
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)
vi Caddyfile
# Change domain from getmaplepress.com to app.maplepress.io
# (Keep all other config the same)
# ==============================================================================
# STEP 4: Update Backend CORS for New Frontend Domain
# ==============================================================================
cd ~/stacks
vi maplepress-stack.yml
# Change:
# - SECURITY_CORS_ALLOWED_ORIGINS=https://app.maplepress.io,https://www.app.maplepress.io
# ==============================================================================
# STEP 5: Update Frontend .env.production for New Backend
# ==============================================================================
ssh dockeradmin@<worker-7-public-ip>
cd /var/www/monorepo/web/maplepress-frontend
vi .env.production
# Change:
# VITE_API_BASE_URL=https://api.maplepress.io
# Rebuild
npm run build
# Verify new URL in build
grep -r "api.maplepress.io" dist/assets/*.js | head -2
exit
# ==============================================================================
# STEP 6: Coordinated Deployment (Back on Manager)
# ==============================================================================
ssh dockeradmin@<manager-public-ip>
cd ~/stacks
# Remove both stacks
docker stack rm maplepress
docker stack rm maplepress-frontend
sleep 10
# Remove configs
docker config rm maplepress_caddyfile
docker config rm maplepress-frontend_caddyfile
# Deploy both stacks
docker stack deploy -c maplepress-stack.yml maplepress
docker stack deploy -c maplepress-frontend-stack.yml maplepress-frontend
# ==============================================================================
# STEP 7: Verify Both Services
# ==============================================================================
docker service ls | grep maplepress
# Should show 3 services all 1/1:
# maplepress_backend
# maplepress_backend-caddy
# maplepress-frontend_caddy
# ==============================================================================
# STEP 8: Test End-to-End (Local Machine)
# ==============================================================================
# Test backend
curl -I https://api.maplepress.io/health
# Should return: HTTP/2 200
# Test frontend
curl -I https://app.maplepress.io
# Should return: HTTP/2 200
# Test CORS
curl -v -H "Origin: https://app.maplepress.io" https://api.maplepress.io/health 2>&1 | grep "access-control"
# Should show: access-control-allow-origin: https://app.maplepress.io
# Test in browser
open https://app.maplepress.io
# DevTools → Network: Verify calls to api.maplepress.io succeed
```
**✅ Both domain changes complete!**
---
## Operation 4: Force SSL Certificate Renewal
**Scenario:** You need to manually renew SSL certificates (rarely needed - Caddy auto-renews)
### When You Might Need This
- Testing certificate renewal process
- Certificate was revoked
- Manual intervention required after failed auto-renewal
### Backend Certificate Renewal
```bash
# SSH to worker-6
ssh dockeradmin@<worker-6-public-ip>
# Get Caddy container ID
docker ps | grep maplepress_backend-caddy
# Access Caddy container
docker exec -it <container-id> sh
# Inside container - force certificate renewal
caddy reload --config /etc/caddy/Caddyfile --force
# Or restart Caddy to trigger renewal
exit
# Back on worker-6
docker service update --force maplepress_backend-caddy
# Watch logs for certificate acquisition
docker service logs -f maplepress_backend-caddy | grep -i certificate
```
### Frontend Certificate Renewal
```bash
# SSH to worker-7
ssh dockeradmin@<worker-7-public-ip>
# Get Caddy container ID
docker ps | grep maplepress-frontend
# Force reload
docker exec <container-id> caddy reload --config /etc/caddy/Caddyfile --force
# Or force restart
exit
docker service update --force maplepress-frontend_caddy
# Watch logs
docker service logs -f maplepress-frontend_caddy | grep -i certificate
```
### Verify New Certificate
```bash
# From local machine
openssl s_client -connect getmaplepress.ca:443 -servername getmaplepress.ca < /dev/null 2>/dev/null | openssl x509 -noout -dates
# Should show:
# notBefore=Nov 5 12:00:00 2025 GMT
# notAfter=Feb 3 12:00:00 2026 GMT
```
---
## Operation 5: Scale Backend Horizontally
**Scenario:** Your backend needs to handle more traffic - add more replicas
### Considerations
- Each replica needs database connections
- Cassandra can handle the load (QUORUM with 3 nodes)
- Redis connections are pooled
- Stateless design allows easy horizontal scaling
### Scale to 3 Replicas
```bash
# On manager node
cd ~/stacks
vi maplepress-stack.yml
# Find backend service, change replicas
# FROM:
# deploy:
# replicas: 1
# TO:
# deploy:
# replicas: 3
# Redeploy
docker stack deploy -c maplepress-stack.yml maplepress
# Watch replicas come up
watch docker service ps maplepress_backend
# Press Ctrl+C when all show Running
# Verify all healthy
docker service ps maplepress_backend --filter "desired-state=running"
# Should show 3 replicas
```
### Load Balancing
Caddy automatically load balances between replicas:
```bash
# Test load balancing
for i in {1..10}; do
curl -s https://getmaplepress.ca/health
sleep 1
done
# Check which replicas handled requests
docker service logs maplepress_backend | grep "GET /health" | tail -20
# You should see different container IDs handling requests
```
### Scale Back Down
```bash
# Edit stack file
vi ~/stacks/maplepress-stack.yml
# Change back to replicas: 1
# Redeploy
docker stack deploy -c maplepress-stack.yml maplepress
# Verify
docker service ps maplepress_backend
# Should show only 1 replica running, others Shutdown
```
---
## Quick Reference: Domain Change Checklist
### Backend Domain Change
- [ ] Update DNS A records (point new domain to worker-6)
- [ ] Wait for DNS propagation (5-60 minutes)
- [ ] Backup Caddyfile: `cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)`
- [ ] Update backend Caddyfile with new domain
- [ ] Redeploy backend stack
- [ ] Verify SSL certificate obtained for new domain
- [ ] Update frontend `.env.production` with new backend URL
- [ ] Rebuild and redeploy frontend
- [ ] Test CORS end-to-end
### Frontend Domain Change
- [ ] Update DNS A records (point new domain to worker-7)
- [ ] Wait for DNS propagation
- [ ] Backup frontend Caddyfile
- [ ] Update frontend Caddyfile with new domain
- [ ] **Update backend CORS** in `maplepress-stack.yml`
- [ ] Redeploy backend (for CORS)
- [ ] Redeploy frontend stack
- [ ] Verify SSL certificate
- [ ] Test in browser (no CORS errors)
---
## Troubleshooting Domain Changes
### Problem: SSL Certificate Not Obtained
**Symptom:** After domain change, HTTPS doesn't work
```bash
# Check Caddy logs
docker service logs maplepress_backend-caddy --tail 100 | grep -i "acme\|certificate"
# Common issues:
# 1. DNS not propagated - wait longer
# 2. Port 80 not accessible - check firewall
# 3. Let's Encrypt rate limit - wait 1 hour
```
**Fix:**
```bash
# Verify DNS resolves
dig <new-domain> +short
# Must show correct worker IP
# Verify port 80 accessible
curl http://<new-domain>
# Should redirect to HTTPS
# If rate limited, wait and retry
# Let's Encrypt limit: 5 certificates per domain per week
```
### Problem: CORS Errors After Domain Change
**Symptom:** Frontend shows CORS errors in browser console
**Cause:** Forgot to update backend CORS configuration
**Fix:**
```bash
# Check backend CORS config
cat ~/stacks/maplepress-stack.yml | grep CORS
# Should include NEW frontend domain
# Update if needed
vi ~/stacks/maplepress-stack.yml
# Add new frontend domain to SECURITY_CORS_ALLOWED_ORIGINS
# Redeploy backend
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress
# Test CORS
curl -v -H "Origin: https://<new-frontend-domain>" https://<backend-domain>/health 2>&1 | grep "access-control"
```
### Problem: Old Domain Still Works
**Symptom:** Both old and new domains work
**Cause:** Caddyfile includes both domains
**Expected Behavior:** This is fine during migration - you can support both
**To Remove Old Domain:**
```bash
# Edit Caddyfile and remove old domain
vi ~/stacks/caddy-config/Caddyfile
# Remove old domain from the domain list
# Redeploy
docker stack rm maplepress
sleep 10
docker config rm maplepress_caddyfile
docker stack deploy -c maplepress-stack.yml maplepress
```
---
**Last Updated**: November 2025
**Maintained By**: Infrastructure Team