Initial commit: Open sourcing all of the Maple Open Technologies code.
This commit is contained in:
commit
755d54a99d
2010 changed files with 448675 additions and 0 deletions
898
cloud/infrastructure/production/setup/99_extra.md
Normal file
898
cloud/infrastructure/production/setup/99_extra.md
Normal file
|
|
@ -0,0 +1,898 @@
|
|||
# Extra Operations and Domain Changes
|
||||
|
||||
**Audience**: DevOps Engineers, Infrastructure Team
|
||||
**Time to Complete**: Varies by operation
|
||||
**Prerequisites**: Completed guides 01-07 (full MaplePress deployment)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This guide covers additional operations and changes that you might need to perform on your production infrastructure:
|
||||
|
||||
1. **Domain Changes**
|
||||
- Changing backend domain (e.g., `getmaplepress.ca` → `getmaplepress.net`)
|
||||
- Changing frontend domain (e.g., `getmaplepress.com` → `getmaplepress.app`)
|
||||
2. **SSL Certificate Management**
|
||||
3. **Scaling Operations**
|
||||
4. **Backup and Recovery**
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Change Backend Domain](#operation-1-change-backend-domain)
|
||||
2. [Change Frontend Domain](#operation-2-change-frontend-domain)
|
||||
3. [Change Both Domains](#operation-3-change-both-domains-at-once)
|
||||
4. [Force SSL Certificate Renewal](#operation-4-force-ssl-certificate-renewal)
|
||||
5. [Scale Backend Horizontally](#operation-5-scale-backend-horizontally)
|
||||
|
||||
---
|
||||
|
||||
## Operation 1: Change Backend Domain
|
||||
|
||||
**Scenario:** Changing backend API domain from `getmaplepress.ca` → `getmaplepress.net`
|
||||
|
||||
**Impact:**
|
||||
- ✅ Backend becomes available at new domain
|
||||
- ❌ Old domain stops working
|
||||
- ⚠️ Frontend needs CORS update to allow new backend domain
|
||||
- ⚠️ SSL certificate automatically obtained for new domain
|
||||
- ⚠️ Downtime: ~2-5 minutes during redeployment
|
||||
|
||||
### Step 1: DNS Configuration
|
||||
|
||||
**First, point the new domain to worker-6:**
|
||||
|
||||
1. Log into your DNS provider (DigitalOcean, Cloudflare, etc.)
|
||||
2. Create DNS A records for new domain:
|
||||
|
||||
```
|
||||
Type: A Record
|
||||
Name: getmaplepress.net
|
||||
Value: <worker-6-public-ip>
|
||||
TTL: 300 (5 minutes)
|
||||
|
||||
Type: A Record
|
||||
Name: www.getmaplepress.net
|
||||
Value: <worker-6-public-ip>
|
||||
TTL: 300
|
||||
```
|
||||
|
||||
3. Wait for DNS propagation (5-60 minutes):
|
||||
|
||||
```bash
|
||||
# Test DNS from your local machine
|
||||
dig getmaplepress.net +short
|
||||
# Should show: <worker-6-public-ip>
|
||||
|
||||
dig www.getmaplepress.net +short
|
||||
# Should show: <worker-6-public-ip>
|
||||
|
||||
# Alternative test
|
||||
nslookup getmaplepress.net
|
||||
```
|
||||
|
||||
### Step 2: Update Backend Caddyfile
|
||||
|
||||
**On manager node:**
|
||||
|
||||
```bash
|
||||
ssh dockeradmin@<manager-public-ip>
|
||||
cd ~/stacks/caddy-config
|
||||
|
||||
# Backup old Caddyfile
|
||||
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)
|
||||
|
||||
# Edit Caddyfile
|
||||
vi Caddyfile
|
||||
```
|
||||
|
||||
**Change this:**
|
||||
|
||||
```caddy
|
||||
# OLD DOMAIN
|
||||
getmaplepress.ca www.getmaplepress.ca {
|
||||
reverse_proxy maplepress-backend:8000 {
|
||||
# ... config ...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**To this:**
|
||||
|
||||
```caddy
|
||||
# NEW DOMAIN
|
||||
getmaplepress.net www.getmaplepress.net {
|
||||
reverse_proxy maplepress-backend:8000 {
|
||||
header_up X-Real-IP {remote_host}
|
||||
header_up X-Forwarded-For {remote_host}
|
||||
header_up X-Forwarded-Proto {scheme}
|
||||
header_up X-Forwarded-Host {host}
|
||||
|
||||
# IMPORTANT: Preserve Origin header for CORS
|
||||
header_up Origin {http.request.header.Origin}
|
||||
}
|
||||
|
||||
log {
|
||||
output stdout
|
||||
format json
|
||||
level INFO
|
||||
}
|
||||
|
||||
header {
|
||||
X-Frame-Options "SAMEORIGIN"
|
||||
X-Content-Type-Options "nosniff"
|
||||
X-XSS-Protection "1; mode=block"
|
||||
Strict-Transport-Security "max-age=31536000; includeSubDomains"
|
||||
Referrer-Policy "strict-origin-when-cross-origin"
|
||||
-Server
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Save: `Esc`, `:wq`, `Enter`
|
||||
|
||||
### Step 3: Update CORS Configuration
|
||||
|
||||
**Update the stack file to allow the frontend to call the new backend domain:**
|
||||
|
||||
```bash
|
||||
# Still on manager node
|
||||
cd ~/stacks
|
||||
vi maplepress-stack.yml
|
||||
```
|
||||
|
||||
**Find this line:**
|
||||
|
||||
```yaml
|
||||
- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com
|
||||
```
|
||||
|
||||
**No change needed** - The CORS config is for what origins can call the backend, not the backend's domain itself. The frontend (`getmaplepress.com`) will now call `getmaplepress.net` instead of `getmaplepress.ca`.
|
||||
|
||||
### Step 4: Redeploy Backend Stack
|
||||
|
||||
```bash
|
||||
# Remove old stack
|
||||
docker stack rm maplepress
|
||||
sleep 10
|
||||
|
||||
# Remove old config (contains old domain)
|
||||
docker config rm maplepress_caddyfile
|
||||
|
||||
# Deploy with new domain
|
||||
docker stack deploy -c maplepress-stack.yml maplepress
|
||||
|
||||
# Watch services come up
|
||||
docker service ps maplepress_backend
|
||||
docker service ps maplepress_backend-caddy
|
||||
```
|
||||
|
||||
### Step 5: Verify SSL Certificate
|
||||
|
||||
**Caddy will automatically obtain SSL certificates for the new domain:**
|
||||
|
||||
```bash
|
||||
# Watch Caddy logs for certificate acquisition
|
||||
docker service logs -f maplepress_backend-caddy
|
||||
|
||||
# You should see logs like:
|
||||
# "certificate obtained successfully"
|
||||
# "serving https://getmaplepress.net"
|
||||
```
|
||||
|
||||
**Test from local machine:**
|
||||
|
||||
```bash
|
||||
# Test new domain with HTTPS
|
||||
curl -I https://getmaplepress.net/health
|
||||
# Should return: HTTP/2 200
|
||||
|
||||
# Verify SSL certificate
|
||||
curl -vI https://getmaplepress.net/health 2>&1 | grep "subject:"
|
||||
# Should show: subject: CN=getmaplepress.net
|
||||
|
||||
# Test CORS
|
||||
curl -v -H "Origin: https://getmaplepress.com" https://getmaplepress.net/health 2>&1 | grep "access-control-allow-origin"
|
||||
# Should show: access-control-allow-origin: https://getmaplepress.com
|
||||
```
|
||||
|
||||
### Step 6: Update Frontend to Use New Backend Domain
|
||||
|
||||
**On your local machine:**
|
||||
|
||||
```bash
|
||||
cd ~/go/src/codeberg.org/mapleopentech/monorepo/web/maplepress-frontend
|
||||
|
||||
# Update production environment file
|
||||
vi .env.production
|
||||
```
|
||||
|
||||
**Change:**
|
||||
|
||||
```bash
|
||||
# OLD
|
||||
VITE_API_BASE_URL=https://getmaplepress.ca
|
||||
|
||||
# NEW
|
||||
VITE_API_BASE_URL=https://getmaplepress.net
|
||||
```
|
||||
|
||||
**Rebuild and redeploy frontend:**
|
||||
|
||||
```bash
|
||||
# Build with new backend URL
|
||||
npm run build
|
||||
|
||||
# Verify the new URL is in the build
|
||||
grep -r "getmaplepress.net" dist/assets/*.js | head -2
|
||||
# Should show: getmaplepress.net
|
||||
|
||||
# SSH to worker-7 and update the frontend build
|
||||
ssh dockeradmin@<worker-7-public-ip>
|
||||
cd /var/www/monorepo/web/maplepress-frontend
|
||||
|
||||
# Pull latest code
|
||||
git pull origin main
|
||||
|
||||
# Rebuild
|
||||
npm run build
|
||||
|
||||
# Verify symlink
|
||||
ls -la /var/www/maplepress-frontend
|
||||
# Should point to: /var/www/monorepo/web/maplepress-frontend/dist
|
||||
|
||||
exit
|
||||
```
|
||||
|
||||
### Step 7: Test End-to-End
|
||||
|
||||
```bash
|
||||
# Visit frontend in browser
|
||||
open https://getmaplepress.com
|
||||
|
||||
# Open DevTools (F12) → Network tab
|
||||
# Verify API calls now go to: https://getmaplepress.net
|
||||
# Verify status: 200 (not 0 or CORS errors)
|
||||
```
|
||||
|
||||
### Step 8: (Optional) Keep Old Domain Working
|
||||
|
||||
If you want both domains to work temporarily:
|
||||
|
||||
```bash
|
||||
# Edit Caddyfile to include BOTH domains
|
||||
vi ~/stacks/caddy-config/Caddyfile
|
||||
```
|
||||
|
||||
```caddy
|
||||
# Support both old and new domains
|
||||
getmaplepress.ca www.getmaplepress.ca, getmaplepress.net www.getmaplepress.net {
|
||||
reverse_proxy maplepress-backend:8000 {
|
||||
# ... same config ...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then redeploy as in Step 4.
|
||||
|
||||
### Rollback Procedure
|
||||
|
||||
If something goes wrong:
|
||||
|
||||
```bash
|
||||
# 1. Restore old Caddyfile
|
||||
cd ~/stacks/caddy-config
|
||||
cp Caddyfile.backup.YYYYMMDD Caddyfile
|
||||
|
||||
# 2. Redeploy
|
||||
cd ~/stacks
|
||||
docker stack rm maplepress
|
||||
sleep 10
|
||||
docker config rm maplepress_caddyfile
|
||||
docker stack deploy -c maplepress-stack.yml maplepress
|
||||
|
||||
# 3. Restore frontend .env.production
|
||||
cd ~/go/src/codeberg.org/mapleopentech/monorepo/web/maplepress-frontend
|
||||
# Change back to: VITE_API_BASE_URL=https://getmaplepress.ca
|
||||
# Rebuild and redeploy
|
||||
```
|
||||
|
||||
**✅ Backend domain change complete!**
|
||||
|
||||
---
|
||||
|
||||
## Operation 2: Change Frontend Domain
|
||||
|
||||
**Scenario:** Changing frontend domain from `getmaplepress.com` → `getmaplepress.app`
|
||||
|
||||
**Impact:**
|
||||
- ✅ Frontend becomes available at new domain
|
||||
- ❌ Old domain stops working
|
||||
- ⚠️ Backend CORS needs update to allow new frontend domain
|
||||
- ⚠️ SSL certificate automatically obtained for new domain
|
||||
- ⚠️ Downtime: ~2-5 minutes during redeployment
|
||||
|
||||
### Step 1: DNS Configuration
|
||||
|
||||
**Point the new domain to worker-7:**
|
||||
|
||||
```
|
||||
Type: A Record
|
||||
Name: getmaplepress.app
|
||||
Value: <worker-7-public-ip>
|
||||
TTL: 300
|
||||
|
||||
Type: A Record
|
||||
Name: www.getmaplepress.app
|
||||
Value: <worker-7-public-ip>
|
||||
TTL: 300
|
||||
```
|
||||
|
||||
**Test DNS propagation:**
|
||||
|
||||
```bash
|
||||
dig getmaplepress.app +short
|
||||
# Should show: <worker-7-public-ip>
|
||||
|
||||
nslookup getmaplepress.app
|
||||
```
|
||||
|
||||
### Step 2: Update Frontend Caddyfile
|
||||
|
||||
**On manager node:**
|
||||
|
||||
```bash
|
||||
ssh dockeradmin@<manager-public-ip>
|
||||
cd ~/stacks/maplepress-frontend-caddy-config
|
||||
|
||||
# Backup
|
||||
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)
|
||||
|
||||
# Edit
|
||||
vi Caddyfile
|
||||
```
|
||||
|
||||
**Change this:**
|
||||
|
||||
```caddy
|
||||
# OLD DOMAIN
|
||||
getmaplepress.com www.getmaplepress.com {
|
||||
root * /var/www/maplepress-frontend
|
||||
# ... config ...
|
||||
}
|
||||
```
|
||||
|
||||
**To this:**
|
||||
|
||||
```caddy
|
||||
# NEW DOMAIN
|
||||
getmaplepress.app www.getmaplepress.app {
|
||||
root * /var/www/maplepress-frontend
|
||||
file_server
|
||||
try_files {path} /index.html
|
||||
encode gzip
|
||||
|
||||
log {
|
||||
output stdout
|
||||
format json
|
||||
level INFO
|
||||
}
|
||||
|
||||
header {
|
||||
X-Frame-Options "SAMEORIGIN"
|
||||
X-Content-Type-Options "nosniff"
|
||||
X-XSS-Protection "1; mode=block"
|
||||
Strict-Transport-Security "max-age=31536000; includeSubDomains"
|
||||
Referrer-Policy "strict-origin-when-cross-origin"
|
||||
-Server
|
||||
}
|
||||
|
||||
@static {
|
||||
path *.js *.css *.png *.jpg *.jpeg *.gif *.svg *.woff *.woff2 *.ttf *.eot *.ico
|
||||
}
|
||||
header @static Cache-Control "public, max-age=31536000, immutable"
|
||||
}
|
||||
```
|
||||
|
||||
Save: `Esc`, `:wq`, `Enter`
|
||||
|
||||
### Step 3: Update Backend CORS Configuration
|
||||
|
||||
**CRITICAL:** The backend needs to allow the new frontend domain:
|
||||
|
||||
```bash
|
||||
cd ~/stacks
|
||||
vi maplepress-stack.yml
|
||||
```
|
||||
|
||||
**Find this line:**
|
||||
|
||||
```yaml
|
||||
- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com
|
||||
```
|
||||
|
||||
**Change to:**
|
||||
|
||||
```yaml
|
||||
- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.app,https://www.getmaplepress.app
|
||||
```
|
||||
|
||||
**If you want to support BOTH old and new domains temporarily:**
|
||||
|
||||
```yaml
|
||||
- SECURITY_CORS_ALLOWED_ORIGINS=https://getmaplepress.com,https://www.getmaplepress.com,https://getmaplepress.app,https://www.getmaplepress.app
|
||||
```
|
||||
|
||||
### Step 4: Redeploy Backend (for CORS update)
|
||||
|
||||
```bash
|
||||
# Backend CORS config changed, must redeploy
|
||||
docker stack rm maplepress
|
||||
sleep 10
|
||||
docker config rm maplepress_caddyfile
|
||||
docker stack deploy -c maplepress-stack.yml maplepress
|
||||
|
||||
# Verify backend running
|
||||
docker service ps maplepress_backend
|
||||
```
|
||||
|
||||
### Step 5: Redeploy Frontend
|
||||
|
||||
```bash
|
||||
# Remove frontend stack
|
||||
docker stack rm maplepress-frontend
|
||||
sleep 10
|
||||
docker config rm maplepress-frontend_caddyfile
|
||||
|
||||
# Deploy with new domain
|
||||
docker stack deploy -c maplepress-frontend-stack.yml maplepress-frontend
|
||||
|
||||
# Watch it come up
|
||||
docker service ps maplepress-frontend_caddy
|
||||
```
|
||||
|
||||
### Step 6: Verify SSL Certificate
|
||||
|
||||
**Test from local machine:**
|
||||
|
||||
```bash
|
||||
# Test new frontend domain
|
||||
curl -I https://getmaplepress.app
|
||||
# Should return: HTTP/2 200
|
||||
|
||||
# Verify SSL certificate
|
||||
curl -vI https://getmaplepress.app 2>&1 | grep "subject:"
|
||||
# Should show: subject: CN=getmaplepress.app
|
||||
```
|
||||
|
||||
### Step 7: Test CORS from New Frontend
|
||||
|
||||
```bash
|
||||
# Visit new frontend in browser
|
||||
open https://getmaplepress.app
|
||||
|
||||
# Open DevTools (F12)
|
||||
# Network tab: Verify API calls succeed
|
||||
# Console tab: Should be NO CORS errors
|
||||
```
|
||||
|
||||
### Step 8: Verify Backend Accepts New Origin
|
||||
|
||||
```bash
|
||||
# Test CORS from backend perspective
|
||||
curl -v -H "Origin: https://getmaplepress.app" https://getmaplepress.ca/health 2>&1 | grep "access-control-allow-origin"
|
||||
# Should show: access-control-allow-origin: https://getmaplepress.app
|
||||
```
|
||||
|
||||
### Rollback Procedure
|
||||
|
||||
```bash
|
||||
# 1. Restore old frontend Caddyfile
|
||||
cd ~/stacks/maplepress-frontend-caddy-config
|
||||
cp Caddyfile.backup.YYYYMMDD Caddyfile
|
||||
|
||||
# 2. Restore old backend CORS config
|
||||
cd ~/stacks
|
||||
vi maplepress-stack.yml
|
||||
# Change back to: https://getmaplepress.com,https://www.getmaplepress.com
|
||||
|
||||
# 3. Redeploy both
|
||||
docker stack rm maplepress
|
||||
docker stack rm maplepress-frontend
|
||||
sleep 10
|
||||
docker config rm maplepress_caddyfile
|
||||
docker config rm maplepress-frontend_caddyfile
|
||||
docker stack deploy -c maplepress-stack.yml maplepress
|
||||
docker stack deploy -c maplepress-frontend-stack.yml maplepress-frontend
|
||||
```
|
||||
|
||||
**✅ Frontend domain change complete!**
|
||||
|
||||
---
|
||||
|
||||
## Operation 3: Change Both Domains at Once
|
||||
|
||||
**Scenario:** Changing both domains simultaneously:
|
||||
- Backend: `getmaplepress.ca` → `api.maplepress.io`
|
||||
- Frontend: `getmaplepress.com` → `app.maplepress.io`
|
||||
|
||||
**Benefits:**
|
||||
- Single maintenance window
|
||||
- Coordinated cutover
|
||||
- Clean brand migration
|
||||
|
||||
**Downtime:** ~5-10 minutes
|
||||
|
||||
### Complete Process
|
||||
|
||||
```bash
|
||||
# ==============================================================================
|
||||
# STEP 1: DNS Configuration (Do this first, wait for propagation)
|
||||
# ==============================================================================
|
||||
|
||||
# Backend DNS:
|
||||
# A Record: api.maplepress.io → <worker-6-public-ip>
|
||||
# A Record: www.api.maplepress.io → <worker-6-public-ip>
|
||||
|
||||
# Frontend DNS:
|
||||
# A Record: app.maplepress.io → <worker-7-public-ip>
|
||||
# A Record: www.app.maplepress.io → <worker-7-public-ip>
|
||||
|
||||
# Test DNS
|
||||
dig api.maplepress.io +short # Should show worker-6 IP
|
||||
dig app.maplepress.io +short # Should show worker-7 IP
|
||||
|
||||
# ==============================================================================
|
||||
# STEP 2: Update Backend Caddyfile
|
||||
# ==============================================================================
|
||||
ssh dockeradmin@<manager-public-ip>
|
||||
cd ~/stacks/caddy-config
|
||||
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)
|
||||
vi Caddyfile
|
||||
|
||||
# Change domain from getmaplepress.ca to api.maplepress.io
|
||||
# (Keep all other config the same)
|
||||
|
||||
# ==============================================================================
|
||||
# STEP 3: Update Frontend Caddyfile
|
||||
# ==============================================================================
|
||||
cd ~/stacks/maplepress-frontend-caddy-config
|
||||
cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)
|
||||
vi Caddyfile
|
||||
|
||||
# Change domain from getmaplepress.com to app.maplepress.io
|
||||
# (Keep all other config the same)
|
||||
|
||||
# ==============================================================================
|
||||
# STEP 4: Update Backend CORS for New Frontend Domain
|
||||
# ==============================================================================
|
||||
cd ~/stacks
|
||||
vi maplepress-stack.yml
|
||||
|
||||
# Change:
|
||||
# - SECURITY_CORS_ALLOWED_ORIGINS=https://app.maplepress.io,https://www.app.maplepress.io
|
||||
|
||||
# ==============================================================================
|
||||
# STEP 5: Update Frontend .env.production for New Backend
|
||||
# ==============================================================================
|
||||
ssh dockeradmin@<worker-7-public-ip>
|
||||
cd /var/www/monorepo/web/maplepress-frontend
|
||||
vi .env.production
|
||||
|
||||
# Change:
|
||||
# VITE_API_BASE_URL=https://api.maplepress.io
|
||||
|
||||
# Rebuild
|
||||
npm run build
|
||||
|
||||
# Verify new URL in build
|
||||
grep -r "api.maplepress.io" dist/assets/*.js | head -2
|
||||
|
||||
exit
|
||||
|
||||
# ==============================================================================
|
||||
# STEP 6: Coordinated Deployment (Back on Manager)
|
||||
# ==============================================================================
|
||||
ssh dockeradmin@<manager-public-ip>
|
||||
cd ~/stacks
|
||||
|
||||
# Remove both stacks
|
||||
docker stack rm maplepress
|
||||
docker stack rm maplepress-frontend
|
||||
sleep 10
|
||||
|
||||
# Remove configs
|
||||
docker config rm maplepress_caddyfile
|
||||
docker config rm maplepress-frontend_caddyfile
|
||||
|
||||
# Deploy both stacks
|
||||
docker stack deploy -c maplepress-stack.yml maplepress
|
||||
docker stack deploy -c maplepress-frontend-stack.yml maplepress-frontend
|
||||
|
||||
# ==============================================================================
|
||||
# STEP 7: Verify Both Services
|
||||
# ==============================================================================
|
||||
docker service ls | grep maplepress
|
||||
# Should show 3 services all 1/1:
|
||||
# maplepress_backend
|
||||
# maplepress_backend-caddy
|
||||
# maplepress-frontend_caddy
|
||||
|
||||
# ==============================================================================
|
||||
# STEP 8: Test End-to-End (Local Machine)
|
||||
# ==============================================================================
|
||||
# Test backend
|
||||
curl -I https://api.maplepress.io/health
|
||||
# Should return: HTTP/2 200
|
||||
|
||||
# Test frontend
|
||||
curl -I https://app.maplepress.io
|
||||
# Should return: HTTP/2 200
|
||||
|
||||
# Test CORS
|
||||
curl -v -H "Origin: https://app.maplepress.io" https://api.maplepress.io/health 2>&1 | grep "access-control"
|
||||
# Should show: access-control-allow-origin: https://app.maplepress.io
|
||||
|
||||
# Test in browser
|
||||
open https://app.maplepress.io
|
||||
# DevTools → Network: Verify calls to api.maplepress.io succeed
|
||||
```
|
||||
|
||||
**✅ Both domain changes complete!**
|
||||
|
||||
---
|
||||
|
||||
## Operation 4: Force SSL Certificate Renewal
|
||||
|
||||
**Scenario:** You need to manually renew SSL certificates (rarely needed - Caddy auto-renews)
|
||||
|
||||
### When You Might Need This
|
||||
|
||||
- Testing certificate renewal process
|
||||
- Certificate was revoked
|
||||
- Manual intervention required after failed auto-renewal
|
||||
|
||||
### Backend Certificate Renewal
|
||||
|
||||
```bash
|
||||
# SSH to worker-6
|
||||
ssh dockeradmin@<worker-6-public-ip>
|
||||
|
||||
# Get Caddy container ID
|
||||
docker ps | grep maplepress_backend-caddy
|
||||
|
||||
# Access Caddy container
|
||||
docker exec -it <container-id> sh
|
||||
|
||||
# Inside container - force certificate renewal
|
||||
caddy reload --config /etc/caddy/Caddyfile --force
|
||||
|
||||
# Or restart Caddy to trigger renewal
|
||||
exit
|
||||
|
||||
# Back on worker-6
|
||||
docker service update --force maplepress_backend-caddy
|
||||
|
||||
# Watch logs for certificate acquisition
|
||||
docker service logs -f maplepress_backend-caddy | grep -i certificate
|
||||
```
|
||||
|
||||
### Frontend Certificate Renewal
|
||||
|
||||
```bash
|
||||
# SSH to worker-7
|
||||
ssh dockeradmin@<worker-7-public-ip>
|
||||
|
||||
# Get Caddy container ID
|
||||
docker ps | grep maplepress-frontend
|
||||
|
||||
# Force reload
|
||||
docker exec <container-id> caddy reload --config /etc/caddy/Caddyfile --force
|
||||
|
||||
# Or force restart
|
||||
exit
|
||||
docker service update --force maplepress-frontend_caddy
|
||||
|
||||
# Watch logs
|
||||
docker service logs -f maplepress-frontend_caddy | grep -i certificate
|
||||
```
|
||||
|
||||
### Verify New Certificate
|
||||
|
||||
```bash
|
||||
# From local machine
|
||||
openssl s_client -connect getmaplepress.ca:443 -servername getmaplepress.ca < /dev/null 2>/dev/null | openssl x509 -noout -dates
|
||||
|
||||
# Should show:
|
||||
# notBefore=Nov 5 12:00:00 2025 GMT
|
||||
# notAfter=Feb 3 12:00:00 2026 GMT
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Operation 5: Scale Backend Horizontally
|
||||
|
||||
**Scenario:** Your backend needs to handle more traffic - add more replicas
|
||||
|
||||
### Considerations
|
||||
|
||||
- Each replica needs database connections
|
||||
- Cassandra can handle the load (QUORUM with 3 nodes)
|
||||
- Redis connections are pooled
|
||||
- Stateless design allows easy horizontal scaling
|
||||
|
||||
### Scale to 3 Replicas
|
||||
|
||||
```bash
|
||||
# On manager node
|
||||
cd ~/stacks
|
||||
vi maplepress-stack.yml
|
||||
|
||||
# Find backend service, change replicas
|
||||
# FROM:
|
||||
# deploy:
|
||||
# replicas: 1
|
||||
|
||||
# TO:
|
||||
# deploy:
|
||||
# replicas: 3
|
||||
|
||||
# Redeploy
|
||||
docker stack deploy -c maplepress-stack.yml maplepress
|
||||
|
||||
# Watch replicas come up
|
||||
watch docker service ps maplepress_backend
|
||||
# Press Ctrl+C when all show Running
|
||||
|
||||
# Verify all healthy
|
||||
docker service ps maplepress_backend --filter "desired-state=running"
|
||||
# Should show 3 replicas
|
||||
```
|
||||
|
||||
### Load Balancing
|
||||
|
||||
Caddy automatically load balances between replicas:
|
||||
|
||||
```bash
|
||||
# Test load balancing
|
||||
for i in {1..10}; do
|
||||
curl -s https://getmaplepress.ca/health
|
||||
sleep 1
|
||||
done
|
||||
|
||||
# Check which replicas handled requests
|
||||
docker service logs maplepress_backend | grep "GET /health" | tail -20
|
||||
# You should see different container IDs handling requests
|
||||
```
|
||||
|
||||
### Scale Back Down
|
||||
|
||||
```bash
|
||||
# Edit stack file
|
||||
vi ~/stacks/maplepress-stack.yml
|
||||
|
||||
# Change back to replicas: 1
|
||||
# Redeploy
|
||||
docker stack deploy -c maplepress-stack.yml maplepress
|
||||
|
||||
# Verify
|
||||
docker service ps maplepress_backend
|
||||
# Should show only 1 replica running, others Shutdown
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference: Domain Change Checklist
|
||||
|
||||
### Backend Domain Change
|
||||
|
||||
- [ ] Update DNS A records (point new domain to worker-6)
|
||||
- [ ] Wait for DNS propagation (5-60 minutes)
|
||||
- [ ] Backup Caddyfile: `cp Caddyfile Caddyfile.backup.$(date +%Y%m%d)`
|
||||
- [ ] Update backend Caddyfile with new domain
|
||||
- [ ] Redeploy backend stack
|
||||
- [ ] Verify SSL certificate obtained for new domain
|
||||
- [ ] Update frontend `.env.production` with new backend URL
|
||||
- [ ] Rebuild and redeploy frontend
|
||||
- [ ] Test CORS end-to-end
|
||||
|
||||
### Frontend Domain Change
|
||||
|
||||
- [ ] Update DNS A records (point new domain to worker-7)
|
||||
- [ ] Wait for DNS propagation
|
||||
- [ ] Backup frontend Caddyfile
|
||||
- [ ] Update frontend Caddyfile with new domain
|
||||
- [ ] **Update backend CORS** in `maplepress-stack.yml`
|
||||
- [ ] Redeploy backend (for CORS)
|
||||
- [ ] Redeploy frontend stack
|
||||
- [ ] Verify SSL certificate
|
||||
- [ ] Test in browser (no CORS errors)
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting Domain Changes
|
||||
|
||||
### Problem: SSL Certificate Not Obtained
|
||||
|
||||
**Symptom:** After domain change, HTTPS doesn't work
|
||||
|
||||
```bash
|
||||
# Check Caddy logs
|
||||
docker service logs maplepress_backend-caddy --tail 100 | grep -i "acme\|certificate"
|
||||
|
||||
# Common issues:
|
||||
# 1. DNS not propagated - wait longer
|
||||
# 2. Port 80 not accessible - check firewall
|
||||
# 3. Let's Encrypt rate limit - wait 1 hour
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
|
||||
```bash
|
||||
# Verify DNS resolves
|
||||
dig <new-domain> +short
|
||||
# Must show correct worker IP
|
||||
|
||||
# Verify port 80 accessible
|
||||
curl http://<new-domain>
|
||||
# Should redirect to HTTPS
|
||||
|
||||
# If rate limited, wait and retry
|
||||
# Let's Encrypt limit: 5 certificates per domain per week
|
||||
```
|
||||
|
||||
### Problem: CORS Errors After Domain Change
|
||||
|
||||
**Symptom:** Frontend shows CORS errors in browser console
|
||||
|
||||
**Cause:** Forgot to update backend CORS configuration
|
||||
|
||||
**Fix:**
|
||||
|
||||
```bash
|
||||
# Check backend CORS config
|
||||
cat ~/stacks/maplepress-stack.yml | grep CORS
|
||||
# Should include NEW frontend domain
|
||||
|
||||
# Update if needed
|
||||
vi ~/stacks/maplepress-stack.yml
|
||||
# Add new frontend domain to SECURITY_CORS_ALLOWED_ORIGINS
|
||||
|
||||
# Redeploy backend
|
||||
docker stack rm maplepress
|
||||
sleep 10
|
||||
docker config rm maplepress_caddyfile
|
||||
docker stack deploy -c maplepress-stack.yml maplepress
|
||||
|
||||
# Test CORS
|
||||
curl -v -H "Origin: https://<new-frontend-domain>" https://<backend-domain>/health 2>&1 | grep "access-control"
|
||||
```
|
||||
|
||||
### Problem: Old Domain Still Works
|
||||
|
||||
**Symptom:** Both old and new domains work
|
||||
|
||||
**Cause:** Caddyfile includes both domains
|
||||
|
||||
**Expected Behavior:** This is fine during migration - you can support both
|
||||
|
||||
**To Remove Old Domain:**
|
||||
|
||||
```bash
|
||||
# Edit Caddyfile and remove old domain
|
||||
vi ~/stacks/caddy-config/Caddyfile
|
||||
# Remove old domain from the domain list
|
||||
|
||||
# Redeploy
|
||||
docker stack rm maplepress
|
||||
sleep 10
|
||||
docker config rm maplepress_caddyfile
|
||||
docker stack deploy -c maplepress-stack.yml maplepress
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: November 2025
|
||||
**Maintained By**: Infrastructure Team
|
||||
Loading…
Add table
Add a link
Reference in a new issue