monorepo/cloud/infrastructure/production/operations/ENVIRONMENT_VARIABLES.md

Managing Environment Variables in Production

Audience: DevOps Engineers, System Administrators Last Updated: November 2025 Applies To: MapleFile Backend

---

Table of Contents

1. Overview
2. Architecture
3. Updating Environment Variables
4. Updating Secrets
5. Common Scenarios
6. Verification and Rollback
7. Troubleshooting
8. Best Practices

---

Overview

What Are Environment Variables?

Environment variables configure your application's behavior without changing code:

• Database connections (Cassandra, Redis)
• External APIs (Mailgun, AWS S3)
• Application settings (CORS origins, JWT duration, log level)
• Feature flags (leader election, etc.)

Where Configuration Lives

All configuration is managed from the MANAGER node (<MANAGER_IP>):

Manager Node (<MANAGER_IP>)
└── ~/stacks/
    ├── maplefile-stack.yml          # Stack definition with environment variables
    ├── maplefile-caddy-config/
    │   └── Caddyfile                # Caddy reverse proxy config
    └── secrets/ (managed separately by Docker Swarm)
        ├── maplefile_jwt_secret
        ├── maplefile_mailgun_api_key
        ├── redis_password
        ├── spaces_access_key
        └── spaces_secret_key

⚠️ NEVER edit configuration on worker nodes! Workers receive configuration from the manager via Docker Swarm.

---

Architecture

How Configuration Works

MapleFile backend uses two types of configuration:

1. Environment Variables (in maplefile-stack.yml)

Non-sensitive configuration defined directly in the stack file:

services:
  backend:
    environment:
      # Application
      - APP_ENVIRONMENT=production
      - SERVER_PORT=8000

      # Database
      - DATABASE_HOSTS=cassandra-1,cassandra-2,cassandra-3
      - DATABASE_KEYSPACE=maplefile

      # Mailgun
      - MAILGUN_DOMAIN=mg.example.com
      - MAILGUN_FROM_EMAIL=no-reply@mg.example.com

      # CORS
      - SECURITY_ALLOWED_ORIGINS=https://maplefile.com

2. Docker Secrets (sensitive credentials)

Sensitive values managed by Docker Swarm and mounted into containers:

services:
  backend:
    secrets:
      - maplefile_jwt_secret
      - redis_password
      - maplefile_mailgun_api_key
      - spaces_access_key
      - spaces_secret_key
    command:
      - |
        # Secrets are read from /run/secrets/ and exported
        export JWT_SECRET=$(cat /run/secrets/maplefile_jwt_secret)
        export CACHE_PASSWORD=$(cat /run/secrets/redis_password)
        export MAILGUN_API_KEY=$(cat /run/secrets/maplefile_mailgun_api_key)
        exec /app/maplefile-backend daemon

Configuration Flow

Manager Node
    ├── maplefile-stack.yml (environment variables)
    └── Docker Swarm Secrets (sensitive values)
          ↓
    Docker Stack Deploy
          ↓
    Docker Swarm Manager
          ↓
    Worker-8 (Backend)

Environment Variables vs Secrets

Type	Use For	Location	Example
Environment Variables	Non-sensitive config	maplefile-stack.yml	APP_ENVIRONMENT=production
Secrets	Sensitive credentials	Docker Swarm secrets	API keys, passwords, JWT secret

Why separate?

• Environment variables: Visible in docker inspect (OK for non-sensitive data)
• Secrets: Encrypted by Docker Swarm, only accessible inside containers (required for credentials)

---

Updating Environment Variables

Environment variables are defined directly in the maplefile-stack.yml file.

Step 1: SSH to Manager Node

ssh dockeradmin@<MANAGER_IP>

Step 2: Backup Current Stack File

Always backup before making changes:

cd ~/stacks
cp maplefile-stack.yml maplefile-stack.yml.backup-$(date +%Y%m%d-%H%M%S)

# Verify backup created
ls -la maplefile-stack.yml.backup-*

Step 3: Edit Stack File

# Open editor
nano ~/stacks/maplefile-stack.yml

Find the environment: section and make your changes:

services:
  backend:
    environment:
      # Change log level from INFO to DEBUG
      - LOG_LEVEL=debug

      # Update CORS origins
      - SECURITY_ALLOWED_ORIGINS=https://maplefile.com,https://www.maplefile.com

Save changes:

• Press Ctrl+O to save
• Press Enter to confirm
• Press Ctrl+X to exit

Step 4: Verify Changes

# Check what you changed
cat ~/stacks/maplefile-stack.yml | grep -A5 "environment:"

# Or search for specific variable
cat ~/stacks/maplefile-stack.yml | grep LOG_LEVEL

Step 5: Redeploy the Stack

cd ~/stacks

# Redeploy stack (picks up new environment variables)
docker stack deploy -c maplefile-stack.yml maplefile

Expected output:

Updating service maplefile_backend (id: xyz123...)
Updating service maplefile_backend-caddy (id: abc456...)

Note: Docker Swarm will perform a rolling update with zero downtime.

Step 6: Monitor Deployment

# Watch service update progress
docker service ps maplefile_backend

# Check logs for errors
docker service logs maplefile_backend --tail 50

# Verify service is healthy
docker service ls | grep maplefile

Expected healthy state:

NAME                       REPLICAS   IMAGE
maplefile_backend          1/1        registry.digitalocean.com/ssp/maplefile-backend:prod
maplefile_backend-caddy    1/1        caddy:2.9.1-alpine

---

Updating Secrets

When to Update Secrets

• API keys rotated (Mailgun, AWS S3)
• Passwords changed (Redis)
• Security incident (compromised credentials)
• JWT secret rotation (security best practice)

Understanding Docker Secrets

Docker secrets are IMMUTABLE. Once created, they cannot be changed. To update a secret:

1. Remove the stack completely
2. Delete the old secret
3. Create new secret with updated value
4. Redeploy stack

Step 1: SSH to Manager

ssh dockeradmin@<MANAGER_IP>

Step 2: List Current Secrets

docker secret ls

You should see:

NAME                            CREATED
maplefile_jwt_secret            8 hours ago
maplefile_mailgun_api_key       8 hours ago
redis_password                  10 days ago
spaces_access_key               9 days ago
spaces_secret_key               9 days ago

Step 3: Remove Stack

Must remove stack first to release secrets:

docker stack rm maplefile

# Wait for stack to fully shutdown
sleep 20

# Verify stack removed
docker stack ls | grep maplefile
# Should return nothing

Step 4: Remove Old Secret

# Remove the secret you want to update
docker secret rm maplefile_mailgun_api_key

# Verify removed
docker secret ls | grep mailgun
# Should return nothing

Step 5: Create New Secret

Method 1: Using echo (recommended):

# Create new secret from command line
echo "key-NEW_MAILGUN_API_KEY_HERE" | docker secret create maplefile_mailgun_api_key -

# Verify created
docker secret ls | grep mailgun

Method 2: Using file:

# Create temporary file
echo "key-NEW_MAILGUN_API_KEY_HERE" > /tmp/mailgun_key.txt

# Create secret from file
docker secret create maplefile_mailgun_api_key /tmp/mailgun_key.txt

# Remove temporary file (important!)
rm /tmp/mailgun_key.txt

# Verify created
docker secret ls | grep mailgun

⚠️ Important:

• No quotes around the value
• No trailing newlines or spaces
• Exact format required by the application

Step 6: Redeploy Stack

cd ~/stacks

# Deploy stack with new secret
docker stack deploy -c maplefile-stack.yml maplefile

# Watch services start
docker service ls

Step 7: Verify Secret Updated

# Check service logs for successful startup
docker service logs maplefile_backend --tail 50

# Look for successful initialization
docker service logs maplefile_backend --tail 100 | grep -i "connected\|started"

# Test the service
curl -I https://maplefile.ca/health
# Should return: HTTP/2 200

---

Common Scenarios

Scenario 1: Update Mailgun API Key

Problem: Email sending fails with 401 Forbidden

Solution:

# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Remove stack
docker stack rm maplefile
sleep 20

# Remove old secret
docker secret rm maplefile_mailgun_api_key

# Create new secret (get key from Mailgun dashboard)
echo "key-YOUR_NEW_MAILGUN_API_KEY" | docker secret create maplefile_mailgun_api_key -

# Redeploy
cd ~/stacks
docker stack deploy -c maplefile-stack.yml maplefile

# Monitor
docker service logs -f maplefile_backend --tail 20
# Test email sending from app

Scenario 2: Update Mailgun Domain

Problem: Need to change from mg.example.com to maplefile.ca

Solution:

# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Backup stack file
cd ~/stacks
cp maplefile-stack.yml maplefile-stack.yml.backup-$(date +%Y%m%d)

# Edit stack file
nano maplefile-stack.yml

# Find and update:
#   - MAILGUN_DOMAIN=maplefile.ca
#   - MAILGUN_FROM_EMAIL=noreply@maplefile.ca
#   - MAILGUN_BACKEND_DOMAIN=maplefile.ca

# Save and redeploy
docker stack deploy -c maplefile-stack.yml maplefile

# Monitor
docker service logs maplefile_backend --tail 50 | grep -i mailgun

Scenario 3: Update CORS Origins

Problem: Frontend domain changed or new domain added

Solution:

# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Backup
cd ~/stacks
cp maplefile-stack.yml maplefile-stack.yml.backup-$(date +%Y%m%d)

# Edit
nano maplefile-stack.yml

# Find and update:
# - SECURITY_ALLOWED_ORIGINS=https://maplefile.com,https://www.maplefile.com,https://new-domain.com

# Save and redeploy
docker stack deploy -c maplefile-stack.yml maplefile

# Test from browser (check for CORS errors in console)

Scenario 4: Change JWT Secret (Security Incident)

Problem: JWT secret potentially compromised

⚠️ WARNING: This will invalidate ALL user sessions!

Solution:

# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Generate new secure secret (64 characters)
NEW_SECRET=$(openssl rand -base64 48)
echo "New JWT secret generated (not shown for security)"

# Remove stack
docker stack rm maplefile
sleep 20

# Remove old secret
docker secret rm maplefile_jwt_secret

# Create new secret
echo "$NEW_SECRET" | docker secret create maplefile_jwt_secret -

# Redeploy
cd ~/stacks
docker stack deploy -c maplefile-stack.yml maplefile

# Monitor startup
docker service logs maplefile_backend --tail 50

# ⚠️ All users will need to log in again!

Scenario 5: Enable Debug Logging

Problem: Need detailed logs for troubleshooting

Solution:

# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Backup
cd ~/stacks
cp maplefile-stack.yml maplefile-stack.yml.backup-$(date +%Y%m%d)

# Edit
nano maplefile-stack.yml

# Find and change:
# - LOG_LEVEL=debug  # Was: info

# Save and redeploy
docker stack deploy -c maplefile-stack.yml maplefile

# Watch detailed logs
docker service logs -f maplefile_backend --tail 100

# ⚠️ Remember to set back to info when done!

Scenario 6: Update AWS S3 Credentials

Problem: S3 access keys rotated (DigitalOcean Spaces)

Solution:

# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Remove stack
docker stack rm maplefile
sleep 20

# Remove old secrets
docker secret rm spaces_access_key
docker secret rm spaces_secret_key

# Create new secrets (get from DigitalOcean Spaces dashboard)
echo "YOUR_NEW_ACCESS_KEY" | docker secret create spaces_access_key -
echo "YOUR_NEW_SECRET_KEY" | docker secret create spaces_secret_key -

# Verify created
docker secret ls | grep spaces

# Redeploy
cd ~/stacks
docker stack deploy -c maplefile-stack.yml maplefile

# Test S3 access
docker service logs maplefile_backend --tail 50 | grep -i "s3\|storage"

Scenario 7: Update Database Hosts

Problem: Cassandra node hostname changed

Solution:

# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Backup
cd ~/stacks
cp maplefile-stack.yml maplefile-stack.yml.backup-$(date +%Y%m%d)

# Edit
nano maplefile-stack.yml

# Find and update:
# - DATABASE_HOSTS=cassandra-1,cassandra-2,cassandra-3,cassandra-4

# Save and redeploy
docker stack deploy -c maplefile-stack.yml maplefile

# Monitor connection
docker service logs maplefile_backend --tail 100 | grep -i cassandra

Scenario 8: Update Redis Password

Problem: Redis password changed

Solution:

# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Remove stack
docker stack rm maplefile
sleep 20

# Remove old secret
docker secret rm redis_password

# Create new secret
echo "NEW_REDIS_PASSWORD_HERE" | docker secret create redis_password -

# Redeploy
cd ~/stacks
docker stack deploy -c maplefile-stack.yml maplefile

# Monitor Redis connection
docker service logs maplefile_backend --tail 50 | grep -i redis

---

Verification and Rollback

Verify Changes Applied

Check service updated:

# Check service update time
docker service ps maplefile_backend --format "table {{.Name}}\t{{.Image}}\t{{.CurrentState}}"

# Recent "Running" state means it restarted

Check environment inside container:

# Get container ID
CONTAINER_ID=$(docker ps -q -f name=maplefile_backend)

# Check environment variable
docker exec $CONTAINER_ID env | grep LOG_LEVEL
# Should show: LOG_LEVEL=debug

# DON'T print secrets to terminal!
# Instead, check if they exist:
docker exec $CONTAINER_ID sh -c 'test -f /run/secrets/maplefile_jwt_secret && echo "JWT secret exists" || echo "JWT secret missing"'

Check application logs:

# Look for initialization messages
docker service logs maplefile_backend --tail 100 | grep -i "connected\|initialized"

# Check for errors
docker service logs maplefile_backend --tail 100 | grep -i "error\|fatal\|panic"

Rollback Configuration

If something goes wrong:

# SSH to manager
ssh dockeradmin@<MANAGER_IP>
cd ~/stacks

# List backups
ls -la maplefile-stack.yml.backup-*

# Restore from backup
cp maplefile-stack.yml.backup-YYYYMMDD-HHMMSS maplefile-stack.yml

# Redeploy with old config
docker stack deploy -c maplefile-stack.yml maplefile

# Verify rollback successful
docker service logs maplefile_backend --tail 50

Rollback Secrets

To rollback a secret:

# Remove stack
docker stack rm maplefile
sleep 20

# Remove new secret
docker secret rm maplefile_mailgun_api_key

# Recreate old secret (you need to have saved the old value!)
echo "OLD_SECRET_VALUE" | docker secret create maplefile_mailgun_api_key -

# Redeploy
docker stack deploy -c maplefile-stack.yml maplefile

⚠️ Important: This is why you should always backup secret values before changing them!

Rollback Service (Docker Swarm)

If service is failing after update:

# Docker Swarm can rollback to previous image version
docker service rollback maplefile_backend

# Watch rollback
docker service ps maplefile_backend

---

Troubleshooting

Problem: Changes Not Applied

Symptom: Updated stack file but service still uses old values

Diagnosis:

# Check when stack was last deployed
docker stack ps maplefile --format "table {{.Name}}\t{{.CurrentState}}"

# Check service definition
docker service inspect maplefile_backend --format '{{json .Spec.TaskTemplate.ContainerSpec.Env}}' | jq

Solution:

# Force redeploy by removing and recreating
docker stack rm maplefile
sleep 20
docker stack deploy -c maplefile-stack.yml maplefile

Problem: Service Won't Start After Update

Symptom: Service stuck in "Starting" or "Failed" state

Diagnosis:

# Check service status
docker service ps maplefile_backend --no-trunc

# Check logs for startup errors
docker service logs maplefile_backend --tail 100

Common causes:

1. Invalid environment value:

   # Check for syntax errors in stack file
   cat ~/stacks/maplefile-stack.yml | grep -A50 "environment:"
   

2. Missing required variable:

   # Check logs for "missing" or "required"
   docker service logs maplefile_backend | grep -i "missing\|required"
   

3. Secret not found:

   # Verify secret exists
   docker secret ls | grep maplefile
   
   # If missing, recreate it
   echo "SECRET_VALUE" | docker secret create maplefile_jwt_secret -
   

Solution:

• Fix the invalid value
• Redeploy
• If still failing, rollback to backup

Problem: Secret Not Updating

Symptom: Created new secret but service still uses old value

Cause: Stack still references old secret

Solution:

# Must remove stack completely first
docker stack rm maplefile
sleep 20

# Verify stack removed
docker stack ls
docker ps | grep maplefile  # Should return nothing

# Now redeploy
docker stack deploy -c maplefile-stack.yml maplefile

Problem: Can't Remove Secret - "In Use"

Symptom: Error response from daemon: secret is in use by service

Cause: Service is still using the secret

Solution:

# Must remove stack first
docker stack rm maplefile
sleep 20

# Now you can remove secret
docker secret rm maplefile_jwt_secret

# Recreate and redeploy
echo "NEW_SECRET" | docker secret create maplefile_jwt_secret -
docker stack deploy -c maplefile-stack.yml maplefile

Problem: YAML Syntax Error

Symptom: error parsing YAML file

Diagnosis:

# Check YAML syntax
cat ~/stacks/maplefile-stack.yml

# Common issues:
# - Inconsistent indentation (use spaces, not tabs)
# - Missing colons
# - Incorrect nesting

Solution:

# Restore from backup
cp maplefile-stack.yml.backup-LATEST maplefile-stack.yml

# Try again with correct YAML syntax

---

Best Practices

1. Always Backup Before Changes

# Good practice - timestamped backups
cp maplefile-stack.yml maplefile-stack.yml.backup-$(date +%Y%m%d-%H%M%S)

# Keep backups organized
mkdir -p ~/stacks/backups/$(date +%Y%m%d)
cp maplefile-stack.yml ~/stacks/backups/$(date +%Y%m%d)/

2. Document Secret Values Before Changing

# Save old secret value to temporary secure location
# (NOT in version control!)
docker secret inspect maplefile_mailgun_api_key --format '{{.ID}}' > /tmp/old_secret_id.txt

# Or write it down securely before removing

3. Test in Development First

For major changes:

# If you have a dev environment, test there first
# Then apply same changes to production

4. Use Strong Secrets

# Generate secure random secrets
openssl rand -base64 48  # JWT secret (64 chars)
openssl rand -hex 32     # API tokens (64 chars)

# Don't use:
# - Weak passwords (password123)
# - Default values (secret)
# - Short strings (abc)

5. Rotate Secrets Regularly

Security schedule:

Secret	Rotation Frequency	Priority
JWT Secret	Every 6 months	High
API Keys (Mailgun, S3)	When provider requires	Medium
Redis Password	Yearly	Medium

6. Monitor After Changes

# After deploying changes, monitor for at least 5 minutes
docker service logs -f maplefile_backend --tail 50

# Check for:
# - Successful startup messages
# - No error messages
# - Expected functionality (test key features)

7. Keep Stack Files in Version Control

# Initialize git if not already done
cd ~/stacks
git init

# Add stack files (but NOT secrets!)
git add maplefile-stack.yml
git add maplefile-caddy-config/Caddyfile

# Commit
git commit -m "Update Mailgun domain configuration"

Add to .gitignore:

# Create .gitignore
cat > ~/stacks/.gitignore << 'EOF'
# Never commit backups
*.backup-*

# Never commit secrets
secrets/

# Never commit temporary files
*.tmp
*.log
EOF

8. Use Comments in Stack File

services:
  backend:
    environment:
      # Updated 2025-11-14: Changed to EU region for better performance
      - MAILGUN_API_BASE=https://api.eu.mailgun.net/v3

---

Quick Reference

Essential Commands

# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Edit stack file
nano ~/stacks/maplefile-stack.yml

# Redeploy stack (for environment variable changes)
docker stack deploy -c ~/stacks/maplefile-stack.yml maplefile

# Update secret (requires removing stack first)
docker stack rm maplefile
sleep 20
docker secret rm maplefile_mailgun_api_key
echo "NEW_KEY" | docker secret create maplefile_mailgun_api_key -
docker stack deploy -c ~/stacks/maplefile-stack.yml maplefile

# Watch logs
docker service logs -f maplefile_backend --tail 50

# Check service health
docker service ls | grep maplefile

# Rollback service
docker service rollback maplefile_backend

File Locations

Item	Location
Stack Definition	~/stacks/maplefile-stack.yml
Caddy Config	~/stacks/maplefile-caddy-config/Caddyfile
Secrets	Managed by Docker Swarm (use docker secret commands)
Backups	~/stacks/*.backup-*

Docker Secrets

Secret Name	Purpose
maplefile_jwt_secret	JWT token signing
maplefile_mailgun_api_key	Mailgun email API
redis_password	Redis cache authentication
spaces_access_key	DigitalOcean Spaces access key
spaces_secret_key	DigitalOcean Spaces secret key

---

Related Documentation

• Guide 09: MapleFile Backend Deployment
• Guide 10: MapleFile Caddy Setup
• Guide 12: Horizontal Scaling

---

Questions?

• Check service logs: docker service logs maplefile_backend
• Review stack file: cat ~/stacks/maplefile-stack.yml
• List secrets: docker secret ls

Last Updated: November 2025