# Managing Environment Variables in Production

**Audience**: DevOps Engineers, System Administrators
**Last Updated**: November 2025
**Applies To**: MapleFile Backend

---

## Table of Contents

1. [Overview](#overview)
2. [Architecture](#architecture)
3. [Updating Environment Variables](#updating-environment-variables)
4. [Updating Secrets](#updating-secrets)
5. [Common Scenarios](#common-scenarios)
6. [Verification and Rollback](#verification-and-rollback)
7. [Troubleshooting](#troubleshooting)
8. [Best Practices](#best-practices)

---

## Overview

### What Are Environment Variables?

Environment variables configure your application's behavior without changing code:
- **Database connections** (Cassandra, Redis)
- **External APIs** (Mailgun, AWS S3)
- **Application settings** (CORS origins, JWT duration, log level)
- **Feature flags** (leader election, etc.)

### Where Configuration Lives

**All configuration is managed from the MANAGER node** (<MANAGER_IP>):

```
Manager Node (<MANAGER_IP>)
└── ~/stacks/
    ├── maplefile-stack.yml          # Stack definition with environment variables
    ├── maplefile-caddy-config/
    │   └── Caddyfile                # Caddy reverse proxy config
    └── secrets/ (managed separately by Docker Swarm)
        ├── maplefile_jwt_secret
        ├── maplefile_mailgun_api_key
        ├── redis_password
        ├── spaces_access_key
        └── spaces_secret_key
```

**⚠️ NEVER edit configuration on worker nodes!** Workers receive configuration from the manager via Docker Swarm.

---

## Architecture

### How Configuration Works

MapleFile backend uses **two types of configuration**:

#### 1. Environment Variables (in maplefile-stack.yml)

**Non-sensitive configuration** defined directly in the stack file:

```yaml
services:
  backend:
    environment:
      # Application
      - APP_ENVIRONMENT=production
      - SERVER_PORT=8000

      # Database
      - DATABASE_HOSTS=cassandra-1,cassandra-2,cassandra-3
      - DATABASE_KEYSPACE=maplefile

      # Mailgun
      - MAILGUN_DOMAIN=mg.example.com
      - MAILGUN_FROM_EMAIL=no-reply@mg.example.com

      # CORS
      - SECURITY_ALLOWED_ORIGINS=https://maplefile.com
```

#### 2. Docker Secrets (sensitive credentials)

**Sensitive values** managed by Docker Swarm and mounted into containers:

```yaml
services:
  backend:
    secrets:
      - maplefile_jwt_secret
      - redis_password
      - maplefile_mailgun_api_key
      - spaces_access_key
      - spaces_secret_key
    command:
      - |
        # Secrets are read from /run/secrets/ and exported
        export JWT_SECRET=$(cat /run/secrets/maplefile_jwt_secret)
        export CACHE_PASSWORD=$(cat /run/secrets/redis_password)
        export MAILGUN_API_KEY=$(cat /run/secrets/maplefile_mailgun_api_key)
        exec /app/maplefile-backend daemon
```

### Configuration Flow

```
Manager Node
    ├── maplefile-stack.yml (environment variables)
    └── Docker Swarm Secrets (sensitive values)
          ↓
    Docker Stack Deploy
          ↓
    Docker Swarm Manager
          ↓
    Worker-8 (Backend)
```

### Environment Variables vs Secrets

| Type | Use For | Location | Example |
|------|---------|----------|---------|
| **Environment Variables** | Non-sensitive config | `maplefile-stack.yml` | `APP_ENVIRONMENT=production` |
| **Secrets** | Sensitive credentials | Docker Swarm secrets | API keys, passwords, JWT secret |

**Why separate?**
- **Environment variables**: Visible in `docker inspect` (OK for non-sensitive data)
- **Secrets**: Encrypted by Docker Swarm, only accessible inside containers (required for credentials)

---

## Updating Environment Variables

Environment variables are defined directly in the `maplefile-stack.yml` file.

### Step 1: SSH to Manager Node

```bash
ssh dockeradmin@<MANAGER_IP>
```

### Step 2: Backup Current Stack File

**Always backup before making changes:**

```bash
cd ~/stacks
cp maplefile-stack.yml maplefile-stack.yml.backup-$(date +%Y%m%d-%H%M%S)

# Verify backup created
ls -la maplefile-stack.yml.backup-*
```

### Step 3: Edit Stack File

```bash
# Open editor
nano ~/stacks/maplefile-stack.yml
```

**Find the `environment:` section and make your changes:**

```yaml
services:
  backend:
    environment:
      # Change log level from INFO to DEBUG
      - LOG_LEVEL=debug

      # Update CORS origins
      - SECURITY_ALLOWED_ORIGINS=https://maplefile.com,https://www.maplefile.com
```

**Save changes:**
- Press `Ctrl+O` to save
- Press `Enter` to confirm
- Press `Ctrl+X` to exit

### Step 4: Verify Changes

```bash
# Check what you changed
cat ~/stacks/maplefile-stack.yml | grep -A5 "environment:"

# Or search for specific variable
cat ~/stacks/maplefile-stack.yml | grep LOG_LEVEL
```

### Step 5: Redeploy the Stack

```bash
cd ~/stacks

# Redeploy stack (picks up new environment variables)
docker stack deploy -c maplefile-stack.yml maplefile
```

**Expected output:**
```
Updating service maplefile_backend (id: xyz123...)
Updating service maplefile_backend-caddy (id: abc456...)
```

**Note:** Docker Swarm will perform a rolling update with zero downtime.

### Step 6: Monitor Deployment

```bash
# Watch service update progress
docker service ps maplefile_backend

# Check logs for errors
docker service logs maplefile_backend --tail 50

# Verify service is healthy
docker service ls | grep maplefile
```

**Expected healthy state:**
```
NAME                       REPLICAS   IMAGE
maplefile_backend          1/1        registry.digitalocean.com/ssp/maplefile-backend:prod
maplefile_backend-caddy    1/1        caddy:2.9.1-alpine
```

---

## Updating Secrets

### When to Update Secrets

- **API keys rotated** (Mailgun, AWS S3)
- **Passwords changed** (Redis)
- **Security incident** (compromised credentials)
- **JWT secret rotation** (security best practice)

### Understanding Docker Secrets

**Docker secrets are IMMUTABLE.** Once created, they cannot be changed. To update a secret:

1. Remove the stack completely
2. Delete the old secret
3. Create new secret with updated value
4. Redeploy stack

### Step 1: SSH to Manager

```bash
ssh dockeradmin@<MANAGER_IP>
```

### Step 2: List Current Secrets

```bash
docker secret ls
```

**You should see:**
```
NAME                            CREATED
maplefile_jwt_secret            8 hours ago
maplefile_mailgun_api_key       8 hours ago
redis_password                  10 days ago
spaces_access_key               9 days ago
spaces_secret_key               9 days ago
```

### Step 3: Remove Stack

**Must remove stack first to release secrets:**

```bash
docker stack rm maplefile

# Wait for stack to fully shutdown
sleep 20

# Verify stack removed
docker stack ls | grep maplefile
# Should return nothing
```

### Step 4: Remove Old Secret

```bash
# Remove the secret you want to update
docker secret rm maplefile_mailgun_api_key

# Verify removed
docker secret ls | grep mailgun
# Should return nothing
```

### Step 5: Create New Secret

**Method 1: Using echo (recommended):**

```bash
# Create new secret from command line
echo "key-NEW_MAILGUN_API_KEY_HERE" | docker secret create maplefile_mailgun_api_key -

# Verify created
docker secret ls | grep mailgun
```

**Method 2: Using file:**

```bash
# Create temporary file
echo "key-NEW_MAILGUN_API_KEY_HERE" > /tmp/mailgun_key.txt

# Create secret from file
docker secret create maplefile_mailgun_api_key /tmp/mailgun_key.txt

# Remove temporary file (important!)
rm /tmp/mailgun_key.txt

# Verify created
docker secret ls | grep mailgun
```

**⚠️ Important:**
- No quotes around the value
- No trailing newlines or spaces
- Exact format required by the application

### Step 6: Redeploy Stack

```bash
cd ~/stacks

# Deploy stack with new secret
docker stack deploy -c maplefile-stack.yml maplefile

# Watch services start
docker service ls
```

### Step 7: Verify Secret Updated

```bash
# Check service logs for successful startup
docker service logs maplefile_backend --tail 50

# Look for successful initialization
docker service logs maplefile_backend --tail 100 | grep -i "connected\|started"

# Test the service
curl -I https://maplefile.ca/health
# Should return: HTTP/2 200
```

---

## Common Scenarios

### Scenario 1: Update Mailgun API Key

**Problem:** Email sending fails with 401 Forbidden

**Solution:**

```bash
# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Remove stack
docker stack rm maplefile
sleep 20

# Remove old secret
docker secret rm maplefile_mailgun_api_key

# Create new secret (get key from Mailgun dashboard)
echo "key-YOUR_NEW_MAILGUN_API_KEY" | docker secret create maplefile_mailgun_api_key -

# Redeploy
cd ~/stacks
docker stack deploy -c maplefile-stack.yml maplefile

# Monitor
docker service logs -f maplefile_backend --tail 20
# Test email sending from app
```

### Scenario 2: Update Mailgun Domain

**Problem:** Need to change from `mg.example.com` to `maplefile.ca`

**Solution:**

```bash
# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Backup stack file
cd ~/stacks
cp maplefile-stack.yml maplefile-stack.yml.backup-$(date +%Y%m%d)

# Edit stack file
nano maplefile-stack.yml

# Find and update:
#   - MAILGUN_DOMAIN=maplefile.ca
#   - MAILGUN_FROM_EMAIL=noreply@maplefile.ca
#   - MAILGUN_BACKEND_DOMAIN=maplefile.ca

# Save and redeploy
docker stack deploy -c maplefile-stack.yml maplefile

# Monitor
docker service logs maplefile_backend --tail 50 | grep -i mailgun
```

### Scenario 3: Update CORS Origins

**Problem:** Frontend domain changed or new domain added

**Solution:**

```bash
# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Backup
cd ~/stacks
cp maplefile-stack.yml maplefile-stack.yml.backup-$(date +%Y%m%d)

# Edit
nano maplefile-stack.yml

# Find and update:
# - SECURITY_ALLOWED_ORIGINS=https://maplefile.com,https://www.maplefile.com,https://new-domain.com

# Save and redeploy
docker stack deploy -c maplefile-stack.yml maplefile

# Test from browser (check for CORS errors in console)
```

### Scenario 4: Change JWT Secret (Security Incident)

**Problem:** JWT secret potentially compromised

**⚠️ WARNING:** This will invalidate ALL user sessions!

**Solution:**

```bash
# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Generate new secure secret (64 characters)
NEW_SECRET=$(openssl rand -base64 48)
echo "New JWT secret generated (not shown for security)"

# Remove stack
docker stack rm maplefile
sleep 20

# Remove old secret
docker secret rm maplefile_jwt_secret

# Create new secret
echo "$NEW_SECRET" | docker secret create maplefile_jwt_secret -

# Redeploy
cd ~/stacks
docker stack deploy -c maplefile-stack.yml maplefile

# Monitor startup
docker service logs maplefile_backend --tail 50

# ⚠️ All users will need to log in again!
```

### Scenario 5: Enable Debug Logging

**Problem:** Need detailed logs for troubleshooting

**Solution:**

```bash
# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Backup
cd ~/stacks
cp maplefile-stack.yml maplefile-stack.yml.backup-$(date +%Y%m%d)

# Edit
nano maplefile-stack.yml

# Find and change:
# - LOG_LEVEL=debug  # Was: info

# Save and redeploy
docker stack deploy -c maplefile-stack.yml maplefile

# Watch detailed logs
docker service logs -f maplefile_backend --tail 100

# ⚠️ Remember to set back to info when done!
```

### Scenario 6: Update AWS S3 Credentials

**Problem:** S3 access keys rotated (DigitalOcean Spaces)

**Solution:**

```bash
# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Remove stack
docker stack rm maplefile
sleep 20

# Remove old secrets
docker secret rm spaces_access_key
docker secret rm spaces_secret_key

# Create new secrets (get from DigitalOcean Spaces dashboard)
echo "YOUR_NEW_ACCESS_KEY" | docker secret create spaces_access_key -
echo "YOUR_NEW_SECRET_KEY" | docker secret create spaces_secret_key -

# Verify created
docker secret ls | grep spaces

# Redeploy
cd ~/stacks
docker stack deploy -c maplefile-stack.yml maplefile

# Test S3 access
docker service logs maplefile_backend --tail 50 | grep -i "s3\|storage"
```

### Scenario 7: Update Database Hosts

**Problem:** Cassandra node hostname changed

**Solution:**

```bash
# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Backup
cd ~/stacks
cp maplefile-stack.yml maplefile-stack.yml.backup-$(date +%Y%m%d)

# Edit
nano maplefile-stack.yml

# Find and update:
# - DATABASE_HOSTS=cassandra-1,cassandra-2,cassandra-3,cassandra-4

# Save and redeploy
docker stack deploy -c maplefile-stack.yml maplefile

# Monitor connection
docker service logs maplefile_backend --tail 100 | grep -i cassandra
```

### Scenario 8: Update Redis Password

**Problem:** Redis password changed

**Solution:**

```bash
# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Remove stack
docker stack rm maplefile
sleep 20

# Remove old secret
docker secret rm redis_password

# Create new secret
echo "NEW_REDIS_PASSWORD_HERE" | docker secret create redis_password -

# Redeploy
cd ~/stacks
docker stack deploy -c maplefile-stack.yml maplefile

# Monitor Redis connection
docker service logs maplefile_backend --tail 50 | grep -i redis
```

---

## Verification and Rollback

### Verify Changes Applied

**Check service updated:**

```bash
# Check service update time
docker service ps maplefile_backend --format "table {{.Name}}\t{{.Image}}\t{{.CurrentState}}"

# Recent "Running" state means it restarted
```

**Check environment inside container:**

```bash
# Get container ID
CONTAINER_ID=$(docker ps -q -f name=maplefile_backend)

# Check environment variable
docker exec $CONTAINER_ID env | grep LOG_LEVEL
# Should show: LOG_LEVEL=debug

# DON'T print secrets to terminal!
# Instead, check if they exist:
docker exec $CONTAINER_ID sh -c 'test -f /run/secrets/maplefile_jwt_secret && echo "JWT secret exists" || echo "JWT secret missing"'
```

**Check application logs:**

```bash
# Look for initialization messages
docker service logs maplefile_backend --tail 100 | grep -i "connected\|initialized"

# Check for errors
docker service logs maplefile_backend --tail 100 | grep -i "error\|fatal\|panic"
```

### Rollback Configuration

**If something goes wrong:**

```bash
# SSH to manager
ssh dockeradmin@<MANAGER_IP>
cd ~/stacks

# List backups
ls -la maplefile-stack.yml.backup-*

# Restore from backup
cp maplefile-stack.yml.backup-YYYYMMDD-HHMMSS maplefile-stack.yml

# Redeploy with old config
docker stack deploy -c maplefile-stack.yml maplefile

# Verify rollback successful
docker service logs maplefile_backend --tail 50
```

### Rollback Secrets

**To rollback a secret:**

```bash
# Remove stack
docker stack rm maplefile
sleep 20

# Remove new secret
docker secret rm maplefile_mailgun_api_key

# Recreate old secret (you need to have saved the old value!)
echo "OLD_SECRET_VALUE" | docker secret create maplefile_mailgun_api_key -

# Redeploy
docker stack deploy -c maplefile-stack.yml maplefile
```

**⚠️ Important:** This is why you should always backup secret values before changing them!

### Rollback Service (Docker Swarm)

**If service is failing after update:**

```bash
# Docker Swarm can rollback to previous image version
docker service rollback maplefile_backend

# Watch rollback
docker service ps maplefile_backend
```

---

## Troubleshooting

### Problem: Changes Not Applied

**Symptom:** Updated stack file but service still uses old values

**Diagnosis:**

```bash
# Check when stack was last deployed
docker stack ps maplefile --format "table {{.Name}}\t{{.CurrentState}}"

# Check service definition
docker service inspect maplefile_backend --format '{{json .Spec.TaskTemplate.ContainerSpec.Env}}' | jq
```

**Solution:**

```bash
# Force redeploy by removing and recreating
docker stack rm maplefile
sleep 20
docker stack deploy -c maplefile-stack.yml maplefile
```

### Problem: Service Won't Start After Update

**Symptom:** Service stuck in "Starting" or "Failed" state

**Diagnosis:**

```bash
# Check service status
docker service ps maplefile_backend --no-trunc

# Check logs for startup errors
docker service logs maplefile_backend --tail 100
```

**Common causes:**

1. **Invalid environment value:**
   ```bash
   # Check for syntax errors in stack file
   cat ~/stacks/maplefile-stack.yml | grep -A50 "environment:"
   ```

2. **Missing required variable:**
   ```bash
   # Check logs for "missing" or "required"
   docker service logs maplefile_backend | grep -i "missing\|required"
   ```

3. **Secret not found:**
   ```bash
   # Verify secret exists
   docker secret ls | grep maplefile

   # If missing, recreate it
   echo "SECRET_VALUE" | docker secret create maplefile_jwt_secret -
   ```

**Solution:**
- Fix the invalid value
- Redeploy
- If still failing, rollback to backup

### Problem: Secret Not Updating

**Symptom:** Created new secret but service still uses old value

**Cause:** Stack still references old secret

**Solution:**

```bash
# Must remove stack completely first
docker stack rm maplefile
sleep 20

# Verify stack removed
docker stack ls
docker ps | grep maplefile  # Should return nothing

# Now redeploy
docker stack deploy -c maplefile-stack.yml maplefile
```

### Problem: Can't Remove Secret - "In Use"

**Symptom:** `Error response from daemon: secret is in use by service`

**Cause:** Service is still using the secret

**Solution:**

```bash
# Must remove stack first
docker stack rm maplefile
sleep 20

# Now you can remove secret
docker secret rm maplefile_jwt_secret

# Recreate and redeploy
echo "NEW_SECRET" | docker secret create maplefile_jwt_secret -
docker stack deploy -c maplefile-stack.yml maplefile
```

### Problem: YAML Syntax Error

**Symptom:** `error parsing YAML file`

**Diagnosis:**

```bash
# Check YAML syntax
cat ~/stacks/maplefile-stack.yml

# Common issues:
# - Inconsistent indentation (use spaces, not tabs)
# - Missing colons
# - Incorrect nesting
```

**Solution:**

```bash
# Restore from backup
cp maplefile-stack.yml.backup-LATEST maplefile-stack.yml

# Try again with correct YAML syntax
```

---

## Best Practices

### 1. Always Backup Before Changes

```bash
# Good practice - timestamped backups
cp maplefile-stack.yml maplefile-stack.yml.backup-$(date +%Y%m%d-%H%M%S)

# Keep backups organized
mkdir -p ~/stacks/backups/$(date +%Y%m%d)
cp maplefile-stack.yml ~/stacks/backups/$(date +%Y%m%d)/
```

### 2. Document Secret Values Before Changing

```bash
# Save old secret value to temporary secure location
# (NOT in version control!)
docker secret inspect maplefile_mailgun_api_key --format '{{.ID}}' > /tmp/old_secret_id.txt

# Or write it down securely before removing
```

### 3. Test in Development First

**For major changes:**

```bash
# If you have a dev environment, test there first
# Then apply same changes to production
```

### 4. Use Strong Secrets

```bash
# Generate secure random secrets
openssl rand -base64 48  # JWT secret (64 chars)
openssl rand -hex 32     # API tokens (64 chars)

# Don't use:
# - Weak passwords (password123)
# - Default values (secret)
# - Short strings (abc)
```

### 5. Rotate Secrets Regularly

**Security schedule:**

| Secret | Rotation Frequency | Priority |
|--------|-------------------|----------|
| JWT Secret | Every 6 months | High |
| API Keys (Mailgun, S3) | When provider requires | Medium |
| Redis Password | Yearly | Medium |

### 6. Monitor After Changes

```bash
# After deploying changes, monitor for at least 5 minutes
docker service logs -f maplefile_backend --tail 50

# Check for:
# - Successful startup messages
# - No error messages
# - Expected functionality (test key features)
```

### 7. Keep Stack Files in Version Control

```bash
# Initialize git if not already done
cd ~/stacks
git init

# Add stack files (but NOT secrets!)
git add maplefile-stack.yml
git add maplefile-caddy-config/Caddyfile

# Commit
git commit -m "Update Mailgun domain configuration"
```

**Add to .gitignore:**

```bash
# Create .gitignore
cat > ~/stacks/.gitignore << 'EOF'
# Never commit backups
*.backup-*

# Never commit secrets
secrets/

# Never commit temporary files
*.tmp
*.log
EOF
```

### 8. Use Comments in Stack File

```yaml
services:
  backend:
    environment:
      # Updated 2025-11-14: Changed to EU region for better performance
      - MAILGUN_API_BASE=https://api.eu.mailgun.net/v3
```

---

## Quick Reference

### Essential Commands

```bash
# SSH to manager
ssh dockeradmin@<MANAGER_IP>

# Edit stack file
nano ~/stacks/maplefile-stack.yml

# Redeploy stack (for environment variable changes)
docker stack deploy -c ~/stacks/maplefile-stack.yml maplefile

# Update secret (requires removing stack first)
docker stack rm maplefile
sleep 20
docker secret rm maplefile_mailgun_api_key
echo "NEW_KEY" | docker secret create maplefile_mailgun_api_key -
docker stack deploy -c ~/stacks/maplefile-stack.yml maplefile

# Watch logs
docker service logs -f maplefile_backend --tail 50

# Check service health
docker service ls | grep maplefile

# Rollback service
docker service rollback maplefile_backend
```

### File Locations

| Item | Location |
|------|----------|
| Stack Definition | `~/stacks/maplefile-stack.yml` |
| Caddy Config | `~/stacks/maplefile-caddy-config/Caddyfile` |
| Secrets | Managed by Docker Swarm (use `docker secret` commands) |
| Backups | `~/stacks/*.backup-*` |

### Docker Secrets

| Secret Name | Purpose |
|-------------|---------|
| `maplefile_jwt_secret` | JWT token signing |
| `maplefile_mailgun_api_key` | Mailgun email API |
| `redis_password` | Redis cache authentication |
| `spaces_access_key` | DigitalOcean Spaces access key |
| `spaces_secret_key` | DigitalOcean Spaces secret key |

---

## Related Documentation

- [Guide 09: MapleFile Backend Deployment](../setup/09_maplefile_backend.md)
- [Guide 10: MapleFile Caddy Setup](../setup/10_maplefile_caddy.md)
- [Guide 12: Horizontal Scaling](../setup/12_horizontal_scaling.md)

---

**Questions?**
- Check service logs: `docker service logs maplefile_backend`
- Review stack file: `cat ~/stacks/maplefile-stack.yml`
- List secrets: `docker secret ls`

**Last Updated**: November 2025