Initial commit: Open sourcing all of the Maple Open Technologies code.

2025-12-02 14:33:08 -05:00 · 2025-12-02 14:33:08 -05:00 · 755d54a99d
commit 755d54a99d
2010 changed files with 448675 additions and 0 deletions
--- a/cloud/infrastructure/production/setup/04.5_spaces.md
+++ b/cloud/infrastructure/production/setup/04.5_spaces.md
@ -0,0 +1,511 @@
+# DigitalOcean Spaces Setup (S3-Compatible Object Storage)
+
+**Audience**: Junior DevOps Engineers, Infrastructure Team
+**Time to Complete**: 15-20 minutes
+**Prerequisites**: DigitalOcean account with billing enabled
+
+---
+
+## Overview
+
+This guide sets up **DigitalOcean Spaces** - an S3-compatible object storage service for storing files, uploads, and media for your MaplePress backend.
+
+**What You'll Build:**
+- DigitalOcean Space (bucket) for file storage
+- API keys (access key + secret key) for programmatic access
+- Docker Swarm secrets for secure credential storage
+- Configuration ready for backend integration
+
+**Why DigitalOcean Spaces?**
+- S3-compatible API (works with AWS SDK)
+- Simple pricing: $5/mo for 250GB + 1TB transfer
+- CDN included (speeds up file delivery globally)
+- No egress fees within same region
+- Integrated with your existing DigitalOcean infrastructure
+
+---
+
+## Table of Contents
+
+1. [Create DigitalOcean Space](#step-1-create-digitalocean-space)
+2. [Generate API Keys](#step-2-generate-api-keys)
+3. [Create Docker Secrets](#step-3-create-docker-secrets)
+4. [Verify Configuration](#step-4-verify-configuration)
+5. [Test Access](#step-5-test-access)
+6. [Troubleshooting](#troubleshooting)
+
+---
+
+## Step 1: Create DigitalOcean Space
+
+### 1.1 Create Space via Dashboard
+
+1. Log into DigitalOcean dashboard: https://cloud.digitalocean.com
+2. Click **Manage** → **Spaces Object Storage** in left sidebar
+3. Click **Create a Space**
+4. Configure:
+   - **Choose a datacenter region**: Select same region as your droplets (e.g., `NYC3` or `Toronto`)
+   - **Enable CDN**: ✅ Yes (recommended - improves performance globally)
+   - **Choose a unique name**: `maplepress` (must be globally unique)
+   - **Select a project**: Your project (e.g., "MaplePress Production")
+5. Click **Create a Space**
+
+**Expected output:**
+- Space created successfully
+- You'll see the space URL: `https://maplepress.tor1.digitaloceanspaces.com`
+
+### 1.2 Record Space Information
+
+**Save these values** (you'll need them later):
+
+```bash
+# Space Name
+SPACE_NAME=maplepress
+
+# Endpoint (without https://)
+SPACE_ENDPOINT=tor1.digitaloceanspaces.com
+
+# Region code
+SPACE_REGION=tor1
+
+# Full URL (for reference)
+SPACE_URL=https://maplepress.tor1.digitaloceanspaces.com
+```
+
+**Region codes for reference:**
+- Toronto: `tor1.digitaloceanspaces.com`
+- San Francisco 3: `sfo3.digitaloceanspaces.com`
+- Singapore: `sgp1.digitaloceanspaces.com`
+- Amsterdam: `ams3.digitaloceanspaces.com`
+- Frankfurt: `fra1.digitaloceanspaces.com`
+
+**✅ Checkpoint:** Space created and URL recorded
+
+---
+
+## Step 2: Generate API Keys
+
+### 2.1 Create Spaces Access Keys
+
+1. In DigitalOcean dashboard, go to **API** in left sidebar
+2. Scroll down to **Spaces access keys** section
+3. Click **Generate New Key**
+4. Configure:
+   - **Name**: `maplepress-backend-prod`
+   - **Description**: "Backend service access to Spaces" (optional)
+5. Click **Generate Key**
+
+**⚠️ CRITICAL:** The secret key is **only shown once**! Copy it immediately.
+
+### 2.2 Save Credentials Securely
+
+You'll see:
+- **Access Key**: `DO00ABC123XYZ...` (20 characters)
+- **Secret Key**: `abc123def456...` (40 characters)
+
+**SAVE BOTH IN YOUR PASSWORD MANAGER NOW!**
+
+Example:
+```
+DigitalOcean Spaces - MaplePress Production
+Access Key: DO00ABC123XYZ456
+Secret Key: abc123def456ghi789jkl012mno345pqr678stu901
+Endpoint: nyc3.digitaloceanspaces.com
+Bucket: maplepress
+```
+
+### 2.3 Update Local .env File
+
+**On your local machine:**
+
+```bash
+# Navigate to production infrastructure
+cd ~/monorepo/cloud/infrastructure/production
+
+# Edit .env file
+vi .env
+
+# Add these lines:
+SPACES_ACCESS_KEY=DO00ABC123XYZ456
+SPACES_SECRET_KEY=abc123def456ghi789jkl012mno345pqr678stu901
+SPACES_ENDPOINT=tor1.digitaloceanspaces.com
+SPACES_REGION=tor1
+SPACES_BUCKET=maplepress
+```
+
+Save: `Esc`, `:wq`, `Enter`
+
+**✅ Checkpoint:** API keys saved securely in password manager and `.env` file
+
+---
+
+## Step 3: Create Docker Secrets
+
+**On manager node:**
+
+```bash
+# SSH to manager
+ssh dockeradmin@<manager-public-ip>
+```
+
+### 3.1 Create Spaces Access Key Secret
+
+```bash
+# Create secret for access key
+echo -n "DO00ABC123XYZ456" | docker secret create spaces_access_key -
+
+# Verify
+docker secret ls | grep spaces_access_key
+# Should show: spaces_access_key   About a minute ago
+```
+
+**Important:** Replace `DO00ABC123XYZ456` with your actual access key!
+
+### 3.2 Create Spaces Secret Key Secret
+
+```bash
+# Create secret for secret key
+echo -n "abc123def456ghi789jkl012mno345pqr678stu901" | docker secret create spaces_secret_key -
+
+# Verify
+docker secret ls | grep spaces_secret_key
+# Should show: spaces_secret_key   About a minute ago
+```
+
+**Important:** Replace with your actual secret key!
+
+### 3.3 Verify All Secrets
+
+```bash
+# List all secrets
+docker secret ls
+```
+
+**You should see:**
+
+```
+ID                          NAME                         CREATED
+abc123...                   maplepress_jwt_secret        from 05_backend.md
+abc124...                   maplepress_ip_encryption_key from 05_backend.md
+def456...                   redis_password               from 03_redis.md
+ghi789...                   meilisearch_master_key       from 04_meilisearch.md
+jkl012...                   spaces_access_key            NEW!
+mno345...                   spaces_secret_key            NEW!
+```
+
+**✅ Checkpoint:** All secrets created successfully
+
+---
+
+## Step 4: Verify Configuration
+
+### 4.1 Test Space Access from Local Machine
+
+**Install AWS CLI (if not already installed):**
+
+```bash
+# On your local machine (Mac)
+brew install awscli
+
+# Or on Linux:
+sudo apt install awscli
+```
+
+**Configure AWS CLI for DigitalOcean Spaces:**
+
+```bash
+# Create AWS credentials file
+mkdir -p ~/.aws
+vi ~/.aws/credentials
+
+# Add this profile:
+[digitalocean]
+aws_access_key_id = DO00ABC123XYZ456
+aws_secret_access_key = abc123def456ghi789jkl012mno345pqr678stu901
+```
+
+Save: `Esc`, `:wq`, `Enter`
+
+### 4.2 Test Listing Space Contents
+
+```bash
+# List contents of your space
+aws s3 ls s3://maplepress \
+  --endpoint-url https://tor1.digitaloceanspaces.com \
+  --profile digitalocean
+
+# Should show empty (new space) or list existing files
+```
+
+### 4.3 Test File Upload
+
+```bash
+# Create test file
+echo "Hello from MaplePress!" > test-file.txt
+
+# Upload to space
+aws s3 cp test-file.txt s3://maplepress/test-file.txt \
+  --endpoint-url https://tor1.digitaloceanspaces.com \
+  --profile digitalocean \
+  --acl public-read
+
+# Should show: upload: ./test-file.txt to s3://maplepress/test-file.txt
+```
+
+### 4.4 Test File Download
+
+```bash
+# Download from space
+aws s3 cp s3://maplepress/test-file.txt downloaded-test.txt \
+  --endpoint-url https://tor1.digitaloceanspaces.com \
+  --profile digitalocean
+
+# Verify content
+cat downloaded-test.txt
+# Should show: Hello from MaplePress!
+
+# Clean up
+rm test-file.txt downloaded-test.txt
+```
+
+### 4.5 Test Public URL Access
+
+```bash
+# Try accessing via browser or curl
+curl https://maplepress.tor1.digitaloceanspaces.com/test-file.txt
+
+# Should show: Hello from MaplePress!
+```
+
+**✅ Checkpoint:** Successfully uploaded, listed, downloaded, and accessed file
+
+---
+
+## Step 5: Test Access
+
+### 5.1 Verify Endpoint Resolution
+
+```bash
+# Test DNS resolution
+dig tor1.digitaloceanspaces.com +short
+
+# Should return IP addresses (e.g., 192.81.xxx.xxx)
+```
+
+### 5.2 Test HTTPS Connection
+
+```bash
+# Test SSL/TLS connection
+curl -I https://tor1.digitaloceanspaces.com
+
+# Should return:
+# HTTP/2 403 (Forbidden is OK - means endpoint is reachable)
+```
+
+### 5.3 Check Space Permissions
+
+1. Go to DigitalOcean dashboard → Spaces
+2. Click on your space (`maplepress`)
+3. Click **Settings** tab
+4. Check **File Listing**: Should be ❌ Restricted (recommended for security)
+5. Individual files can be made public via ACL when uploading
+
+**✅ Checkpoint:** Spaces endpoint is accessible and working
+
+---
+
+## Troubleshooting
+
+### Problem: "Space name already exists"
+
+**Symptom:** Can't create space with chosen name
+
+**Cause:** Space names are globally unique across all DigitalOcean customers
+
+**Solution:**
+
+Try these naming patterns:
+- `maplepress-<your-company>`
+- `maplepress-<random-string>`
+- `mp-prod-<date>` (e.g., `mp-prod-2025`)
+
+Check availability by trying different names in the creation form.
+
+### Problem: "Access Denied" When Testing
+
+**Symptom:** AWS CLI returns `AccessDenied` error
+
+**Causes and Solutions:**
+
+1. **Wrong credentials:**
+   ```bash
+   # Verify credentials in ~/.aws/credentials match DigitalOcean dashboard
+   cat ~/.aws/credentials
+   ```
+
+2. **Wrong endpoint:**
+   ```bash
+   # Make sure endpoint matches your space region
+   # NYC3: nyc3.digitaloceanspaces.com
+   # SFO3: sfo3.digitaloceanspaces.com
+   ```
+
+3. **Wrong bucket name:**
+   ```bash
+   # Verify bucket name matches space name exactly
+   aws s3 ls --endpoint-url https://tor1.digitaloceanspaces.com --profile digitalocean
+   # Should list your space
+   ```
+
+### Problem: "NoSuchBucket" Error
+
+**Symptom:** AWS CLI says bucket doesn't exist
+
+**Check:**
+
+```bash
+# List all spaces in your account
+aws s3 ls --endpoint-url https://tor1.digitaloceanspaces.com --profile digitalocean
+
+# Make sure your space appears in the list
+```
+
+**If space is missing:**
+- Check you're in the correct DigitalOcean account
+- Check space wasn't accidentally deleted
+- Check endpoint URL matches space region
+
+### Problem: Files Not Publicly Accessible
+
+**Symptom:** Get 403 Forbidden when accessing file URL
+
+**Cause:** File ACL is private (default)
+
+**Solution:**
+
+```bash
+# Upload with public-read ACL
+aws s3 cp file.txt s3://maplepress/file.txt \
+  --endpoint-url https://tor1.digitaloceanspaces.com \
+  --profile digitalocean \
+  --acl public-read
+
+# Or make existing file public
+aws s3api put-object-acl \
+  --bucket maplepress \
+  --key file.txt \
+  --acl public-read \
+  --endpoint-url https://tor1.digitaloceanspaces.com \
+  --profile digitalocean
+```
+
+**Note:** Your backend will control ACLs programmatically. Public access should only be granted to files that need to be publicly accessible (e.g., user-uploaded images for display).
+
+### Problem: CDN Not Working
+
+**Symptom:** Files load slowly or CDN URL doesn't work
+
+**Check:**
+
+1. Verify CDN is enabled:
+   - DigitalOcean dashboard → Spaces → Your space → Settings
+   - **CDN** should show: ✅ Enabled
+
+2. Use CDN URL instead of direct URL:
+   ```bash
+   # Direct URL (slower):
+   https://maplepress.tor1.digitaloceanspaces.com/file.txt
+
+   # CDN URL (faster):
+   https://maplepress.tor1.cdn.digitaloceanspaces.com/file.txt
+   ```
+
+3. Clear CDN cache if needed:
+   - Spaces → Your space → Settings → CDN
+   - Click **Purge Cache**
+
+### Problem: High Storage Costs
+
+**Symptom:** Unexpected charges for Spaces
+
+**Check:**
+
+```bash
+# Calculate total space usage
+aws s3 ls s3://maplepress --recursive --human-readable --summarize \
+  --endpoint-url https://tor1.digitaloceanspaces.com \
+  --profile digitalocean
+
+# Shows: Total Size: X.XX GB
+```
+
+**Pricing reference:**
+- $5/mo includes 250GB storage + 1TB outbound transfer
+- Additional storage: $0.02/GB per month
+- Additional transfer: $0.01/GB
+
+**Optimization tips:**
+- Delete old/unused files regularly
+- Use CDN to reduce direct space access
+- Compress images before uploading
+- Set up lifecycle policies to auto-delete old files
+
+---
+
+## Next Steps
+
+✅ **You now have:**
+- DigitalOcean Space created and configured
+- API keys generated and secured
+- Docker Swarm secrets created
+- Verified access from local machine
+
+**Next guide:**
+- **05_backend.md** - Deploy MaplePress backend
+- Backend will use these Spaces credentials automatically
+- Files uploaded via backend API will be stored in your Space
+
+**Space Configuration for Backend:**
+
+The backend will use these environment variables (configured in 05_backend.md):
+
+```yaml
+environment:
+  - AWS_ACCESS_KEY_FILE=/run/secrets/spaces_access_key
+  - AWS_SECRET_KEY_FILE=/run/secrets/spaces_secret_key
+  - AWS_ENDPOINT=https://tor1.digitaloceanspaces.com
+  - AWS_REGION=tor1
+  - AWS_BUCKET_NAME=maplepress
+```
+
+**Useful Commands:**
+
+```bash
+# List all files in space
+aws s3 ls s3://maplepress --recursive \
+  --endpoint-url https://tor1.digitaloceanspaces.com \
+  --profile digitalocean
+
+# Get space size
+aws s3 ls s3://maplepress --recursive --summarize \
+  --endpoint-url https://tor1.digitaloceanspaces.com \
+  --profile digitalocean
+
+# Delete test file
+aws s3 rm s3://maplepress/test-file.txt \
+  --endpoint-url https://tor1.digitaloceanspaces.com \
+  --profile digitalocean
+
+# Sync local directory to space
+aws s3 sync ./local-folder s3://maplepress/uploads/ \
+  --endpoint-url https://tor1.digitaloceanspaces.com \
+  --profile digitalocean
+```
+
+---
+
+**Last Updated**: January 2025
+**Maintained By**: Infrastructure Team
+
+**Changelog:**
+- January 2025: Initial DigitalOcean Spaces setup guide for MaplePress production deployment