mapleopentech/monorepo
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
monorepo/cloud/infrastructure/production/setup/04.5_spaces.md

12 KiB
Raw Blame History

DigitalOcean Spaces Setup (S3-Compatible Object Storage)

Audience: Junior DevOps Engineers, Infrastructure Team Time to Complete: 15-20 minutes Prerequisites: DigitalOcean account with billing enabled

Overview

This guide sets up DigitalOcean Spaces - an S3-compatible object storage service for storing files, uploads, and media for your MaplePress backend.

What You'll Build:

  • DigitalOcean Space (bucket) for file storage
  • API keys (access key + secret key) for programmatic access
  • Docker Swarm secrets for secure credential storage
  • Configuration ready for backend integration

Why DigitalOcean Spaces?

  • S3-compatible API (works with AWS SDK)
  • Simple pricing: $5/mo for 250GB + 1TB transfer
  • CDN included (speeds up file delivery globally)
  • No egress fees within same region
  • Integrated with your existing DigitalOcean infrastructure

Table of Contents

  1. Create DigitalOcean Space
  2. Generate API Keys
  3. Create Docker Secrets
  4. Verify Configuration
  5. Test Access
  6. Troubleshooting

Step 1: Create DigitalOcean Space

1.1 Create Space via Dashboard

  1. Log into DigitalOcean dashboard: https://cloud.digitalocean.com
  2. Click Manage → Spaces Object Storage in left sidebar
  3. Click Create a Space
  4. Configure:
    • Choose a datacenter region: Select same region as your droplets (e.g., NYC3 or Toronto)
    • Enable CDN: ✅ Yes (recommended - improves performance globally)
    • Choose a unique name: maplepress (must be globally unique)
    • Select a project: Your project (e.g., "MaplePress Production")
  5. Click Create a Space

Expected output:

  • Space created successfully
  • You'll see the space URL: https://maplepress.tor1.digitaloceanspaces.com

1.2 Record Space Information

Save these values (you'll need them later):

# Space Name
SPACE_NAME=maplepress

# Endpoint (without https://)
SPACE_ENDPOINT=tor1.digitaloceanspaces.com

# Region code
SPACE_REGION=tor1

# Full URL (for reference)
SPACE_URL=https://maplepress.tor1.digitaloceanspaces.com

Region codes for reference:

  • Toronto: tor1.digitaloceanspaces.com
  • San Francisco 3: sfo3.digitaloceanspaces.com
  • Singapore: sgp1.digitaloceanspaces.com
  • Amsterdam: ams3.digitaloceanspaces.com
  • Frankfurt: fra1.digitaloceanspaces.com

✅ Checkpoint: Space created and URL recorded

Step 2: Generate API Keys

2.1 Create Spaces Access Keys

  1. In DigitalOcean dashboard, go to API in left sidebar
  2. Scroll down to Spaces access keys section
  3. Click Generate New Key
  4. Configure:
    • Name: maplepress-backend-prod
    • Description: "Backend service access to Spaces" (optional)
  5. Click Generate Key

⚠️ CRITICAL: The secret key is only shown once! Copy it immediately.

2.2 Save Credentials Securely

You'll see:

  • Access Key: DO00ABC123XYZ... (20 characters)
  • Secret Key: abc123def456... (40 characters)

SAVE BOTH IN YOUR PASSWORD MANAGER NOW!

Example:

DigitalOcean Spaces - MaplePress Production
Access Key: DO00ABC123XYZ456
Secret Key: abc123def456ghi789jkl012mno345pqr678stu901
Endpoint: nyc3.digitaloceanspaces.com
Bucket: maplepress

2.3 Update Local .env File

On your local machine:

# Navigate to production infrastructure
cd ~/monorepo/cloud/infrastructure/production

# Edit .env file
vi .env

# Add these lines:
SPACES_ACCESS_KEY=DO00ABC123XYZ456
SPACES_SECRET_KEY=abc123def456ghi789jkl012mno345pqr678stu901
SPACES_ENDPOINT=tor1.digitaloceanspaces.com
SPACES_REGION=tor1
SPACES_BUCKET=maplepress

Save: Esc, :wq, Enter

✅ Checkpoint: API keys saved securely in password manager and .env file

Step 3: Create Docker Secrets

On manager node:

# SSH to manager
ssh dockeradmin@<manager-public-ip>

3.1 Create Spaces Access Key Secret

# Create secret for access key
echo -n "DO00ABC123XYZ456" | docker secret create spaces_access_key -

# Verify
docker secret ls | grep spaces_access_key
# Should show: spaces_access_key   About a minute ago

Important: Replace DO00ABC123XYZ456 with your actual access key!

3.2 Create Spaces Secret Key Secret

# Create secret for secret key
echo -n "abc123def456ghi789jkl012mno345pqr678stu901" | docker secret create spaces_secret_key -

# Verify
docker secret ls | grep spaces_secret_key
# Should show: spaces_secret_key   About a minute ago

Important: Replace with your actual secret key!

3.3 Verify All Secrets

# List all secrets
docker secret ls

You should see:

ID                          NAME                         CREATED
abc123...                   maplepress_jwt_secret        from 05_backend.md
abc124...                   maplepress_ip_encryption_key from 05_backend.md
def456...                   redis_password               from 03_redis.md
ghi789...                   meilisearch_master_key       from 04_meilisearch.md
jkl012...                   spaces_access_key            NEW!
mno345...                   spaces_secret_key            NEW!

✅ Checkpoint: All secrets created successfully

Step 4: Verify Configuration

4.1 Test Space Access from Local Machine

Install AWS CLI (if not already installed):

# On your local machine (Mac)
brew install awscli

# Or on Linux:
sudo apt install awscli

Configure AWS CLI for DigitalOcean Spaces:

# Create AWS credentials file
mkdir -p ~/.aws
vi ~/.aws/credentials

# Add this profile:
[digitalocean]
aws_access_key_id = DO00ABC123XYZ456
aws_secret_access_key = abc123def456ghi789jkl012mno345pqr678stu901

Save: Esc, :wq, Enter

4.2 Test Listing Space Contents

# List contents of your space
aws s3 ls s3://maplepress \
  --endpoint-url https://tor1.digitaloceanspaces.com \
  --profile digitalocean

# Should show empty (new space) or list existing files

4.3 Test File Upload

# Create test file
echo "Hello from MaplePress!" > test-file.txt

# Upload to space
aws s3 cp test-file.txt s3://maplepress/test-file.txt \
  --endpoint-url https://tor1.digitaloceanspaces.com \
  --profile digitalocean \
  --acl public-read

# Should show: upload: ./test-file.txt to s3://maplepress/test-file.txt

4.4 Test File Download

# Download from space
aws s3 cp s3://maplepress/test-file.txt downloaded-test.txt \
  --endpoint-url https://tor1.digitaloceanspaces.com \
  --profile digitalocean

# Verify content
cat downloaded-test.txt
# Should show: Hello from MaplePress!

# Clean up
rm test-file.txt downloaded-test.txt

4.5 Test Public URL Access

# Try accessing via browser or curl
curl https://maplepress.tor1.digitaloceanspaces.com/test-file.txt

# Should show: Hello from MaplePress!

✅ Checkpoint: Successfully uploaded, listed, downloaded, and accessed file

Step 5: Test Access

5.1 Verify Endpoint Resolution

# Test DNS resolution
dig tor1.digitaloceanspaces.com +short

# Should return IP addresses (e.g., 192.81.xxx.xxx)

5.2 Test HTTPS Connection

# Test SSL/TLS connection
curl -I https://tor1.digitaloceanspaces.com

# Should return:
# HTTP/2 403 (Forbidden is OK - means endpoint is reachable)

5.3 Check Space Permissions

  1. Go to DigitalOcean dashboard → Spaces
  2. Click on your space (maplepress)
  3. Click Settings tab
  4. Check File Listing: Should be ❌ Restricted (recommended for security)
  5. Individual files can be made public via ACL when uploading

✅ Checkpoint: Spaces endpoint is accessible and working

Troubleshooting

Problem: "Space name already exists"

Symptom: Can't create space with chosen name

Cause: Space names are globally unique across all DigitalOcean customers

Solution:

Try these naming patterns:

  • maplepress-<your-company>
  • maplepress-<random-string>
  • mp-prod-<date> (e.g., mp-prod-2025)

Check availability by trying different names in the creation form.

Problem: "Access Denied" When Testing

Symptom: AWS CLI returns AccessDenied error

Causes and Solutions:

  1. Wrong credentials:

    # Verify credentials in ~/.aws/credentials match DigitalOcean dashboard
cat ~/.aws/credentials

  2. Wrong endpoint:

    # Make sure endpoint matches your space region
# NYC3: nyc3.digitaloceanspaces.com
# SFO3: sfo3.digitaloceanspaces.com

  3. Wrong bucket name:

    # Verify bucket name matches space name exactly
aws s3 ls --endpoint-url https://tor1.digitaloceanspaces.com --profile digitalocean
# Should list your space

Problem: "NoSuchBucket" Error

Symptom: AWS CLI says bucket doesn't exist

Check:

# List all spaces in your account
aws s3 ls --endpoint-url https://tor1.digitaloceanspaces.com --profile digitalocean

# Make sure your space appears in the list

If space is missing:

  • Check you're in the correct DigitalOcean account
  • Check space wasn't accidentally deleted
  • Check endpoint URL matches space region

Problem: Files Not Publicly Accessible

Symptom: Get 403 Forbidden when accessing file URL

Cause: File ACL is private (default)

Solution:

# Upload with public-read ACL
aws s3 cp file.txt s3://maplepress/file.txt \
  --endpoint-url https://tor1.digitaloceanspaces.com \
  --profile digitalocean \
  --acl public-read

# Or make existing file public
aws s3api put-object-acl \
  --bucket maplepress \
  --key file.txt \
  --acl public-read \
  --endpoint-url https://tor1.digitaloceanspaces.com \
  --profile digitalocean

Note: Your backend will control ACLs programmatically. Public access should only be granted to files that need to be publicly accessible (e.g., user-uploaded images for display).

Problem: CDN Not Working

Symptom: Files load slowly or CDN URL doesn't work

Check:

  1. Verify CDN is enabled:

    • DigitalOcean dashboard → Spaces → Your space → Settings
    • CDN should show: ✅ Enabled

  2. Use CDN URL instead of direct URL:

    # Direct URL (slower):
https://maplepress.tor1.digitaloceanspaces.com/file.txt

# CDN URL (faster):
https://maplepress.tor1.cdn.digitaloceanspaces.com/file.txt

  3. Clear CDN cache if needed:

    • Spaces → Your space → Settings → CDN
    • Click Purge Cache

Problem: High Storage Costs

Symptom: Unexpected charges for Spaces

Check:

# Calculate total space usage
aws s3 ls s3://maplepress --recursive --human-readable --summarize \
  --endpoint-url https://tor1.digitaloceanspaces.com \
  --profile digitalocean

# Shows: Total Size: X.XX GB

Pricing reference:

  • $5/mo includes 250GB storage + 1TB outbound transfer
  • Additional storage: $0.02/GB per month
  • Additional transfer: $0.01/GB

Optimization tips:

  • Delete old/unused files regularly
  • Use CDN to reduce direct space access
  • Compress images before uploading
  • Set up lifecycle policies to auto-delete old files

Next Steps

✅ You now have:

  • DigitalOcean Space created and configured
  • API keys generated and secured
  • Docker Swarm secrets created
  • Verified access from local machine

Next guide:

  • 05_backend.md - Deploy MaplePress backend
  • Backend will use these Spaces credentials automatically
  • Files uploaded via backend API will be stored in your Space

Space Configuration for Backend:

The backend will use these environment variables (configured in 05_backend.md):

environment:
  - AWS_ACCESS_KEY_FILE=/run/secrets/spaces_access_key
  - AWS_SECRET_KEY_FILE=/run/secrets/spaces_secret_key
  - AWS_ENDPOINT=https://tor1.digitaloceanspaces.com
  - AWS_REGION=tor1
  - AWS_BUCKET_NAME=maplepress

Useful Commands:

# List all files in space
aws s3 ls s3://maplepress --recursive \
  --endpoint-url https://tor1.digitaloceanspaces.com \
  --profile digitalocean

# Get space size
aws s3 ls s3://maplepress --recursive --summarize \
  --endpoint-url https://tor1.digitaloceanspaces.com \
  --profile digitalocean

# Delete test file
aws s3 rm s3://maplepress/test-file.txt \
  --endpoint-url https://tor1.digitaloceanspaces.com \
  --profile digitalocean

# Sync local directory to space
aws s3 sync ./local-folder s3://maplepress/uploads/ \
  --endpoint-url https://tor1.digitaloceanspaces.com \
  --profile digitalocean

Last Updated: January 2025 Maintained By: Infrastructure Team

Changelog:

  • January 2025: Initial DigitalOcean Spaces setup guide for MaplePress production deployment