Initial commit: Open sourcing all of the Maple Open Technologies code.

2025-12-02 14:33:08 -05:00 · 2025-12-02 14:33:08 -05:00 · 755d54a99d
commit 755d54a99d
2010 changed files with 448675 additions and 0 deletions
--- a/cloud/infrastructure/production/setup/00-getting-started.md
+++ b/cloud/infrastructure/production/setup/00-getting-started.md
@ -0,0 +1,612 @@
+# Getting Started with Production Deployment
+
+**Audience**: Junior DevOps Engineers, Infrastructure Team
+**Time to Complete**: 10-15 minutes (one-time setup)
+**Prerequisites**:
+- Basic Linux command line knowledge
+- DigitalOcean account with billing enabled
+- SSH access to your local machine
+
+---
+
+## Overview
+
+This guide prepares your local machine for deploying Maple Open Technologies infrastructure to DigitalOcean **from scratch**. You'll set up your workspace and prepare to create servers (droplets), databases, and networking—all through command-line tools.
+
+**What you'll accomplish:**
+- Set up your local workspace
+- Prepare the `.env` configuration file
+- Understand how to store and use infrastructure details as you create them
+- Get ready to run deployment scripts that create resources on DigitalOcean
+
+**What you WON'T need:** Existing secrets or passwords (you'll generate these as you go)
+
+---
+
+## Table of Contents
+
+1. [Prerequisites Check](#prerequisites-check)
+2. [Setting Up Your Local Workspace](#setting-up-your-local-workspace)
+3. [Understanding the `.env` File](#understanding-the-env-file)
+4. [Next Steps](#next-steps)
+
+---
+
+## Prerequisites Check
+
+Before starting, verify you have:
+
+### 1. DigitalOcean Account
+
+```bash
+# You should be able to log in to DigitalOcean
+# Visit: https://cloud.digitalocean.com/
+```
+
+**Need an account?** Sign up at https://www.digitalocean.com/
+
+### 2. DigitalOcean API Token
+
+You'll need a Personal Access Token to create resources from command line.
+
+**Create one:**
+1. Log into DigitalOcean: https://cloud.digitalocean.com/
+2. Click **API** in left sidebar
+3. Click **Generate New Token**
+4. Name: "Production Deployment"
+5. Scopes: Check **Read** and **Write**
+6. Click **Generate Token**
+7. **COPY THE TOKEN IMMEDIATELY** (you can't see it again)
+
+Save this token somewhere safe - you'll add it to `.env` shortly.
+
+### 3. SSH Key Pair
+
+You need SSH keys to access the servers you'll create.
+
+**Check if you already have keys:**
+```bash
+ls -la ~/.ssh/id_rsa.pub
+# If you see the file, you're good! Skip to next section.
+```
+
+**Don't have keys? Create them:**
+```bash
+# Generate new SSH key pair
+ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
+
+# Press Enter to accept default location
+# Enter a passphrase (optional but recommended)
+
+# Verify creation
+ls -la ~/.ssh/id_rsa.pub
+# Should show the file exists
+```
+
+**Add SSH key to DigitalOcean:**
+
+1. Copy your public key:
+   ```bash
+   cat ~/.ssh/id_rsa.pub
+   # Copy the entire output
+   ```
+
+2. Go to DigitalOcean: https://cloud.digitalocean.com/
+3. Click **Settings** → **Security**
+4. Click **Add SSH Key**
+5. Paste your public key
+6. Name it: "My Local Machine"
+7. Click **Add SSH Key**
+
+### 4. Command Line Tools
+
+Verify you have these installed:
+
+```bash
+# Check git
+git --version
+# Should show: git version 2.x.x
+
+# Check ssh
+ssh -V
+# Should show: OpenSSH_x.x
+
+# Check curl
+curl --version
+# Should show: curl 7.x.x or 8.x.x
+```
+
+**Missing tools?** Install them:
+- **macOS**: Tools should be pre-installed or install via `brew install git`
+- **Linux**: `sudo apt install git curl openssh-client` (Ubuntu/Debian)
+- **Windows**: Use WSL2 (Windows Subsystem for Linux)
+
+---
+
+## Setting Up Your Local Workspace
+
+### Step 1: Clone the Repository
+
+```bash
+# Navigate to where you keep code projects
+cd ~/Projects  # or wherever you prefer
+
+# Clone the monorepo
+git clone https://codeberg.org/mapleopentech/monorepo.git
+
+# Navigate to infrastructure directory
+cd monorepo/cloud/infrastructure/production
+
+# Verify you're in the right place
+pwd
+# Should show: /Users/yourname/Projects/monorepo/cloud/infrastructure/production
+```
+
+### Step 2: Create Your `.env` File from Template
+
+The repository includes a `.env.template` file with all configuration variables you'll need. Your actual `.env` file (with real values) is gitignored and will never be committed to the repository.
+
+```bash
+# Copy the template to create your .env file
+cp .env.template .env
+
+# The .env file is automatically gitignored (safe from accidental commits)
+
+# Verify it was created
+ls -la .env
+# Should show: -rw-r--r-- ... .env
+
+# Also verify .env is gitignored
+git check-ignore -v .env
+# Should show: .gitignore:2:.env    .env
+```
+
+**What's the difference?**
+- `.env.template` = Safe to commit, contains `CHANGEME` placeholders
+- `.env` = Your private file, contains real IPs/passwords/tokens, NEVER commit!
+
+### Step 3: Set Secure File Permissions
+
+**Important**: This file will contain sensitive information.
+
+```bash
+# Make it readable/writable only by you
+chmod 600 .env
+
+# Verify permissions changed
+ls -la .env
+# Should show: -rw------- 1 youruser youruser ... .env
+```
+
+### Step 4: Add Your DigitalOcean API Token
+
+```bash
+# Open .env file in your editor
+nano .env
+# Or: vim .env
+# Or: code .env
+```
+
+**Find this line:**
+```bash
+DIGITALOCEAN_TOKEN=CHANGEME
+```
+
+**Replace with your token:**
+```bash
+DIGITALOCEAN_TOKEN=dop_v1_abc123xyz789yourtoken
+```
+
+**Save and close** the file (in nano: `Ctrl+X`, then `Y`, then `Enter`)
+
+### Step 5: Verify Gitignore Protection
+
+**Critical**: Verify `.env` won't be committed to Git:
+
+```bash
+# Check if .env is ignored
+git check-ignore -v .env
+
+# Expected output:
+# .gitignore:XX:.env    .env
+
+# Also verify git status doesn't show .env
+git status
+
+# .env should NOT appear in untracked files
+```
+
+✅ If `.env` appears in git status, STOP and check your `.gitignore` file
+
+---
+
+## Understanding the `.env` File
+
+### What is the `.env` File For?
+
+The `.env` file is your **infrastructure notebook**. As you create resources (servers, databases, etc.) on DigitalOcean, you'll record important details here:
+
+- IP addresses of servers you create
+- Passwords you generate
+- API keys and tokens
+- Configuration values
+
+**Think of it like a worksheet**: You start with `CHANGEME` placeholders and fill them in as you build your infrastructure.
+
+### What is "source .env"?
+
+**Simple explanation**: The `source` command reads your `.env` file and loads all values into your current terminal session so deployment scripts can use them.
+
+**Analogy**: It's like loading ingredients onto your kitchen counter before cooking. The `.env` file is your pantry (storage), and `source` puts everything on the counter (active memory).
+
+**Example**:
+```bash
+# BEFORE running "source .env"
+echo $DIGITALOCEAN_TOKEN
+# Output: (blank - doesn't exist yet)
+
+# AFTER running "source .env"
+source .env
+echo $DIGITALOCEAN_TOKEN
+# Output: dop_v1_abc123... (the value from your .env file)
+```
+
+**Important**: You need to run `source .env` in EVERY new terminal window before running deployment commands.
+
+### The `.env` File Structure
+
+Your `.env` file looks like this:
+
+```bash
+# Each line is: VARIABLE_NAME=value
+DIGITALOCEAN_TOKEN=dop_v1_abc123xyz
+CASSANDRA_NODE1_IP=CHANGEME
+CASSANDRA_ADMIN_PASSWORD=CHANGEME
+
+# Lines starting with # are comments (ignored)
+# Blank lines are also ignored
+```
+
+**Initial state**: Most values are `CHANGEME`
+**As you deploy**: You'll replace `CHANGEME` with actual values
+**Final state**: All `CHANGEME` values replaced with real infrastructure details
+
+### How You'll Use It During Deployment
+
+Here's the workflow you'll follow in the next guides:
+
+1. **Create a resource** (e.g., create a database server on DigitalOcean)
+2. **Note important details** (e.g., IP address: 10.137.0.11, password: abc123)
+3. **Update `.env` file** (replace `CASSANDRA_NODE1_IP=CHANGEME` with `CASSANDRA_NODE1_IP=10.137.0.11`)
+4. **Load the values** (run `source .env`)
+5. **Run next deployment script** (which uses those values)
+
+### Using Environment Variables
+
+Every time you start a new terminal for deployment work:
+
+```bash
+# Step 1: Go to infrastructure directory
+cd ~/monorepo/cloud/infrastructure/production
+
+# Step 2: Load all values from .env into this terminal session
+source .env
+
+# Step 3: Verify it worked (check one variable)
+echo "DigitalOcean Token: ${DIGITALOCEAN_TOKEN:0:15}..."
+# Should show: DigitalOcean Token: dop_v1_abc123...
+```
+
+### Quick Verification
+
+Test that your DigitalOcean token loaded correctly:
+
+```bash
+# Make sure you ran "source .env" first!
+
+# Check the token (shows first 20 characters only)
+echo "DIGITALOCEAN_TOKEN: ${DIGITALOCEAN_TOKEN:0:20}..."
+
+# Expected output:
+# DIGITALOCEAN_TOKEN: dop_v1_abc123xyz789...
+```
+
+**If you see blank output:**
+1. Did you run `source .env`? (run it now)
+2. Did you add your token to `.env`? (check Step 4 above)
+3. Did you save the file after editing?
+
+### Variable Naming Convention
+
+We use consistent prefixes to organize variables:
+
+| Prefix | Purpose | Example |
+|--------|---------|---------|
+| `CASSANDRA_*` | Cassandra database | `CASSANDRA_NODE1_IP` |
+| `REDIS_*` | Redis cache | `REDIS_PASSWORD` |
+| `AWS_*` | Cloud storage | `AWS_ACCESS_KEY_ID` |
+| `BACKEND_*` | Application backend | `BACKEND_ADMIN_HMAC_SECRET` |
+| `*_MAILGUN_*` | Email services | `MAPLEFILE_MAILGUN_API_KEY` |
+
+---
+
+## Common Mistakes to Avoid
+
+### ❌ Mistake 1: Committing .env File
+
+```bash
+# NEVER DO THIS!
+git add .env
+git commit -m "Add environment variables"
+
+# Always check before committing
+git status
+git diff --cached
+```
+
+### ❌ Mistake 2: Forgetting to Load Variables
+
+**Symptom**: Scripts fail with errors like "variable not set" or blank values
+
+```bash
+# ❌ Wrong - running script without loading .env first
+./scripts/create-droplet.sh
+# Error: DIGITALOCEAN_TOKEN: variable not set
+
+# ✅ Correct - load .env first, then run script
+source .env
+./scripts/create-droplet.sh
+
+# ✅ Also correct - load and run in one line
+(source .env && ./scripts/create-droplet.sh)
+```
+
+**Remember**: Each new terminal needs `source .env` run again!
+
+### ❌ Mistake 3: Using Wrong Permissions
+
+```bash
+# Too permissive - others can read your secrets!
+chmod 644 .env  # ❌ Wrong
+
+# Correct - only you can read/write
+chmod 600 .env  # ✅ Correct
+```
+
+### ❌ Mistake 4: Leaving CHANGEME Values
+
+```bash
+# ❌ Wrong - still has placeholder
+CASSANDRA_NODE1_IP=CHANGEME
+
+# ✅ Correct - replaced with actual value after creating server
+CASSANDRA_NODE1_IP=10.137.0.11
+```
+
+---
+
+## Troubleshooting
+
+### Problem: "Permission denied" when reading .env
+
+**Cause**: File permissions too restrictive or wrong owner
+
+**Solution**:
+```bash
+# Check current permissions and owner
+ls -la .env
+
+# Fix permissions
+chmod 600 .env
+
+# If you're not the owner, fix ownership
+sudo chown $(whoami):$(whoami) .env
+```
+
+### Problem: Variables not loading
+
+**Symptoms**: Scripts fail with "variable not set" errors or echo commands show blank
+
+**Solution - Check each step**:
+
+```bash
+# Step 1: Verify .env file exists in current directory
+ls -la .env
+# Should show: -rw------- 1 youruser youruser ... .env
+
+# Step 2: Check it has content (not empty)
+head .env
+# Should show lines like: DIGITALOCEAN_TOKEN=dop_v1_abc123...
+
+# Step 3: Load variables into current terminal
+source .env
+# (no output is normal - silence means success)
+
+# Step 4: Verify loading worked by printing a variable
+echo "DIGITALOCEAN_TOKEN is: ${DIGITALOCEAN_TOKEN:0:20}..."
+# Should print: DIGITALOCEAN_TOKEN is: dop_v1_abc123xyz789...
+# NOT: DIGITALOCEAN_TOKEN is: (blank)
+```
+
+**Still not working?** Check these:
+- Are you in the correct directory? Run `pwd` to verify
+- Is the `.env` file formatted correctly? No spaces around `=` sign
+- Did you save the file after editing?
+- Did you replace `CHANGEME` with actual values?
+
+### Problem: Git showing .env file
+
+**Symptoms**: `git status` shows `.env` as untracked or modified
+
+**Solution**:
+```bash
+# Verify gitignore is working
+git check-ignore -v .env
+
+# If not ignored, check .gitignore exists
+cat .gitignore | grep "\.env"
+
+# If needed, manually add to gitignore
+echo ".env" >> .gitignore
+```
+
+### Problem: Accidentally committed secrets
+
+**⚠️ CRITICAL - Act immediately!**
+
+**If not yet pushed**:
+```bash
+# Remove from staging
+git reset HEAD .env
+
+# Or undo last commit
+git reset --soft HEAD~1
+```
+
+**If already pushed**:
+1. **DO NOT PANIC** - but act quickly
+2. Immediately contact team lead
+3. All secrets in that file must be rotated (changed)
+4. Team lead will help remove from Git history
+
+---
+
+## Quick Reference Commands
+
+### Daily Workflow (Copy-Paste Template)
+
+Every time you open a new terminal for deployment work, run these commands in order:
+
+```bash
+# Step 1: Go to the infrastructure directory
+cd ~/monorepo/cloud/infrastructure/production
+
+# Step 2: Load configuration into this terminal session
+source .env
+
+# Step 3: Verify token loaded correctly
+echo "Token loaded: ${DIGITALOCEAN_TOKEN:0:15}..."
+
+# Step 4: Now you can run deployment commands
+# (You'll use these in the next guides)
+```
+
+**Why these steps?**
+- Step 1: Ensures you're in the right folder where `.env` exists
+- Step 2: Loads your DigitalOcean token and other config values
+- Step 3: Confirms everything loaded correctly
+- Step 4: Ready to create infrastructure!
+
+### One-time Setup Summary
+
+```bash
+# Clone repository
+git clone https://codeberg.org/mapleopentech/monorepo.git
+cd monorepo/cloud/infrastructure/production
+
+# Create .env file
+cp .env.template .env
+
+# Add your DigitalOcean token
+nano .env
+
+# Set permissions
+chmod 600 .env
+
+# Verify gitignored
+git check-ignore -v .env
+
+# Load and verify
+source .env
+echo "Token: ${DIGITALOCEAN_TOKEN:0:15}..."
+```
+
+---
+
+## Next Steps
+
+✅ **You've completed:**
+- Local workspace setup
+- `.env` file creation
+- DigitalOcean API token configuration
+- Understanding of how to use environment variables
+
+**Next, you'll create infrastructure on DigitalOcean:**
+
+1. **[Initialize Docker Swarm](01_init_docker_swarm.md)** - Create Docker Swarm cluster
+2. **[Deploy Cassandra](02_cassandra.md)** - Set up Cassandra database cluster
+3. **[Deploy Redis](03_redis.md)** - Set up Redis cache server
+4. **[Deploy Meilisearch](04_meilisearch.md)** - Set up Meilisearch search engine
+5. **[Configure Spaces](04.5_spaces.md)** - Set up DigitalOcean Spaces object storage
+6. **[Deploy Backend](05_maplepress_backend.md)** - Deploy backend application
+7. **[Setup Caddy](06_maplepress_caddy.md)** - Configure automatic SSL/TLS with Caddy
+
+---
+
+## Important Notes
+
+### You're Building From Scratch
+
+- **No existing infrastructure**: You'll create everything step by step
+- **Generate secrets as needed**: Each guide will tell you when to create passwords/keys
+- **Update `.env` as you go**: After creating each resource, add details to `.env`
+- **Keep notes**: Write down IPs, passwords as you create them
+
+### The `.env` File Will Grow
+
+**Right now:** Only has `DIGITALOCEAN_TOKEN`
+
+**After creating droplets:** Will have server IP addresses
+
+**After setting up databases:** Will have passwords and connection strings
+
+**At the end:** Will have all infrastructure details documented
+
+---
+
+## Security Reminders
+
+🔒 **Always**:
+- Verify `.env` is gitignored (check this NOW: `git check-ignore -v .env`)
+- Use `chmod 600` for `.env` files
+- Run `source .env` before running deployment scripts
+- Keep `.env` file backed up securely (encrypted backup)
+
+🚫 **Never**:
+- Commit `.env` files to Git
+- Share `.env` via email or Slack
+- Use permissive file permissions (644, 777)
+- Leave `CHANGEME` values in production
+
+---
+
+## Quick Pre-flight Check
+
+Before continuing to the next guide, verify:
+
+```bash
+# 1. You're in the right directory
+pwd
+# Should show: .../monorepo/cloud/infrastructure/production
+
+# 2. .env file exists with correct permissions
+ls -la .env
+# Should show: -rw------- ... .env
+
+# 3. Your token is loaded
+source .env
+echo "Token: ${DIGITALOCEAN_TOKEN:0:15}..."
+# Should show: Token: dop_v1_abc123...
+
+# 4. Git won't commit .env
+git check-ignore -v .env
+# Should show: .gitignore:XX:.env    .env
+```
+
+✅ **All checks passed?** Continue to [Create DigitalOcean Droplets](01-create-droplets.md)
+
+---
+
+**Document Version**: 2.0 (From-Scratch Edition)
+**Last Updated**: November 3, 2025
+**Maintained By**: Infrastructure Team