# Setting Up Docker Swarm Cluster

**Audience**: Junior DevOps Engineers, Infrastructure Team
**Time to Complete**: 45-60 minutes
**Prerequisites**:
- Completed [00-getting-started.md](00-getting-started.md)
- DigitalOcean account with API token configured
- SSH key added to your DigitalOcean account

---

## Overview

This guide walks you through creating a **Docker Swarm cluster** with 2 DigitalOcean droplets from scratch. You'll create two Ubuntu 24.04 servers, install Docker on both, and configure them as a swarm with private networking.

**What you'll build:**
- **1 Swarm Manager** (`mapleopentech-swarm-manager-1-prod`) - Controls the cluster
- **1 Swarm Worker** (`mapleopentech-swarm-worker-1-prod`) - Runs containers
- **Private networking** - Nodes communicate via DigitalOcean private IPs

**What is Docker Swarm?**
Docker Swarm is a container orchestration tool that lets you run containers across multiple servers as if they were one system. The manager tells workers what containers to run.

**Naming Convention:**
We use simple sequential numbering for servers. Roles (what each server does) are managed through Docker labels and tags, not hardcoded in hostnames. This makes it easy to repurpose servers as needs change.

---

## Table of Contents

1. [Create DigitalOcean Droplets](#create-digitalocean-droplets)
2. [Configure the Swarm Manager](#configure-the-swarm-manager)
3. [Configure the Swarm Worker](#configure-the-swarm-worker)
4. [Verify the Cluster](#verify-the-cluster)
5. [Update Your .env File](#update-your-env-file)
6. [Troubleshooting](#troubleshooting)

---

## Create DigitalOcean Droplets

### Step 1: Create the Swarm Manager Droplet

Log into DigitalOcean: https://cloud.digitalocean.com/

1. Click **Create** � **Droplets** (top right corner)

2. **Choose Region:**
   - Select **Toronto 1**
   - This tutorial uses Toronto - you'll use the `default-tor1` VPC

3. **Choose an Image:**
   - Select **Ubuntu**
   - Choose **24.04 (LTS) x64**

4. **Choose Size:**
   - **Droplet Type**: Basic
   - **CPU Options**: Regular
   - **Size**: $12/month (2 GB RAM / 1 vCPU / 50 GB SSD)

5. **VPC Network:**
   - Select **default-tor1** (auto-created by DigitalOcean)

6. **Authentication:**
   - Select **SSH Key**
   - Check the SSH key you added earlier

7. **Finalize:**
   - **Hostname**: `mapleopentech-swarm-manager-1-prod`
   - **Tags**: `production`, `swarm`, `manager`
   - **Monitoring**: Enable

8. Click **Create Droplet** button (bottom right)

9. **Wait 1-2 minutes** for droplet to be created

10. **Record IPs:**
    - Copy **Public IP** from droplet list
    - Click droplet → copy **Private IPv4**

**✅ Checkpoint - Update your `.env` file now:**

```bash
# Open .env
nano ~/monorepo/cloud/infrastructure/production/.env

# Add these values (replace with YOUR actual IPs):
SWARM_REGION=tor1
SWARM_VPC_NAME=default-tor1
SWARM_MANAGER_1_HOSTNAME=mapleopentech-swarm-manager-1-prod
SWARM_MANAGER_1_PUBLIC_IP=159.65.123.45     # Your manager's public IP
SWARM_MANAGER_1_PRIVATE_IP=10.116.0.2       # Your manager's private IP
```

### Step 2: Create the Swarm Worker Droplet

Same settings as manager, except:
- **Hostname**: `mapleopentech-swarm-worker-1-prod`
- **Tags**: `production`, `swarm`, `worker`

Click **Create Droplet** and record both IPs.

**✅ Checkpoint - Update your `.env` file:**

```bash
# Add worker info (replace with YOUR actual IPs):
SWARM_WORKER_1_HOSTNAME=mapleopentech-swarm-worker-1-prod
SWARM_WORKER_1_PUBLIC_IP=159.65.123.46      # Your worker's public IP
SWARM_WORKER_1_PRIVATE_IP=10.116.0.3        # Your worker's private IP
```

### Step 3: Verify Private Networking

Check both droplets are in `default-tor1` VPC:
1. **Networking** → **VPC** → Click `default-tor1`
2. Both droplets should be listed
3. Note the subnet (e.g., `10.116.0.0/16`)

Private IPs should start with same prefix (e.g., `10.116.0.2` and `10.116.0.3`).

**✅ Checkpoint - Update your `.env` file:**

```bash
# On your local machine, add:
SWARM_REGION=tor1
SWARM_VPC_NAME=default-tor1
SWARM_VPC_SUBNET=10.116.0.0/16  # Use YOUR actual subnet from VPC dashboard
```

---

## Configure the Swarm Manager

### Step 1: Initial SSH as Root

```bash
# SSH as root (replace with YOUR manager's public IP)
ssh root@159.65.123.45

# Type 'yes' if asked about fingerprint
# You should now see: root@mapleopentech-swarm-manager-1-prod:~#
```

### Step 2: System Updates and Create Admin User

```bash
# Update and upgrade system
apt update && apt upgrade -y

# Install essential packages
apt install -y curl wget apt-transport-https ca-certificates gnupg lsb-release

# Create dedicated Docker admin user
adduser dockeradmin
# Enter a strong password when prompted
# Press Enter for other prompts (or fill them in)

# Add to sudo group
usermod -aG sudo dockeradmin

# Copy SSH keys to new user
rsync --archive --chown=dockeradmin:dockeradmin ~/.ssh /home/dockeradmin
```

**✅ Checkpoint - Update your `.env` file:**

```bash
# On your local machine, add:
DOCKERADMIN_PASSWORD=your_strong_password_here  # The password you just created
```

### Step 3: Secure SSH Configuration

```bash
# Edit SSH config
vi /etc/ssh/sshd_config
```

Find and update these lines (use `Ctrl+W` to search):

```
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
LoginGraceTime 60
```

Save and exit (`Ctrl+X`, `Y`, `Enter`), then restart SSH:

```bash
systemctl restart ssh
```

### Step 4: Reconnect as dockeradmin

```bash
# Exit root session
exit

# On your local machine, reconnect as dockeradmin:
ssh dockeradmin@159.65.123.45  # Your manager's public IP

# You should now see: dockeradmin@mapleopentech-swarm-manager-1-prod:~#
```

### Step 5: Install Docker

```bash
# Install Docker using official convenience script
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Add dockeradmin to docker group (no sudo needed for docker commands)
sudo usermod -aG docker dockeradmin

# Reload groups (or logout/login)
newgrp docker

# Verify Docker is installed
docker --version
# Should show: Docker version 24.x.x or 25.x.x

# Enable Docker to start on boot
sudo systemctl enable docker

# Check Docker is running
sudo systemctl status docker
# Should show: "active (running)" in green
# Press 'q' to exit
```

### Step 6: Configure Firewall for Docker Swarm

Docker Swarm needs specific ports open on the **PRIVATE network**:

```bash
# Install UFW (firewall) if not already installed
sudo apt install ufw -y

# Allow SSH (important - don't lock yourself out!)
sudo ufw allow 22/tcp

# Allow Docker Swarm ports on private network
# Port 2377: Cluster management (TCP)
sudo ufw allow from 10.116.0.0/16 to any port 2377 proto tcp

# Port 7946: Node communication (TCP and UDP)
sudo ufw allow from 10.116.0.0/16 to any port 7946

# Port 4789: Overlay network traffic (UDP)
sudo ufw allow from 10.116.0.0/16 to any port 4789 proto udp

# Enable firewall
sudo ufw --force enable

# Check firewall status
sudo ufw status verbose
```

**IMPORTANT**: Replace `10.116.0.0/16` with your actual private network subnet. If your private IPs are `10.116.x.x`, use `10.116.0.0/16`. If they're `10.108.x.x`, use `10.108.0.0/16`.

### Step 7: Initialize Docker Swarm

```bash
# Get the private IP of this manager droplet
ip addr show eth1 | grep "inet " | awk '{print $2}' | cut -d/ -f1

# Expected output: 10.116.0.2 (or similar)
# This is your PRIVATE IP - copy it

# Initialize swarm using PRIVATE IP
# Replace 10.116.0.2 with YOUR manager's private IP
docker swarm init --advertise-addr 10.116.0.2

# Expected output:
# Swarm initialized: current node (abc123...) is now a manager.
#
# To add a worker to this swarm, run the following command:
#
#     docker swarm join --token SWMTKN-1-xxx... 10.116.0.2:2377
```

### Step 8: Save the Join Token

Copy the join command from the output above.

**✅ Checkpoint - Update your `.env` file:**

```bash
# Extract just the token part and add to .env:
SWARM_JOIN_TOKEN=SWMTKN-1-4abc123xyz789verylongtoken  # Your actual token

# To get token again if needed:
# docker swarm join-token worker -q
```

### Step 9: Verify Manager Status

```bash
# Check swarm status
docker info | grep Swarm
# Should show: Swarm: active

# List nodes in the cluster
docker node ls

# Expected output:
# ID              HOSTNAME        STATUS   AVAILABILITY   MANAGER STATUS
# abc123... *     swarm-manager   Ready    Active         Leader
```

 **Success!** The manager is now running. Keep this SSH session open or note the join command.

---

## Configure the Swarm Worker

### Step 1: Initial SSH as Root

**Open a NEW terminal window** on your local machine (keep manager terminal open):

```bash
# Replace with YOUR swarm worker's PUBLIC IP
ssh root@159.65.123.46

# Type 'yes' if asked about fingerprint
# You should now see: root@mapleopentech-swarm-worker-1-prod:~#
```

### Step 2: System Updates and Create Admin User

```bash
# Update and upgrade system
apt update && apt upgrade -y

# Install essential packages
apt install -y curl wget apt-transport-https ca-certificates gnupg lsb-release

# Create dedicated Docker admin user
adduser dockeradmin
# Enter a strong password when prompted (use the SAME password as manager)

# Add to sudo group
usermod -aG sudo dockeradmin

# Copy SSH keys to new user
rsync --archive --chown=dockeradmin:dockeradmin ~/.ssh /home/dockeradmin
```

**✅ Checkpoint - Verify `.env` has:**
```bash
DOCKERADMIN_PASSWORD=YourStrongPasswordHere  # Same as manager
```

### Step 3: Secure SSH Configuration

```bash
# Edit SSH configuration
nano /etc/ssh/sshd_config

# Find and update these lines:
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
LoginGraceTime 60

# Save: Ctrl+X, then Y, then Enter

# Restart SSH service
systemctl restart ssh
```

### Step 4: Reconnect as dockeradmin

**Exit current session and reconnect:**

```bash
# Exit root session
exit

# SSH back in as dockeradmin
ssh dockeradmin@159.65.123.46  # Replace with YOUR worker's PUBLIC IP

# You should now see: dockeradmin@mapleopentech-swarm-worker-1-prod:~$
```

### Step 5: Install Docker

```bash
# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Add dockeradmin to docker group
sudo usermod -aG docker dockeradmin

# Reload groups
newgrp docker

# Verify Docker is installed
docker --version
# Should show: Docker version 24.x.x or 25.x.x

# Enable Docker to start on boot
sudo systemctl enable docker

# Check Docker is running
sudo systemctl status docker
# Should show: "active (running)" in green
# Press 'q' to exit
```

### Step 6: Configure Firewall

```bash
# Install UFW
sudo apt install ufw -y

# Allow SSH
sudo ufw allow 22/tcp

# Allow Docker Swarm ports on private network
# (Use YOUR network subnet - e.g., 10.116.0.0/16)
sudo ufw allow from 10.116.0.0/16 to any port 2377 proto tcp
sudo ufw allow from 10.116.0.0/16 to any port 7946
sudo ufw allow from 10.116.0.0/16 to any port 4789 proto udp

# Enable firewall
sudo ufw --force enable

# Check firewall status
sudo ufw status verbose
```

### Step 7: Join the Swarm

**Use the join command you saved from Step 8 of the manager setup:**

```bash
# Paste the ENTIRE command you copied earlier
# Example (use YOUR actual token and manager private IP):
docker swarm join --token SWMTKN-1-4abc123xyz789verylongtoken 10.116.0.2:2377

# Expected output:
# This node joined a swarm as a worker.
```

 **Success!** The worker has joined the swarm.

---

## Verify the Cluster

### Step 1: Check Nodes from Manager

**Go back to your manager terminal** (or SSH back in if you closed it):

```bash
ssh dockeradmin@159.65.123.45  # Your manager's PUBLIC IP
```

Run this command:

```bash
# List all nodes in the swarm
docker node ls

# Expected output (2 nodes):
# ID              HOSTNAME                          STATUS   AVAILABILITY   MANAGER STATUS
# abc123... *     mapleopentech-swarm-manager-1-prod    Ready    Active         Leader
# def456...       mapleopentech-swarm-worker-1-prod     Ready    Active
```

**What you should see:**
-  Both nodes listed
-  Both have STATUS = "Ready"
-  Manager shows "Leader"
-  Worker shows nothing in MANAGER STATUS (this is correct)

### Step 2: Test Private Network Communication

**From the manager**, ping the worker's private IP:

```bash
# Replace 10.116.0.3 with YOUR worker's private IP
ping -c 4 10.116.0.3

# Expected output:
# 64 bytes from 10.116.0.3: icmp_seq=1 ttl=64 time=0.5 ms
# 64 bytes from 10.116.0.3: icmp_seq=2 ttl=64 time=0.3 ms
# ...
# 4 packets transmitted, 4 received, 0% packet loss
```

**From the worker**, ping the manager's private IP:

```bash
# Switch to worker terminal and run:
# Replace 10.116.0.2 with YOUR manager's private IP
ping -c 4 10.116.0.2

# Expected output:
# 64 bytes from 10.116.0.2: icmp_seq=1 ttl=64 time=0.5 ms
# ...
# 4 packets transmitted, 4 received, 0% packet loss
```

 **If pings work:** Private networking is correctly configured!
L **If pings fail:** See [Troubleshooting](#troubleshooting) section below

### Step 3: Deploy a Test Service

Let's verify the swarm actually works by deploying a test container:

**From the manager:**

```bash
# Create a simple nginx web server service
docker service create \
  --name test-web \
  --replicas 2 \
  --publish 8080:80 \
  nginx:latest

# Check service status
docker service ls

# Expected output:
# ID        NAME       MODE        REPLICAS   IMAGE
# xyz123    test-web   replicated  2/2        nginx:latest

# See which nodes are running the containers
docker service ps test-web

# Expected output shows containers on both manager and worker
```

**Test the service:**

```bash
# From your local machine, test the service via manager's PUBLIC IP
curl http://159.65.123.45:8080

# Should show HTML output from nginx
# Example: <!DOCTYPE html> <html> ...

# Also test via worker's PUBLIC IP
curl http://159.65.123.46:8080

# Should also work (swarm routes traffic)
```

**Clean up the test service:**

```bash
# Remove the test service (from manager)
docker service rm test-web

# Verify it's gone
docker service ls
# Should show: (empty)
```

 **If test service worked:** Your Docker Swarm cluster is fully operational!

---

## ✅ Final Checkpoint - Verify Your `.env` File

Your `.env` should now have all swarm configuration. Verify it:

```bash
# On your local machine:
cd ~/monorepo/cloud/infrastructure/production

# Check .env has all variables:
grep "SWARM" .env

# Expected output (with YOUR actual values):
# SWARM_REGION=tor1
# SWARM_VPC_NAME=default-tor1
# SWARM_VPC_SUBNET=10.116.0.0/16
# SWARM_MANAGER_1_HOSTNAME=mapleopentech-swarm-manager-1-prod
# SWARM_MANAGER_1_PUBLIC_IP=159.65.123.45
# SWARM_MANAGER_1_PRIVATE_IP=10.116.0.2
# SWARM_WORKER_1_HOSTNAME=mapleopentech-swarm-worker-1-prod
# SWARM_WORKER_1_PUBLIC_IP=159.65.123.46
# SWARM_WORKER_1_PRIVATE_IP=10.116.0.3
# SWARM_JOIN_TOKEN=SWMTKN-1-...

# Load and test:
source .env
echo "✓ Manager: ${SWARM_MANAGER_1_HOSTNAME}"
echo "✓ Worker:  ${SWARM_WORKER_1_HOSTNAME}"
```

---

## Troubleshooting

### Problem: Worker Cannot Join Swarm

**Symptom**: `docker swarm join` fails with "connection refused" or timeout

**Solutions:**

1. **Check firewall on manager:**
   ```bash
   # On manager:
   sudo ufw status verbose

   # Should show rules allowing port 2377 from private network
   # If missing, add it:
   sudo ufw allow from 10.116.0.0/16 to any port 2377 proto tcp
   ```

2. **Verify you're using PRIVATE IP in join command:**
   ```bash
   # Join command should use PRIVATE IP (10.x.x.x), not PUBLIC IP
   # WRONG: docker swarm join --token ... 159.65.123.45:2377
   # RIGHT: docker swarm join --token ... 10.116.0.2:2377
   ```

3. **Check both nodes are in same VPC:**

   **From DigitalOcean dashboard:**
   - Go to **Networking** → **VPC**
   - Click on your VPC (e.g., `default-tor1`)
   - Both droplets should be listed as members

   **From command line (on each node):**
   ```bash
   # On manager:
   ip addr show eth1 | grep "inet "
   # Should show: 10.116.0.2/16

   # On worker:
   ip addr show eth1 | grep "inet "
   # Should show: 10.116.0.3/16 (same 10.116 prefix)

   # If prefix is different, they're in different regions/VPCs!
   ```

### Problem: Nodes Cannot Ping Each Other

**Symptom**: `ping` command fails between nodes

**Solutions:**

1. **Check firewall allows ICMP (ping):**
   ```bash
   # On both nodes:
   sudo ufw allow from 10.116.0.0/16
   ```

2. **Verify private IPs are correct:**
   ```bash
   # On each node, check private IP:
   ip addr show eth1

   # Should show inet 10.x.x.x
   # If you only see eth0, you don't have private networking enabled
   ```

3. **Check DigitalOcean VPC settings:**
   - Go to DigitalOcean dashboard
   - Click **Networking** → **VPC**
   - Click on your VPC (e.g., `default-tor1`)
   - Verify both droplets are listed as members
   - If not, you created them in different regions - delete and recreate in the same region
   - **Remember:** Each region has its own default VPC (Toronto = `default-tor1`, NYC = `default-nyc1`, etc.)

### Problem: Docker Not Installed

**Symptom**: `docker: command not found`

**Solution:**

```bash
# Verify Docker installation
which docker
# Should show: /usr/bin/docker

# If not found, reinstall:
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo systemctl start docker
sudo systemctl enable docker
```

### Problem: Lost Join Token

**Solution:**

```bash
# On manager, regenerate token:
docker swarm join-token worker

# Copy the full command shown in output
```

### Problem: Wrong Node Joined as Manager

**Symptom**: Both nodes show as managers

**Solution:**

```bash
# On the node that should be a worker:
docker swarm leave --force

# Then re-join with worker token (from manager):
docker swarm join-token worker
# Copy and run the command shown
```

### Problem: Firewall Locked You Out

**Symptom**: Cannot SSH after enabling UFW

**Solution:**

- Use DigitalOcean console:
  1. Go to droplet in DigitalOcean dashboard
  2. Click **Access** � **Launch Droplet Console**
  3. Log in as root
  4. Fix firewall:
     ```bash
     ufw allow 22/tcp
     ufw reload
     ```

---

## Quick Reference Commands

### Check Swarm Status (on manager)

```bash
# List all nodes
docker node ls

# Show detailed node info
docker node inspect mapleopentech-swarm-worker-1-prod

# Check swarm status
docker info | grep -A 5 Swarm
```

### Add More Workers (in future)

```bash
# On manager, get join token:
docker swarm join-token worker

# Copy output, then on new worker (e.g., worker-2):
# Create droplet with hostname: mapleopentech-swarm-worker-2-prod
# Install Docker, then paste the join command
```

### Remove a Worker

```bash
# On worker:
docker swarm leave

# On manager (replace with actual hostname):
docker node rm mapleopentech-swarm-worker-1-prod
```

### View Service Logs

```bash
# On manager:
docker service logs <service-name>
```

---

## Security Best Practices

### = Recommendations

1. **Use SSH keys only** - Disable password authentication:
   ```bash
   # On both nodes:
   nano /etc/ssh/sshd_config
   # Set: PasswordAuthentication no
   systemctl restart sshd
   ```

2. **Enable automatic security updates:**
   ```bash
   # On both nodes:
   apt install unattended-upgrades -y
   dpkg-reconfigure -plow unattended-upgrades
   ```

3. **Limit SSH to specific IPs** (if you have static IP):
   ```bash
   # On both nodes:
   ufw delete allow 22/tcp
   ufw allow from YOUR.HOME.IP.ADDRESS to any port 22 proto tcp
   ```

4. **Regular backups** - Enable DigitalOcean droplet backups (Settings � Backups)

5. **Monitor logs:**
   ```bash
   # On both nodes:
   journalctl -u docker -f
   ```

---

## Next Steps

 **You've completed:**
- Created 2 DigitalOcean droplets (Ubuntu 24.04)
- Installed Docker on both
- Configured Docker Swarm with private networking
- Verified cluster connectivity
- Updated `.env` file with infrastructure details

**Next:**
- **[Deploy Cassandra](02_cassandra.md)** - Set up Cassandra database cluster on the swarm
- **[Deploy Redis](03_redis.md)** - Set up Redis cache server
- **[Deploy Meilisearch](04_meilisearch.md)** - Set up Meilisearch search engine

---

## Summary of What You Built
```
┌──────────────────────────────────────────────────────────────────┐
│                      DigitalOcean Cloud                           │
│                                                                   │
│  ┌────────────────────────────┐   ┌─────────────────────────┐  │
│  │ mapleopentech-swarm-manager-1  │   │ mapleopentech-swarm-worker-1│  │
│  │ -prod (Leader)             │◄─►│ -prod (Worker)          │  │
│  │                            │   │                         │  │
│  │ Public:  159.65.123.45     │   │ Public:  159.65.123.46  │  │
│  │ Private: 10.116.0.2        │   │ Private: 10.116.0.3     │  │
│  └────────────────────────────┘   └─────────────────────────┘  │
│             │                               │                    │
│             └─────────────VPC───────────────┘                   │
│                  (Private Network)                               │
│                                                                   │
│  Future: Add mapleopentech-swarm-worker-2-prod, worker-3-prod, etc. │
│                                                                   │
└──────────────────────────────────────────────────────────────────┘

```
---

**Document Version**: 1.0 (From-Scratch Edition)
**Last Updated**: November 3, 2025
**Maintained By**: Infrastructure Team
