diff --git a/cloud/maplepress-backend/docs/Architecture/SHARED_INFRASTRUCTURE_PATTERN.md b/cloud/maplepress-backend/docs/Architecture/SHARED_INFRASTRUCTURE_PATTERN.md new file mode 100644 index 0000000..0ecd0e9 --- /dev/null +++ b/cloud/maplepress-backend/docs/Architecture/SHARED_INFRASTRUCTURE_PATTERN.md @@ -0,0 +1,1258 @@ +# Shared Infrastructure Pattern Guide + +> **A comprehensive guide to implementing a shared development infrastructure for microservices** +> +> This document explains an architectural pattern where multiple backend services, frontends, and applications share a common local development infrastructure. Use this as a blueprint for setting up similar environments in your own projects. + +--- + +## πŸ“š Table of Contents + +1. [What is the Shared Infrastructure Pattern?](#what-is-the-shared-infrastructure-pattern) +2. [Why Use This Pattern?](#why-use-this-pattern) +3. [Architecture Overview](#architecture-overview) +4. [Directory Structure](#directory-structure) +5. [Core Concepts](#core-concepts) +6. [Step-by-Step Implementation](#step-by-step-implementation) +7. [Connecting Applications](#connecting-applications) +8. [Development Workflow](#development-workflow) +9. [Best Practices](#best-practices) +10. [Troubleshooting Common Issues](#troubleshooting-common-issues) + +--- + +## What is the Shared Infrastructure Pattern? + +The **Shared Infrastructure Pattern** separates your development infrastructure (databases, cache, storage, etc.) from your application code. Instead of each project running its own database containers, all projects share a single set of infrastructure services. + +**Traditional Approach:** +``` +project-a/ + β”œβ”€β”€ docker-compose.yml (includes database, cache, app) + └── app code + +project-b/ + β”œβ”€β”€ docker-compose.yml (includes database, cache, app) + └── app code +``` +*Problem: Each project starts its own infrastructure (slow, resource-heavy)* + +**Shared Infrastructure Approach:** +``` +infrastructure/ + β”œβ”€β”€ docker-compose.dev.yml (database, cache, etc.) + └── task commands + +project-a/ + β”œβ”€β”€ docker-compose.dev.yml (app only) + └── app code + +project-b/ + β”œβ”€β”€ docker-compose.dev.yml (app only) + └── app code +``` +*Solution: Infrastructure starts once, apps restart quickly* + +--- + +## Why Use This Pattern? + +### **Benefits** + +βœ… **Faster Development Iterations** + - Infrastructure starts once (2-3 minutes for complex databases) + - Apps restart in seconds + - No waiting for databases during code changes + +βœ… **Production-Like Environment** + - Mimics real microservices architecture + - Services communicate via networks (not localhost) + - Learn proper service separation + +βœ… **Resource Efficiency** + - One database cluster instead of multiple + - Shared cache, storage, and search services + - Lower memory and CPU usage + +βœ… **Consistency Across Team** + - All developers use identical infrastructure + - Same configuration everywhere + - Easier onboarding for new team members + +βœ… **Simplified Management** + - Start/stop infrastructure independently + - Easier to upgrade shared services + - Centralized configuration + +### **Trade-offs** + +⚠️ **Slightly More Complex** + - One extra terminal/directory to manage + - Initial setup more involved than single docker-compose + - Requires understanding of Docker networks + +**Decision Point:** Use this pattern when you have 2+ applications sharing infrastructure. For single apps, a monolithic docker-compose is simpler. + +--- + +## Architecture Overview + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Development Machine β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ infrastructure/development/ β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ Docker Network: shared-dev-network β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ Database β”‚ β”‚ Cache β”‚ β”‚ Storage β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ (Postgresβ”‚ β”‚(Redis) β”‚ β”‚ (S3/ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ MySQL, β”‚ β”‚ β”‚ β”‚ MinIO) β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ Cassandraβ”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ Search β”‚ β”‚ Queue β”‚ β”‚ Others β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚(Elastic, β”‚ β”‚(Rabbit,β”‚ β”‚ (monitoring,β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ Meili) β”‚ β”‚ Kafka) β”‚ β”‚ mail, etc.)β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Applications (connect to shared network) β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ Backend A β”‚ β”‚ Backend B β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ (port 8000) β”‚ β”‚ (port 8001) β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ +β”‚ β”‚ β”‚ Frontend A β”‚ β”‚ Frontend B β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ (npm dev) β”‚ β”‚ (npm dev) β”‚ β”‚ β”‚ +β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### Key Concepts: + +1. **Shared Docker Network** + - All services join one network + - Services discover each other by hostname + - Example: Backend connects to `database:5432` or `cache:6379` + +2. **Persistent Volumes** + - Data survives container restarts + - Named volumes (e.g., `myproject-db-dev`, `myproject-cache-dev`) + - Only deleted when explicitly requested + +3. **Service Isolation** + - Each application gets its own database/schema + - Each application gets its own cache namespace + - Logical separation, physical sharing + +--- + +## Directory Structure + +``` +your-project-root/ +β”œβ”€β”€ infrastructure/ +β”‚ β”œβ”€β”€ development/ +β”‚ β”‚ β”œβ”€β”€ docker-compose.dev.yml # All infrastructure services +β”‚ β”‚ β”œβ”€β”€ Taskfile.yml # Commands (start, stop, etc.) +β”‚ β”‚ β”œβ”€β”€ README.md # Setup instructions +β”‚ β”‚ β”œβ”€β”€ .env # Infrastructure config (optional) +β”‚ β”‚ β”‚ +β”‚ β”‚ β”œβ”€β”€ database/ +β”‚ β”‚ β”‚ └── init-scripts/ # Database initialization +β”‚ β”‚ β”‚ β”œβ”€β”€ 01-create-databases.sql +β”‚ β”‚ β”‚ └── 02-create-users.sql +β”‚ β”‚ β”‚ +β”‚ β”‚ β”œβ”€β”€ cache/ +β”‚ β”‚ β”‚ └── redis.conf # Cache configuration +β”‚ β”‚ β”‚ +β”‚ β”‚ └── nginx/ # Reverse proxy configs +β”‚ β”‚ └── default.conf +β”‚ β”‚ +β”‚ └── production/ # Production deployment +β”‚ └── ... (out of scope) +β”‚ +β”œβ”€β”€ services/ # Backend services +β”‚ β”œβ”€β”€ service-a/ +β”‚ β”‚ β”œβ”€β”€ docker-compose.dev.yml # App only (connects to infra) +β”‚ β”‚ β”œβ”€β”€ Taskfile.yml # App commands +β”‚ β”‚ β”œβ”€β”€ .env.sample # Environment template +β”‚ β”‚ β”œβ”€β”€ .env # Actual config (gitignored) +β”‚ β”‚ └── ... (app code) +β”‚ β”‚ +β”‚ └── service-b/ +β”‚ └── ... (similar structure) +β”‚ +β”œβ”€β”€ web/ # Frontend applications +β”‚ β”œβ”€β”€ app-a/ +β”‚ β”‚ β”œβ”€β”€ package.json # npm run dev +β”‚ β”‚ β”œβ”€β”€ .env.local # Frontend config +β”‚ β”‚ └── ... (React/Vue/Svelte) +β”‚ β”‚ +β”‚ └── app-b/ +β”‚ └── ... +β”‚ +β”œβ”€β”€ Taskfile.yml # Root commands (optional) +└── README.md # Project overview +``` + +**Naming Flexibility:** Use whatever naming suits your project: +- `backend/`, `api/`, `services/`, `cloud/` +- `frontend/`, `web/`, `client/`, `ui/` +- `infra/`, `infrastructure/`, `docker/` + +--- + +## Core Concepts + +### 1. Shared Docker Network + +**Infrastructure creates the network:** +```yaml +# infrastructure/development/docker-compose.dev.yml +networks: + shared-dev-network: + name: my-project-dev + driver: bridge +``` + +**Applications join the network:** +```yaml +# services/service-a/docker-compose.dev.yml +networks: + shared-dev-network: + external: true # Important: marks it as pre-existing + +services: + app: + networks: + - shared-dev-network +``` + +### 2. Service Discovery by Hostname + +Services use **hostnames** (not IP addresses) to find each other: + +```yaml +# infrastructure/development/docker-compose.dev.yml +services: + database: + hostname: database + # or postgres, mysql, db, etc. + + cache: + hostname: cache + # or redis, memcached, etc. +``` + +**In your application code:** +```bash +# Database connection +DATABASE_HOST=database +DATABASE_PORT=5432 + +# Cache connection +CACHE_HOST=cache +CACHE_PORT=6379 +``` + +### 3. Data Persistence with Named Volumes + +```yaml +# infrastructure/development/docker-compose.dev.yml +volumes: + db-data: + name: myproject-db-dev + cache-data: + name: myproject-cache-dev + +services: + database: + volumes: + - db-data:/var/lib/postgresql/data + + cache: + volumes: + - cache-data:/data +``` + +**Data survives:** +- Container restarts +- Code changes +- Docker updates + +**Data deleted only when:** +- You explicitly run cleanup command +- You delete volumes manually + +### 4. Database Isolation Strategies + +**Option A: Separate Databases/Schemas** +```sql +-- PostgreSQL/MySQL example +CREATE DATABASE app_a; +CREATE DATABASE app_b; +``` + +```yaml +# service-a +environment: + DATABASE_NAME: app_a + +# service-b +environment: + DATABASE_NAME: app_b +``` + +**Option B: Schema/Keyspace Separation** (Cassandra, MongoDB) +```sql +-- Cassandra example +CREATE KEYSPACE app_a WITH replication = {...}; +CREATE KEYSPACE app_b WITH replication = {...}; +``` + +**Option C: Cache Database Numbers** (Redis) +```bash +# Redis supports 16 databases (0-15) +# service-a +CACHE_DB=0 + +# service-b +CACHE_DB=1 +``` + +### 5. Port Management + +**Infrastructure exposes ports for debugging:** +```yaml +services: + database: + ports: + - "5432:5432" # Access from host for debugging + + cache: + ports: + - "6379:6379" +``` + +**Applications expose unique ports:** +```yaml +# service-a +ports: + - "8000:8000" + +# service-b +ports: + - "8001:8000" # Different host port, same container port +``` + +### 6. Health Checks + +Ensure services are ready before applications connect: + +```yaml +services: + database: + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 30s + + cache: + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 3s + retries: 3 +``` + +--- + +## Step-by-Step Implementation + +### Phase 1: Create Infrastructure + +#### Step 1.1: Create Directory Structure + +```bash +mkdir -p infrastructure/development/{database,cache,storage}/init-scripts +cd infrastructure/development +``` + +#### Step 1.2: Create `docker-compose.dev.yml` + +**Start with minimal services (add more as needed):** + +```yaml +version: "3.9" + +networks: + shared-dev-network: + name: myproject-dev + driver: bridge + +volumes: + db-data: + name: myproject-db-dev + cache-data: + name: myproject-cache-dev + +services: + database: + image: postgres:16 # or mysql:8, cassandra:5, etc. + container_name: myproject-db-dev + hostname: database + ports: + - "5432:5432" + environment: + POSTGRES_PASSWORD: devpassword + POSTGRES_USER: devuser + volumes: + - db-data:/var/lib/postgresql/data + - ./database/init-scripts:/docker-entrypoint-initdb.d:ro + networks: + - shared-dev-network + healthcheck: + test: ["CMD-SHELL", "pg_isready -U devuser"] + interval: 10s + timeout: 5s + retries: 5 + restart: unless-stopped + + cache: + image: redis:7-alpine + container_name: myproject-cache-dev + hostname: cache + ports: + - "6379:6379" + volumes: + - cache-data:/data + networks: + - shared-dev-network + command: redis-server --appendonly yes + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 3s + retries: 3 + restart: unless-stopped +``` + +**Add more services as needed:** +- Search engines (Elasticsearch, Meilisearch) +- Message queues (RabbitMQ, Kafka) +- Object storage (MinIO, SeaweedFS) +- Monitoring (Prometheus, Grafana) + +#### Step 1.3: Create Database Initialization Scripts + +**PostgreSQL example:** `database/init-scripts/01-create-databases.sql` +```sql +-- Create databases for each application +CREATE DATABASE service_a; +CREATE DATABASE service_b; + +-- Create users if needed +CREATE USER app_a_user WITH PASSWORD 'devpassword'; +GRANT ALL PRIVILEGES ON DATABASE service_a TO app_a_user; + +CREATE USER app_b_user WITH PASSWORD 'devpassword'; +GRANT ALL PRIVILEGES ON DATABASE service_b TO app_b_user; +``` + +**MySQL example:** +```sql +CREATE DATABASE IF NOT EXISTS service_a; +CREATE DATABASE IF NOT EXISTS service_b; + +CREATE USER 'app_a_user'@'%' IDENTIFIED BY 'devpassword'; +GRANT ALL PRIVILEGES ON service_a.* TO 'app_a_user'@'%'; + +CREATE USER 'app_b_user'@'%' IDENTIFIED BY 'devpassword'; +GRANT ALL PRIVILEGES ON service_b.* TO 'app_b_user'@'%'; + +FLUSH PRIVILEGES; +``` + +#### Step 1.4: Create `Taskfile.yml` + +```yaml +version: "3" + +vars: + DOCKER_COMPOSE_CMD: + sh: | + if command -v docker-compose >/dev/null 2>&1; then + echo "docker-compose" + elif docker compose version >/dev/null 2>&1; then + echo "docker compose" + else + echo "docker-compose" + fi + +tasks: + dev:start: + desc: Start all infrastructure services + cmds: + - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml up -d" + - echo "⏳ Waiting for services..." + - task: dev:wait + - echo "βœ… Infrastructure ready!" + - task: dev:status + + dev:wait: + desc: Wait for services to be healthy + silent: true + cmds: + - | + echo "Waiting for database..." + for i in {1..30}; do + if docker exec myproject-db-dev pg_isready -U devuser >/dev/null 2>&1; then + echo "βœ… Database ready" + break + fi + sleep 2 + done + - | + echo "Waiting for cache..." + for i in {1..10}; do + if docker exec myproject-cache-dev redis-cli ping >/dev/null 2>&1; then + echo "βœ… Cache ready" + break + fi + sleep 1 + done + + dev:status: + desc: Show status of all services + cmds: + - docker ps --filter "name=myproject-" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" + + dev:stop: + desc: Stop all services (keeps data) + cmds: + - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml down" + - echo "βœ… Infrastructure stopped (data preserved)" + + dev:restart: + desc: Restart all services + cmds: + - task: dev:stop + - task: dev:start + + dev:clean: + desc: Stop and remove all data (DESTRUCTIVE!) + prompt: This will DELETE ALL DATA. Continue? + cmds: + - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml down -v" + - echo "βœ… All data deleted" + + dev:logs: + desc: View logs (usage task dev:logs -- database) + cmds: + - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml logs -f {{.CLI_ARGS}}" + + # Database-specific commands + db:shell: + desc: Open database shell + cmds: + - docker exec -it myproject-db-dev psql -U devuser + + cache:shell: + desc: Open cache shell + cmds: + - docker exec -it myproject-cache-dev redis-cli +``` + +#### Step 1.5: Create `README.md` + +Document: +- What services are included +- Prerequisites (Docker, Task) +- How to start/stop +- Port mappings +- Connection examples +- Troubleshooting + +#### Step 1.6: Test Infrastructure + +```bash +# Start +task dev:start + +# Verify +task dev:status +# All services should show "healthy" + +# Test database +task db:shell +# \l (list databases) +# \q (quit) + +# Test cache +task cache:shell +# PING +# exit + +# Stop +task dev:stop +``` + +### Phase 2: Connect First Application + +#### Step 2.1: Create Application Structure + +```bash +mkdir -p services/service-a +cd services/service-a +``` + +#### Step 2.2: Create Application `docker-compose.dev.yml` + +```yaml +networks: + shared-dev-network: + external: true # Connect to infrastructure network + +services: + app: + container_name: service-a-dev + build: + context: . + dockerfile: ./Dockerfile.dev + ports: + - "8000:8000" + env_file: + - .env + environment: + # Database connection (use infrastructure hostname) + DATABASE_HOST: database + DATABASE_PORT: 5432 + DATABASE_NAME: service_a + DATABASE_USER: app_a_user + DATABASE_PASSWORD: devpassword + + # Cache connection + CACHE_HOST: cache + CACHE_PORT: 6379 + CACHE_DB: 0 + + # Application settings + APP_PORT: 8000 + APP_ENV: development + + volumes: + - ./:/app # Mount source for live reload + networks: + - shared-dev-network + restart: unless-stopped + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 5s + retries: 3 +``` + +#### Step 2.3: Create Application `Taskfile.yml` + +```yaml +version: "3" + +env: + COMPOSE_PROJECT_NAME: service-a + +vars: + DOCKER_COMPOSE_CMD: + sh: | + if command -v docker-compose >/dev/null 2>&1; then + echo "docker-compose" + elif docker compose version >/dev/null 2>&1; then + echo "docker compose" + else + echo "docker-compose" + fi + +tasks: + dev: + desc: Start application (requires infrastructure running) + deps: [check-infra] + cmds: + - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml up --build" + + check-infra: + desc: Verify infrastructure is running + silent: true + cmds: + - | + if ! docker network inspect myproject-dev >/dev/null 2>&1; then + echo "❌ Infrastructure not running!" + echo "Start with: cd ../../infrastructure/development && task dev:start" + exit 1 + fi + echo "βœ… Infrastructure is running" + + dev:down: + desc: Stop application + cmds: + - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml down" + + dev:restart: + desc: Quick restart + cmds: + - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml restart" + - echo "βœ… Application restarted" + + dev:logs: + desc: View application logs + cmds: + - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml logs -f" + + dev:shell: + desc: Open shell in container + cmds: + - docker exec -it service-a-dev sh + + test: + desc: Run tests + cmds: + - # Your test command (e.g., go test ./..., npm test, pytest) + + lint: + desc: Run linters + cmds: + - # Your lint command +``` + +#### Step 2.4: Create `.env.sample` + +```bash +# Application Configuration +APP_ENV=development +APP_PORT=8000 + +# Database (configured in docker-compose.dev.yml) +# DATABASE_HOST=database +# DATABASE_NAME=service_a + +# Cache (configured in docker-compose.dev.yml) +# CACHE_HOST=cache +# CACHE_DB=0 + +# Application Secrets (override in .env) +JWT_SECRET=change-in-production +API_KEY=your-api-key-here +``` + +#### Step 2.5: Test Application + +```bash +# Ensure infrastructure is running +cd ../../infrastructure/development +task dev:status + +# Start application +cd ../../services/service-a +task dev + +# Test +curl http://localhost:8000/health + +# Stop +task dev:down +``` + +### Phase 3: Add More Applications + +For each additional service: + +1. **Create new service directory** +2. **Copy and modify `docker-compose.dev.yml`** + - Change container name + - Change port (8001, 8002, etc.) + - Change database name + - Change cache DB number (1, 2, etc.) +3. **Copy and modify `Taskfile.yml`** +4. **Update infrastructure init scripts** (add new database) + +### Phase 4: Add Frontend Applications + +Frontends typically run **outside Docker** for faster hot-reload: + +```bash +mkdir -p web/frontend-a +cd web/frontend-a + +# Initialize (React example) +npm create vite@latest . -- --template react +npm install +``` + +**Configure API connection:** `.env.local` +```bash +VITE_API_URL=http://localhost:8000 +``` + +**Start:** +```bash +npm run dev +``` + +**Alternative: Run frontend in Docker** (slower hot-reload) +```yaml +# web/frontend-a/docker-compose.dev.yml +services: + app: + image: node:20-alpine + working_dir: /app + command: npm run dev + ports: + - "5173:5173" + volumes: + - ./:/app + environment: + - VITE_API_URL=http://localhost:8000 +``` + +--- + +## Connecting Applications + +### Service Discovery + +Use **hostnames** (defined in infrastructure docker-compose): + +```yaml +# infrastructure: hostname set +services: + database: + hostname: database + + cache: + hostname: cache + + storage: + hostname: storage +``` + +```bash +# application: use hostnames +DATABASE_HOST=database +DATABASE_PORT=5432 + +CACHE_HOST=cache +CACHE_PORT=6379 + +STORAGE_ENDPOINT=http://storage:9000 +``` + +### Database Isolation + +**Per-application databases:** +```sql +CREATE DATABASE app_a; +CREATE DATABASE app_b; +``` + +**Per-application schemas:** +```sql +CREATE SCHEMA app_a; +CREATE SCHEMA app_b; +``` + +**Per-application cache DBs:** +```bash +# Redis DB 0-15 +CACHE_DB=0 # app-a +CACHE_DB=1 # app-b +``` + +### Network Configuration + +**Infrastructure creates network:** +```yaml +networks: + shared-dev-network: + name: myproject-dev + driver: bridge + +services: + database: + networks: + - shared-dev-network +``` + +**Applications join network:** +```yaml +networks: + shared-dev-network: + external: true # Pre-existing network + +services: + app: + networks: + - shared-dev-network +``` + +--- + +## Development Workflow + +### Daily Workflow + +```bash +# Morning: Start infrastructure (once) +cd infrastructure/development +task dev:start +# Takes 1-3 minutes + +# Start service(s) +cd ../../services/service-a +task dev +# Takes 10-30 seconds + +# Work on code, restart quickly as needed +task dev:restart +# Takes 5-10 seconds + +# Start frontend (separate terminal) +cd ../../web/frontend-a +npm run dev +# Takes 5-10 seconds + +# End of day (optional) +cd ../../infrastructure/development +task dev:stop +``` + +### Adding Features + +```bash +# Run database migrations +cd services/service-a +./migrate up + +# Or use your migration tool +npm run migrate +python manage.py migrate +``` + +### Debugging + +```bash +# Infrastructure logs +cd infrastructure/development +task dev:logs -- database +task dev:logs -- cache + +# Application logs +cd ../../services/service-a +task dev:logs + +# Database shell +cd ../../infrastructure/development +task db:shell + +# Cache shell +task cache:shell + +# Application shell +cd ../../services/service-a +task dev:shell +``` + +### Resetting Data + +```bash +# Reset infrastructure (deletes all data) +cd infrastructure/development +task dev:clean +task dev:start + +# Reset specific database +task db:shell +# DROP DATABASE service_a; +# CREATE DATABASE service_a; +``` + +--- + +## Best Practices + +### 1. Naming Conventions + +**Be consistent across your project:** + +``` +Infrastructure: +- Network: {project}-dev +- Containers: {project}-{service}-dev +- Volumes: {project}-{service}-dev + +Applications: +- Containers: {appname}-dev +- Ports: 8000, 8001, 8002, ... +``` + +### 2. Always Use Health Checks + +```yaml +healthcheck: + test: ["CMD", "curl", "-f", "http://localhost/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s +``` + +### 3. Document Everything + +**Infrastructure README:** +- What services are included +- How to start/stop +- Port mappings +- Connection examples +- Troubleshooting + +**Application README:** +- Link to infrastructure setup +- Application-specific setup +- Environment variables +- How to run tests + +### 4. Use Environment Variables + +**Infrastructure:** Minimal configuration (most settings in docker-compose) + +**Applications:** Use `.env` files: +- Keep `.env.sample` in version control +- Add `.env` to `.gitignore` +- Document all variables + +### 5. Port Management + +Reserve port ranges: +- **5432:** PostgreSQL +- **3306:** MySQL +- **9042:** Cassandra +- **6379:** Redis +- **9200:** Elasticsearch +- **8000-8099:** Backend services +- **5173-5199:** Frontend dev servers + +### 6. Resource Limits (Optional) + +For limited dev machines: + +```yaml +services: + database: + deploy: + resources: + limits: + cpus: '2' + memory: 2G +``` + +### 7. Startup Dependencies + +Use health conditions: + +```yaml +services: + app: + depends_on: + database: + condition: service_healthy +``` + +### 8. Cleanup Strategy + +```bash +# Remove unused Docker resources +docker system prune + +# Remove specific project volumes +docker volume rm myproject-db-dev myproject-cache-dev +``` + +--- + +## Troubleshooting Common Issues + +### Issue 1: Network Not Found + +**Error:** +``` +ERROR: Network shared-dev-network declared as external, but could not be found +``` + +**Solution:** +```bash +# Start infrastructure first +cd infrastructure/development +task dev:start +``` + +### Issue 2: Port Already in Use + +**Error:** +``` +Error: bind: address already in use +``` + +**Solution:** +```bash +# Find process using port +lsof -i :5432 # macOS/Linux +netstat -ano | findstr :5432 # Windows + +# Kill process or change port in docker-compose.yml +ports: + - "5433:5432" # Use different host port +``` + +### Issue 3: Service Unhealthy + +**Error:** +``` +service-name Up 2 minutes (unhealthy) +``` + +**Solution:** +```bash +# Check logs +task dev:logs -- database + +# Wait longer (some services are slow to start) + +# Restart if needed +task dev:restart +``` + +### Issue 4: Can't Connect to Database + +**Error:** +``` +Connection refused: database:5432 +``` + +**Solution:** + +1. Verify application is on shared network: +```yaml +networks: + shared-dev-network: + external: true + +services: + app: + networks: + - shared-dev-network +``` + +2. Verify infrastructure is running: +```bash +docker network inspect myproject-dev +``` + +### Issue 5: Data Persists After Container Restart + +**This is expected!** Data in named volumes persists. + +**To reset:** +```bash +task dev:clean # Deletes volumes +task dev:start +``` + +### Issue 6: Docker Compose Command Not Found + +**Error:** +``` +docker-compose: command not found +``` + +**Solution:** + +The Taskfile auto-detects both `docker-compose` (v1) and `docker compose` (v2). + +If both fail: +```bash +# Install Docker Desktop (includes docker-compose) +# Or check: docker compose version +``` + +### Issue 7: Permission Denied on Linux + +**Error:** +``` +Permission denied: /var/lib/database +``` + +**Solution:** +```bash +# Add user to docker group +sudo usermod -aG docker $USER +# Log out and back in + +# Or run with sudo (not recommended) +``` + +--- + +## Summary + +The **Shared Infrastructure Pattern** provides: + +βœ… **Fast iterations** - Restart apps in seconds, not minutes +βœ… **Production-like** - Proper microservices architecture +βœ… **Resource efficient** - One infrastructure, many apps +βœ… **Team consistency** - Everyone uses same setup +βœ… **Easy maintenance** - Centralized upgrades + +**When to use:** +- 2+ applications sharing infrastructure +- Team of 2+ developers +- Microservices architecture +- Learning distributed systems + +**When not to use:** +- Single application +- Simple monolithic architecture +- Very limited resources + +--- + +## Next Steps + +1. βœ… **Understand the pattern** - Re-read if needed +2. βœ… **Plan your services** - What infrastructure do you need? +3. βœ… **Implement Phase 1** - Set up shared infrastructure +4. βœ… **Implement Phase 2** - Connect first application +5. βœ… **Test thoroughly** - Ensure communication works +6. βœ… **Document** - Write clear READMEs +7. βœ… **Add more services** - Repeat for additional apps + +--- + +## Additional Resources + +- **Docker Compose:** https://docs.docker.com/compose/ +- **Docker Networks:** https://docs.docker.com/network/ +- **Task (Taskfile):** https://taskfile.dev/ +- **Docker Best Practices:** https://docs.docker.com/develop/dev-best-practices/ + +--- + +**Questions or improvements?** This pattern is used in production by many teams. Adapt it to your needs!