# Shared Infrastructure Pattern Guide > **A comprehensive guide to implementing a shared development infrastructure for microservices** > > This document explains an architectural pattern where multiple backend services, frontends, and applications share a common local development infrastructure. Use this as a blueprint for setting up similar environments in your own projects. --- ## πŸ“š Table of Contents 1. [What is the Shared Infrastructure Pattern?](#what-is-the-shared-infrastructure-pattern) 2. [Why Use This Pattern?](#why-use-this-pattern) 3. [Architecture Overview](#architecture-overview) 4. [Directory Structure](#directory-structure) 5. [Core Concepts](#core-concepts) 6. [Step-by-Step Implementation](#step-by-step-implementation) 7. [Connecting Applications](#connecting-applications) 8. [Development Workflow](#development-workflow) 9. [Best Practices](#best-practices) 10. [Troubleshooting Common Issues](#troubleshooting-common-issues) --- ## What is the Shared Infrastructure Pattern? The **Shared Infrastructure Pattern** separates your development infrastructure (databases, cache, storage, etc.) from your application code. Instead of each project running its own database containers, all projects share a single set of infrastructure services. **Traditional Approach:** ``` project-a/ β”œβ”€β”€ docker-compose.yml (includes database, cache, app) └── app code project-b/ β”œβ”€β”€ docker-compose.yml (includes database, cache, app) └── app code ``` *Problem: Each project starts its own infrastructure (slow, resource-heavy)* **Shared Infrastructure Approach:** ``` infrastructure/ β”œβ”€β”€ docker-compose.dev.yml (database, cache, etc.) └── task commands project-a/ β”œβ”€β”€ docker-compose.dev.yml (app only) └── app code project-b/ β”œβ”€β”€ docker-compose.dev.yml (app only) └── app code ``` *Solution: Infrastructure starts once, apps restart quickly* --- ## Why Use This Pattern? ### **Benefits** βœ… **Faster Development Iterations** - Infrastructure starts once (2-3 minutes for complex databases) - Apps restart in seconds - No waiting for databases during code changes βœ… **Production-Like Environment** - Mimics real microservices architecture - Services communicate via networks (not localhost) - Learn proper service separation βœ… **Resource Efficiency** - One database cluster instead of multiple - Shared cache, storage, and search services - Lower memory and CPU usage βœ… **Consistency Across Team** - All developers use identical infrastructure - Same configuration everywhere - Easier onboarding for new team members βœ… **Simplified Management** - Start/stop infrastructure independently - Easier to upgrade shared services - Centralized configuration ### **Trade-offs** ⚠️ **Slightly More Complex** - One extra terminal/directory to manage - Initial setup more involved than single docker-compose - Requires understanding of Docker networks **Decision Point:** Use this pattern when you have 2+ applications sharing infrastructure. For single apps, a monolithic docker-compose is simpler. --- ## Architecture Overview ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Development Machine β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ infrastructure/development/ β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ β”‚ β”‚ Docker Network: shared-dev-network β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ Database β”‚ β”‚ Cache β”‚ β”‚ Storage β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ (Postgresβ”‚ β”‚(Redis) β”‚ β”‚ (S3/ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ MySQL, β”‚ β”‚ β”‚ β”‚ MinIO) β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ Cassandraβ”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ Search β”‚ β”‚ Queue β”‚ β”‚ Others β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚(Elastic, β”‚ β”‚(Rabbit,β”‚ β”‚ (monitoring,β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ Meili) β”‚ β”‚ Kafka) β”‚ β”‚ mail, etc.)β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ Applications (connect to shared network) β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ β”‚ β”‚ Backend A β”‚ β”‚ Backend B β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ (port 8000) β”‚ β”‚ (port 8001) β”‚ β”‚ β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ β”‚ β”‚ Frontend A β”‚ β”‚ Frontend B β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ (npm dev) β”‚ β”‚ (npm dev) β”‚ β”‚ β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ### Key Concepts: 1. **Shared Docker Network** - All services join one network - Services discover each other by hostname - Example: Backend connects to `database:5432` or `cache:6379` 2. **Persistent Volumes** - Data survives container restarts - Named volumes (e.g., `myproject-db-dev`, `myproject-cache-dev`) - Only deleted when explicitly requested 3. **Service Isolation** - Each application gets its own database/schema - Each application gets its own cache namespace - Logical separation, physical sharing --- ## Directory Structure ``` your-project-root/ β”œβ”€β”€ infrastructure/ β”‚ β”œβ”€β”€ development/ β”‚ β”‚ β”œβ”€β”€ docker-compose.dev.yml # All infrastructure services β”‚ β”‚ β”œβ”€β”€ Taskfile.yml # Commands (start, stop, etc.) β”‚ β”‚ β”œβ”€β”€ README.md # Setup instructions β”‚ β”‚ β”œβ”€β”€ .env # Infrastructure config (optional) β”‚ β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ database/ β”‚ β”‚ β”‚ └── init-scripts/ # Database initialization β”‚ β”‚ β”‚ β”œβ”€β”€ 01-create-databases.sql β”‚ β”‚ β”‚ └── 02-create-users.sql β”‚ β”‚ β”‚ β”‚ β”‚ β”œβ”€β”€ cache/ β”‚ β”‚ β”‚ └── redis.conf # Cache configuration β”‚ β”‚ β”‚ β”‚ β”‚ └── nginx/ # Reverse proxy configs β”‚ β”‚ └── default.conf β”‚ β”‚ β”‚ └── production/ # Production deployment β”‚ └── ... (out of scope) β”‚ β”œβ”€β”€ services/ # Backend services β”‚ β”œβ”€β”€ service-a/ β”‚ β”‚ β”œβ”€β”€ docker-compose.dev.yml # App only (connects to infra) β”‚ β”‚ β”œβ”€β”€ Taskfile.yml # App commands β”‚ β”‚ β”œβ”€β”€ .env.sample # Environment template β”‚ β”‚ β”œβ”€β”€ .env # Actual config (gitignored) β”‚ β”‚ └── ... (app code) β”‚ β”‚ β”‚ └── service-b/ β”‚ └── ... (similar structure) β”‚ β”œβ”€β”€ web/ # Frontend applications β”‚ β”œβ”€β”€ app-a/ β”‚ β”‚ β”œβ”€β”€ package.json # npm run dev β”‚ β”‚ β”œβ”€β”€ .env.local # Frontend config β”‚ β”‚ └── ... (React/Vue/Svelte) β”‚ β”‚ β”‚ └── app-b/ β”‚ └── ... β”‚ β”œβ”€β”€ Taskfile.yml # Root commands (optional) └── README.md # Project overview ``` **Naming Flexibility:** Use whatever naming suits your project: - `backend/`, `api/`, `services/`, `cloud/` - `frontend/`, `web/`, `client/`, `ui/` - `infra/`, `infrastructure/`, `docker/` --- ## Core Concepts ### 1. Shared Docker Network **Infrastructure creates the network:** ```yaml # infrastructure/development/docker-compose.dev.yml networks: shared-dev-network: name: my-project-dev driver: bridge ``` **Applications join the network:** ```yaml # services/service-a/docker-compose.dev.yml networks: shared-dev-network: external: true # Important: marks it as pre-existing services: app: networks: - shared-dev-network ``` ### 2. Service Discovery by Hostname Services use **hostnames** (not IP addresses) to find each other: ```yaml # infrastructure/development/docker-compose.dev.yml services: database: hostname: database # or postgres, mysql, db, etc. cache: hostname: cache # or redis, memcached, etc. ``` **In your application code:** ```bash # Database connection DATABASE_HOST=database DATABASE_PORT=5432 # Cache connection CACHE_HOST=cache CACHE_PORT=6379 ``` ### 3. Data Persistence with Named Volumes ```yaml # infrastructure/development/docker-compose.dev.yml volumes: db-data: name: myproject-db-dev cache-data: name: myproject-cache-dev services: database: volumes: - db-data:/var/lib/postgresql/data cache: volumes: - cache-data:/data ``` **Data survives:** - Container restarts - Code changes - Docker updates **Data deleted only when:** - You explicitly run cleanup command - You delete volumes manually ### 4. Database Isolation Strategies **Option A: Separate Databases/Schemas** ```sql -- PostgreSQL/MySQL example CREATE DATABASE app_a; CREATE DATABASE app_b; ``` ```yaml # service-a environment: DATABASE_NAME: app_a # service-b environment: DATABASE_NAME: app_b ``` **Option B: Schema/Keyspace Separation** (Cassandra, MongoDB) ```sql -- Cassandra example CREATE KEYSPACE app_a WITH replication = {...}; CREATE KEYSPACE app_b WITH replication = {...}; ``` **Option C: Cache Database Numbers** (Redis) ```bash # Redis supports 16 databases (0-15) # service-a CACHE_DB=0 # service-b CACHE_DB=1 ``` ### 5. Port Management **Infrastructure exposes ports for debugging:** ```yaml services: database: ports: - "5432:5432" # Access from host for debugging cache: ports: - "6379:6379" ``` **Applications expose unique ports:** ```yaml # service-a ports: - "8000:8000" # service-b ports: - "8001:8000" # Different host port, same container port ``` ### 6. Health Checks Ensure services are ready before applications connect: ```yaml services: database: healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 10s timeout: 5s retries: 5 start_period: 30s cache: healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 10s timeout: 3s retries: 3 ``` --- ## Step-by-Step Implementation ### Phase 1: Create Infrastructure #### Step 1.1: Create Directory Structure ```bash mkdir -p infrastructure/development/{database,cache,storage}/init-scripts cd infrastructure/development ``` #### Step 1.2: Create `docker-compose.dev.yml` **Start with minimal services (add more as needed):** ```yaml version: "3.9" networks: shared-dev-network: name: myproject-dev driver: bridge volumes: db-data: name: myproject-db-dev cache-data: name: myproject-cache-dev services: database: image: postgres:16 # or mysql:8, cassandra:5, etc. container_name: myproject-db-dev hostname: database ports: - "5432:5432" environment: POSTGRES_PASSWORD: devpassword POSTGRES_USER: devuser volumes: - db-data:/var/lib/postgresql/data - ./database/init-scripts:/docker-entrypoint-initdb.d:ro networks: - shared-dev-network healthcheck: test: ["CMD-SHELL", "pg_isready -U devuser"] interval: 10s timeout: 5s retries: 5 restart: unless-stopped cache: image: redis:7-alpine container_name: myproject-cache-dev hostname: cache ports: - "6379:6379" volumes: - cache-data:/data networks: - shared-dev-network command: redis-server --appendonly yes healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 10s timeout: 3s retries: 3 restart: unless-stopped ``` **Add more services as needed:** - Search engines (Elasticsearch, Meilisearch) - Message queues (RabbitMQ, Kafka) - Object storage (MinIO, SeaweedFS) - Monitoring (Prometheus, Grafana) #### Step 1.3: Create Database Initialization Scripts **PostgreSQL example:** `database/init-scripts/01-create-databases.sql` ```sql -- Create databases for each application CREATE DATABASE service_a; CREATE DATABASE service_b; -- Create users if needed CREATE USER app_a_user WITH PASSWORD 'devpassword'; GRANT ALL PRIVILEGES ON DATABASE service_a TO app_a_user; CREATE USER app_b_user WITH PASSWORD 'devpassword'; GRANT ALL PRIVILEGES ON DATABASE service_b TO app_b_user; ``` **MySQL example:** ```sql CREATE DATABASE IF NOT EXISTS service_a; CREATE DATABASE IF NOT EXISTS service_b; CREATE USER 'app_a_user'@'%' IDENTIFIED BY 'devpassword'; GRANT ALL PRIVILEGES ON service_a.* TO 'app_a_user'@'%'; CREATE USER 'app_b_user'@'%' IDENTIFIED BY 'devpassword'; GRANT ALL PRIVILEGES ON service_b.* TO 'app_b_user'@'%'; FLUSH PRIVILEGES; ``` #### Step 1.4: Create `Taskfile.yml` ```yaml version: "3" vars: DOCKER_COMPOSE_CMD: sh: | if command -v docker-compose >/dev/null 2>&1; then echo "docker-compose" elif docker compose version >/dev/null 2>&1; then echo "docker compose" else echo "docker-compose" fi tasks: dev:start: desc: Start all infrastructure services cmds: - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml up -d" - echo "⏳ Waiting for services..." - task: dev:wait - echo "βœ… Infrastructure ready!" - task: dev:status dev:wait: desc: Wait for services to be healthy silent: true cmds: - | echo "Waiting for database..." for i in {1..30}; do if docker exec myproject-db-dev pg_isready -U devuser >/dev/null 2>&1; then echo "βœ… Database ready" break fi sleep 2 done - | echo "Waiting for cache..." for i in {1..10}; do if docker exec myproject-cache-dev redis-cli ping >/dev/null 2>&1; then echo "βœ… Cache ready" break fi sleep 1 done dev:status: desc: Show status of all services cmds: - docker ps --filter "name=myproject-" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" dev:stop: desc: Stop all services (keeps data) cmds: - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml down" - echo "βœ… Infrastructure stopped (data preserved)" dev:restart: desc: Restart all services cmds: - task: dev:stop - task: dev:start dev:clean: desc: Stop and remove all data (DESTRUCTIVE!) prompt: This will DELETE ALL DATA. Continue? cmds: - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml down -v" - echo "βœ… All data deleted" dev:logs: desc: View logs (usage task dev:logs -- database) cmds: - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml logs -f {{.CLI_ARGS}}" # Database-specific commands db:shell: desc: Open database shell cmds: - docker exec -it myproject-db-dev psql -U devuser cache:shell: desc: Open cache shell cmds: - docker exec -it myproject-cache-dev redis-cli ``` #### Step 1.5: Create `README.md` Document: - What services are included - Prerequisites (Docker, Task) - How to start/stop - Port mappings - Connection examples - Troubleshooting #### Step 1.6: Test Infrastructure ```bash # Start task dev:start # Verify task dev:status # All services should show "healthy" # Test database task db:shell # \l (list databases) # \q (quit) # Test cache task cache:shell # PING # exit # Stop task dev:stop ``` ### Phase 2: Connect First Application #### Step 2.1: Create Application Structure ```bash mkdir -p services/service-a cd services/service-a ``` #### Step 2.2: Create Application `docker-compose.dev.yml` ```yaml networks: shared-dev-network: external: true # Connect to infrastructure network services: app: container_name: service-a-dev build: context: . dockerfile: ./Dockerfile.dev ports: - "8000:8000" env_file: - .env environment: # Database connection (use infrastructure hostname) DATABASE_HOST: database DATABASE_PORT: 5432 DATABASE_NAME: service_a DATABASE_USER: app_a_user DATABASE_PASSWORD: devpassword # Cache connection CACHE_HOST: cache CACHE_PORT: 6379 CACHE_DB: 0 # Application settings APP_PORT: 8000 APP_ENV: development volumes: - ./:/app # Mount source for live reload networks: - shared-dev-network restart: unless-stopped healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/health"] interval: 30s timeout: 5s retries: 3 ``` #### Step 2.3: Create Application `Taskfile.yml` ```yaml version: "3" env: COMPOSE_PROJECT_NAME: service-a vars: DOCKER_COMPOSE_CMD: sh: | if command -v docker-compose >/dev/null 2>&1; then echo "docker-compose" elif docker compose version >/dev/null 2>&1; then echo "docker compose" else echo "docker-compose" fi tasks: dev: desc: Start application (requires infrastructure running) deps: [check-infra] cmds: - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml up --build" check-infra: desc: Verify infrastructure is running silent: true cmds: - | if ! docker network inspect myproject-dev >/dev/null 2>&1; then echo "❌ Infrastructure not running!" echo "Start with: cd ../../infrastructure/development && task dev:start" exit 1 fi echo "βœ… Infrastructure is running" dev:down: desc: Stop application cmds: - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml down" dev:restart: desc: Quick restart cmds: - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml restart" - echo "βœ… Application restarted" dev:logs: desc: View application logs cmds: - "{{.DOCKER_COMPOSE_CMD}} -f docker-compose.dev.yml logs -f" dev:shell: desc: Open shell in container cmds: - docker exec -it service-a-dev sh test: desc: Run tests cmds: - # Your test command (e.g., go test ./..., npm test, pytest) lint: desc: Run linters cmds: - # Your lint command ``` #### Step 2.4: Create `.env.sample` ```bash # Application Configuration APP_ENV=development APP_PORT=8000 # Database (configured in docker-compose.dev.yml) # DATABASE_HOST=database # DATABASE_NAME=service_a # Cache (configured in docker-compose.dev.yml) # CACHE_HOST=cache # CACHE_DB=0 # Application Secrets (override in .env) JWT_SECRET=change-in-production API_KEY=your-api-key-here ``` #### Step 2.5: Test Application ```bash # Ensure infrastructure is running cd ../../infrastructure/development task dev:status # Start application cd ../../services/service-a task dev # Test curl http://localhost:8000/health # Stop task dev:down ``` ### Phase 3: Add More Applications For each additional service: 1. **Create new service directory** 2. **Copy and modify `docker-compose.dev.yml`** - Change container name - Change port (8001, 8002, etc.) - Change database name - Change cache DB number (1, 2, etc.) 3. **Copy and modify `Taskfile.yml`** 4. **Update infrastructure init scripts** (add new database) ### Phase 4: Add Frontend Applications Frontends typically run **outside Docker** for faster hot-reload: ```bash mkdir -p web/frontend-a cd web/frontend-a # Initialize (React example) npm create vite@latest . -- --template react npm install ``` **Configure API connection:** `.env.local` ```bash VITE_API_URL=http://localhost:8000 ``` **Start:** ```bash npm run dev ``` **Alternative: Run frontend in Docker** (slower hot-reload) ```yaml # web/frontend-a/docker-compose.dev.yml services: app: image: node:20-alpine working_dir: /app command: npm run dev ports: - "5173:5173" volumes: - ./:/app environment: - VITE_API_URL=http://localhost:8000 ``` --- ## Connecting Applications ### Service Discovery Use **hostnames** (defined in infrastructure docker-compose): ```yaml # infrastructure: hostname set services: database: hostname: database cache: hostname: cache storage: hostname: storage ``` ```bash # application: use hostnames DATABASE_HOST=database DATABASE_PORT=5432 CACHE_HOST=cache CACHE_PORT=6379 STORAGE_ENDPOINT=http://storage:9000 ``` ### Database Isolation **Per-application databases:** ```sql CREATE DATABASE app_a; CREATE DATABASE app_b; ``` **Per-application schemas:** ```sql CREATE SCHEMA app_a; CREATE SCHEMA app_b; ``` **Per-application cache DBs:** ```bash # Redis DB 0-15 CACHE_DB=0 # app-a CACHE_DB=1 # app-b ``` ### Network Configuration **Infrastructure creates network:** ```yaml networks: shared-dev-network: name: myproject-dev driver: bridge services: database: networks: - shared-dev-network ``` **Applications join network:** ```yaml networks: shared-dev-network: external: true # Pre-existing network services: app: networks: - shared-dev-network ``` --- ## Development Workflow ### Daily Workflow ```bash # Morning: Start infrastructure (once) cd infrastructure/development task dev:start # Takes 1-3 minutes # Start service(s) cd ../../services/service-a task dev # Takes 10-30 seconds # Work on code, restart quickly as needed task dev:restart # Takes 5-10 seconds # Start frontend (separate terminal) cd ../../web/frontend-a npm run dev # Takes 5-10 seconds # End of day (optional) cd ../../infrastructure/development task dev:stop ``` ### Adding Features ```bash # Run database migrations cd services/service-a ./migrate up # Or use your migration tool npm run migrate python manage.py migrate ``` ### Debugging ```bash # Infrastructure logs cd infrastructure/development task dev:logs -- database task dev:logs -- cache # Application logs cd ../../services/service-a task dev:logs # Database shell cd ../../infrastructure/development task db:shell # Cache shell task cache:shell # Application shell cd ../../services/service-a task dev:shell ``` ### Resetting Data ```bash # Reset infrastructure (deletes all data) cd infrastructure/development task dev:clean task dev:start # Reset specific database task db:shell # DROP DATABASE service_a; # CREATE DATABASE service_a; ``` --- ## Best Practices ### 1. Naming Conventions **Be consistent across your project:** ``` Infrastructure: - Network: {project}-dev - Containers: {project}-{service}-dev - Volumes: {project}-{service}-dev Applications: - Containers: {appname}-dev - Ports: 8000, 8001, 8002, ... ``` ### 2. Always Use Health Checks ```yaml healthcheck: test: ["CMD", "curl", "-f", "http://localhost/health"] interval: 30s timeout: 10s retries: 3 start_period: 60s ``` ### 3. Document Everything **Infrastructure README:** - What services are included - How to start/stop - Port mappings - Connection examples - Troubleshooting **Application README:** - Link to infrastructure setup - Application-specific setup - Environment variables - How to run tests ### 4. Use Environment Variables **Infrastructure:** Minimal configuration (most settings in docker-compose) **Applications:** Use `.env` files: - Keep `.env.sample` in version control - Add `.env` to `.gitignore` - Document all variables ### 5. Port Management Reserve port ranges: - **5432:** PostgreSQL - **3306:** MySQL - **9042:** Cassandra - **6379:** Redis - **9200:** Elasticsearch - **8000-8099:** Backend services - **5173-5199:** Frontend dev servers ### 6. Resource Limits (Optional) For limited dev machines: ```yaml services: database: deploy: resources: limits: cpus: '2' memory: 2G ``` ### 7. Startup Dependencies Use health conditions: ```yaml services: app: depends_on: database: condition: service_healthy ``` ### 8. Cleanup Strategy ```bash # Remove unused Docker resources docker system prune # Remove specific project volumes docker volume rm myproject-db-dev myproject-cache-dev ``` --- ## Troubleshooting Common Issues ### Issue 1: Network Not Found **Error:** ``` ERROR: Network shared-dev-network declared as external, but could not be found ``` **Solution:** ```bash # Start infrastructure first cd infrastructure/development task dev:start ``` ### Issue 2: Port Already in Use **Error:** ``` Error: bind: address already in use ``` **Solution:** ```bash # Find process using port lsof -i :5432 # macOS/Linux netstat -ano | findstr :5432 # Windows # Kill process or change port in docker-compose.yml ports: - "5433:5432" # Use different host port ``` ### Issue 3: Service Unhealthy **Error:** ``` service-name Up 2 minutes (unhealthy) ``` **Solution:** ```bash # Check logs task dev:logs -- database # Wait longer (some services are slow to start) # Restart if needed task dev:restart ``` ### Issue 4: Can't Connect to Database **Error:** ``` Connection refused: database:5432 ``` **Solution:** 1. Verify application is on shared network: ```yaml networks: shared-dev-network: external: true services: app: networks: - shared-dev-network ``` 2. Verify infrastructure is running: ```bash docker network inspect myproject-dev ``` ### Issue 5: Data Persists After Container Restart **This is expected!** Data in named volumes persists. **To reset:** ```bash task dev:clean # Deletes volumes task dev:start ``` ### Issue 6: Docker Compose Command Not Found **Error:** ``` docker-compose: command not found ``` **Solution:** The Taskfile auto-detects both `docker-compose` (v1) and `docker compose` (v2). If both fail: ```bash # Install Docker Desktop (includes docker-compose) # Or check: docker compose version ``` ### Issue 7: Permission Denied on Linux **Error:** ``` Permission denied: /var/lib/database ``` **Solution:** ```bash # Add user to docker group sudo usermod -aG docker $USER # Log out and back in # Or run with sudo (not recommended) ``` --- ## Summary The **Shared Infrastructure Pattern** provides: βœ… **Fast iterations** - Restart apps in seconds, not minutes βœ… **Production-like** - Proper microservices architecture βœ… **Resource efficient** - One infrastructure, many apps βœ… **Team consistency** - Everyone uses same setup βœ… **Easy maintenance** - Centralized upgrades **When to use:** - 2+ applications sharing infrastructure - Team of 2+ developers - Microservices architecture - Learning distributed systems **When not to use:** - Single application - Simple monolithic architecture - Very limited resources --- ## Next Steps 1. βœ… **Understand the pattern** - Re-read if needed 2. βœ… **Plan your services** - What infrastructure do you need? 3. βœ… **Implement Phase 1** - Set up shared infrastructure 4. βœ… **Implement Phase 2** - Connect first application 5. βœ… **Test thoroughly** - Ensure communication works 6. βœ… **Document** - Write clear READMEs 7. βœ… **Add more services** - Repeat for additional apps --- ## Additional Resources - **Docker Compose:** https://docs.docker.com/compose/ - **Docker Networks:** https://docs.docker.com/network/ - **Task (Taskfile):** https://taskfile.dev/ - **Docker Best Practices:** https://docs.docker.com/develop/dev-best-practices/ --- **Questions or improvements?** This pattern is used in production by many teams. Adapt it to your needs!