Initial commit: Open sourcing all of the Maple Open Technologies code.

cloud/maplefile-backend/migrations/README.md

@@ -0,0 +1,153 @@
+# Database Migrations
+
+This directory contains Cassandra CQL migrations for the MapleFile backend.
+
+## ⚠️ Prerequisites: Keyspace Must Exist
+
+**IMPORTANT:** Before running migrations, the `maplefile` keyspace must exist in Cassandra.
+
+### Why Keyspaces Are Not in Migrations
+
+Following industry best practices:
+- **Keyspace creation = Infrastructure setup** (DevOps responsibility)
+- **Table migrations = Application changes** (Backend responsibility)
+
+This separation allows infrastructure decisions (replication strategy, topology) to be managed independently from application schema.
+
+### Creating the Keyspace
+
+**Development:**
+```bash
+cd cloud/infrastructure/development
+
+# Find Cassandra container
+export CASSANDRA_CONTAINER=$(docker ps --filter "name=cassandra" -q | head -1)
+
+# Create keyspace
+docker exec -it $CASSANDRA_CONTAINER cqlsh -e "
+CREATE KEYSPACE IF NOT EXISTS maplefile
+WITH replication = {
+    'class': 'SimpleStrategy',
+    'replication_factor': 3
+};"
+```
+
+**Production:**
+See `cloud/infrastructure/production/setup/09_maplefile_backend.md` Step 9.3
+
+## Auto-Migration
+
+Migrations run **automatically on backend startup** when `DATABASE_AUTO_MIGRATE=true` (default).
+
+The backend will:
+1. **Expect the `maplefile` keyspace to exist** (created by DevOps)
+2. Run all pending migrations in order (001, 002, 003, etc.)
+3. Track migration state in Cassandra
+4. Handle dirty migration states automatically
+
+## Migration Files
+
+Migrations use the `golang-migrate` tool format:
+
+- **Up migrations**: `{version}_{description}.up.cql` - Applied when migrating forward
+- **Down migrations**: `{version}_{description}.down.cql` - Applied when rolling back
+
+### Current Migrations
+
+- **001-024** - Table and index creation for sessions, users, files, collections, etc.
+
+### Manual Migration
+
+If you need to run migrations manually:
+
+```bash
+# Run all pending migrations
+./maplefile-backend migrate up
+
+# Rollback last migration
+./maplefile-backend migrate down
+
+# Check current version
+./maplefile-backend migrate version
+
+# Force version (fix dirty state)
+./maplefile-backend migrate force <version>
+```
+
+### Disabling Auto-Migration
+
+Set in `.env`:
+```bash
+DATABASE_AUTO_MIGRATE=false
+```
+
+## Creating New Migrations
+
+1. Create new migration files with incremented version:
+   ```bash
+   touch migrations/025_add_new_table.up.cql
+   touch migrations/025_add_new_table.down.cql
+   ```
+
+2. Write the CQL:
+   - **Up migration**: Create/modify schema
+   - **Down migration**: Reverse the changes
+
+3. Commit both files
+
+4. Restart backend or run `./maplefile-backend migrate up`
+
+## Troubleshooting
+
+### "keyspace does not exist" Error
+
+**Cause**: The `maplefile` keyspace hasn't been created by DevOps/infrastructure setup
+
+**Solution**:
+```bash
+# Development
+cd cloud/infrastructure/development
+export CASSANDRA_CONTAINER=$(docker ps --filter "name=cassandra" -q | head -1)
+docker exec -it $CASSANDRA_CONTAINER cqlsh -e "
+CREATE KEYSPACE IF NOT EXISTS maplefile
+WITH replication = {
+    'class': 'SimpleStrategy',
+    'replication_factor': 3
+};"
+
+# Production
+# See cloud/infrastructure/production/setup/09_maplefile_backend.md Step 9.3
+
+# Then restart backend
+docker service update --force maplefile_backend  # Production
+# Or: task dev:restart  # Development
+```
+
+**Prevention**: Always create the keyspace before first backend deployment
+
+### Dirty Migration State
+
+**Symptom**: Backend won't start, logs show "dirty migration"
+
+**Solution**:
+```bash
+# Force clean state at current version
+./maplefile-backend migrate force <version>
+
+# Then retry
+./maplefile-backend migrate up
+```
+
+### Migration Failed
+
+**Symptom**: Backend crashes during migration
+
+**Solution**:
+1. Check backend logs for specific CQL error
+2. Fix the migration file
+3. Force clean state: `./maplefile-backend migrate force <version-1>`
+4. Restart backend or run `./maplefile-backend migrate up`
+
+---
+
+**Related**: See `pkg/storage/database/cassandradb/migration.go` for implementation