Initial commit: Open sourcing all of the Maple Open Technologies code.

2025-12-02 14:33:08 -05:00 · 2025-12-02 14:33:08 -05:00 · 755d54a99d
commit 755d54a99d
2010 changed files with 448675 additions and 0 deletions
--- a/cloud/maplefile-backend/README.md
+++ b/cloud/maplefile-backend/README.md
@ -0,0 +1,496 @@
+# 🚀 MapleFile Backend
+
+> Secure, end-to-end encrypted file storage backend - Zero-knowledge architecture built with Go.
+
+MapleFile provides military-grade file encryption with client-side E2EE (End-to-End Encryption). Features include collection-based organization, granular sharing permissions, JWT authentication, and S3-compatible object storage. Your files are encrypted on your device before reaching our servers - we never see your data.
+
+## 📋 Prerequisites
+
+**⚠️ Required:** You must have the infrastructure running first.
+
+If you haven't set up the infrastructure yet:
+1. Go to [`../infrastructure/README.md`](../infrastructure/README.md)
+2. Follow the setup instructions
+3. Come back here once infrastructure is running
+
+**Verify infrastructure is healthy:**
+```bash
+cd cloud/infrastructure/development
+task dev:status
+# All services should show (healthy)
+```
+
+## 🏁 Getting Started
+
+### Installation
+
+```bash
+# From the monorepo root:
+cd cloud/maplefile-backend
+
+# Create environment file:
+cp .env.sample .env
+
+# Start the backend:
+task dev
+```
+
+The backend runs at **http://localhost:8000**
+
+### Verify Installation
+
+Open a **new terminal** (leave `task dev` running):
+
+```bash
+curl http://localhost:8000/health
+# Should return: {"status":"healthy","service":"maplefile-backend","di":"Wire"}
+```
+
+> **Note:** Your first terminal shows backend logs. Keep it running and use a second terminal for testing.
+
+## 💻 Developing
+
+### Initial Configuration
+
+**Environment Files:**
+- **`.env.sample`** - Template with defaults (committed to git)
+- **`.env`** - Your local configuration (git-ignored, created from `.env.sample`)
+- Use **only `.env`** for configuration (docker-compose loads this file)
+
+The `.env` file defaults work for Docker development. **Optional:** Change `BACKEND_APP_JWT_SECRET` to a random string (use a password generator).
+
+### Running in Development Mode
+
+```bash
+# Start backend with hot-reload
+task dev
+
+# View logs (in another terminal)
+docker logs -f maplefile-backend-dev
+
+# Stop backend
+task dev:down
+# Or press Ctrl+C in the task dev terminal
+```
+
+**What happens when you run `task dev`:**
+- Docker starts the backend container
+- Auto-migrates database tables
+- Starts HTTP server on port 8000
+- Enables hot-reload (auto-restarts on code changes)
+
+Wait for: `✅ Database migrations completed successfully` in the logs
+
+### Daily Workflow
+
+```bash
+# Morning - check infrastructure (from monorepo root)
+cd cloud/infrastructure/development && task dev:status
+
+# Start backend (from monorepo root)
+cd cloud/maplefile-backend && task dev
+
+# Make code changes - backend auto-restarts
+
+# Stop backend when done
+# Press Ctrl+C
+```
+
+### Testing
+
+```bash
+# Run all tests
+task test
+
+# Code quality checks
+task format  # Format code
+task lint    # Run linters
+```
+
+### Database Operations
+
+**View database:**
+```bash
+# From monorepo root
+cd cloud/infrastructure/development
+task cql
+
+# Inside cqlsh:
+USE maplefile;
+DESCRIBE TABLES;
+SELECT * FROM users_by_id;
+```
+
+**Reset database (⚠️ deletes all data):**
+```bash
+task db:clear
+```
+
+## 🔧 Usage
+
+### Testing the API
+
+Create a test user to verify the backend works:
+
+**1. Register a user:**
+```bash
+curl -X POST http://localhost:8000/api/v1/auth/register \
+  -H "Content-Type: application/json" \
+  -d '{
+    "email": "test@example.com",
+    "first_name": "Test",
+    "last_name": "User",
+    "phone": "+1234567890",
+    "country": "Canada",
+    "timezone": "America/Toronto",
+    "salt": "base64-encoded-salt",
+    "kdf_algorithm": "argon2id",
+    "kdf_iterations": 3,
+    "kdf_memory": 65536,
+    "kdf_parallelism": 4,
+    "kdf_salt_length": 16,
+    "kdf_key_length": 32,
+    "encryptedMasterKey": "base64-encoded-encrypted-master-key",
+    "publicKey": "base64-encoded-public-key",
+    "encryptedPrivateKey": "base64-encoded-encrypted-private-key",
+    "encryptedRecoveryKey": "base64-encoded-encrypted-recovery-key",
+    "masterKeyEncryptedWithRecoveryKey": "base64-encoded-master-key-encrypted-with-recovery",
+    "agree_terms_of_service": true,
+    "agree_promotions": false,
+    "agree_to_tracking_across_third_party_apps_and_services": false
+  }'
+```
+
+> **Note:** MapleFile uses end-to-end encryption. The frontend (maplefile-frontend) handles all cryptographic operations. For manual API testing, you'll need to generate valid encryption keys using libsodium. See the frontend registration implementation for reference.
+
+**Response:**
+```json
+{
+  "message": "Registration successful. Please check your email to verify your account.",
+  "user_id": "uuid-here"
+}
+```
+
+**2. Verify email:**
+Check your email for the verification code, then:
+```bash
+curl -X POST http://localhost:8000/api/v1/auth/verify-email \
+  -H "Content-Type: application/json" \
+  -d '{
+    "email": "test@example.com",
+    "verification_code": "123456"
+  }'
+```
+
+**3. Login:**
+```bash
+curl -X POST http://localhost:8000/api/v1/auth/login \
+  -H "Content-Type: application/json" \
+  -d '{
+    "email": "test@example.com"
+  }'
+```
+
+Check your email for the OTP (One-Time Password), then complete login:
+```bash
+curl -X POST http://localhost:8000/api/v1/auth/login/verify-otp \
+  -H "Content-Type: application/json" \
+  -d '{
+    "email": "test@example.com",
+    "otp": "your-otp-code",
+    "encrypted_challenge": "base64-encoded-challenge-response"
+  }'
+```
+
+**Response:**
+```json
+{
+  "access_token": "eyJhbGci...",
+  "refresh_token": "eyJhbGci...",
+  "access_expiry": "2025-11-12T13:00:00Z",
+  "refresh_expiry": "2025-11-19T12:00:00Z"
+}
+```
+
+Save the `access_token` from the response:
+```bash
+export TOKEN="eyJhbGci...your-access-token-here"
+```
+
+**4. Get your profile:**
+```bash
+curl http://localhost:8000/api/v1/me \
+  -H "Authorization: JWT $TOKEN"
+```
+
+**5. Get dashboard:**
+```bash
+curl http://localhost:8000/api/v1/dashboard \
+  -H "Authorization: JWT $TOKEN"
+```
+
+**6. Create a collection (folder):**
+```bash
+curl -X POST http://localhost:8000/api/v1/collections \
+  -H "Content-Type: application/json" \
+  -H "Authorization: JWT $TOKEN" \
+  -d '{
+    "name": "My Documents",
+    "description": "Personal documents",
+    "collection_type": "folder",
+    "encrypted_collection_key": "base64-encoded-encrypted-key"
+  }'
+```
+
+**7. Upload a file:**
+```bash
+# First, get a presigned URL
+curl -X POST http://localhost:8000/api/v1/files/presigned-url \
+  -H "Content-Type: application/json" \
+  -H "Authorization: JWT $TOKEN" \
+  -d '{
+    "file_name": "document.pdf",
+    "file_size": 1024000,
+    "mime_type": "application/pdf",
+    "collection_id": "your-collection-id"
+  }'
+
+# Upload the encrypted file to the presigned URL (using the URL from response)
+curl -X PUT "presigned-url-here" \
+  --upload-file your-encrypted-file.enc
+
+# Report upload completion
+curl -X POST http://localhost:8000/api/v1/files/upload-complete \
+  -H "Content-Type: application/json" \
+  -H "Authorization: JWT $TOKEN" \
+  -d '{
+    "file_id": "file-id-from-presigned-response",
+    "status": "completed"
+  }'
+```
+
+### Frontend Integration
+
+**Access the frontend:**
+- URL: http://localhost:5173
+- The frontend handles all encryption/decryption automatically
+- See [`../../web/maplefile-frontend/README.md`](../../web/maplefile-frontend/README.md)
+
+**Key Features:**
+- 🔐 **Client-side encryption** - Files encrypted before upload
+- 🔑 **E2EE Key Chain** - Password → KEK → Master Key → Collection Keys → File Keys
+- 📁 **Collections** - Organize files in encrypted folders
+- 🤝 **Sharing** - Share collections with read-only, read-write, or admin permissions
+- 🔄 **Sync modes** - Cloud-only, local-only, or hybrid storage
+
+**Next steps:**
+- Frontend setup: [`../../web/maplefile-frontend/README.md`](../../web/maplefile-frontend/README.md)
+- Complete API documentation: See API endpoints in code
+
+## ⚙️ Configuration
+
+### Environment Variables
+
+Key variables in `.env`:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `BACKEND_APP_JWT_SECRET` | `change-me-in-production` | Secret for JWT token signing |
+| `BACKEND_APP_SERVER_PORT` | `8000` | HTTP server port |
+| `BACKEND_DB_HOSTS` | `cassandra-1,cassandra-2,cassandra-3` | Cassandra cluster nodes |
+| `BACKEND_CACHE_HOST` | `redis` | Redis cache host |
+| `BACKEND_MAPLEFILE_S3_ENDPOINT` | `http://seaweedfs:8333` | S3 storage URL |
+| `BACKEND_MAPLEFILE_S3_BUCKET` | `maplefile` | S3 bucket name |
+
+**Docker vs Local:**
+- Docker: Uses container names (`cassandra-1`, `redis`, `seaweedfs`)
+- Local: Change to `localhost`
+
+See `.env.sample` for complete documentation.
+
+### Task Commands
+
+| Command | Description |
+|---------|-------------|
+| `task dev` | Start backend (auto-migrate + hot-reload) |
+| `task dev:down` | Stop backend |
+| `task test` | Run tests |
+| `task format` | Format code |
+| `task lint` | Run linters |
+| `task db:clear` | Reset database (⚠️ deletes data) |
+| `task migrate:up` | Manual migration |
+| `task build` | Build binary |
+
+## 🔍 Troubleshooting
+
+### Backend won't start - "connection refused"
+
+**Error:** `dial tcp 127.0.0.1:9042: connect: connection refused`
+
+**Cause:** `.env` file has `localhost` instead of container names.
+
+**Fix:**
+```bash
+cd cloud/maplefile-backend
+rm .env
+cp .env.sample .env
+task dev
+```
+
+### Infrastructure not running
+
+**Error:** Cassandra or Redis not available
+
+**Fix:**
+```bash
+cd cloud/infrastructure/development
+task dev:start
+task dev:status  # Wait until all show (healthy)
+```
+
+### Port 8000 already in use
+
+**Fix:**
+```bash
+lsof -i :8000  # Find what's using the port
+# Stop the other service, or change BACKEND_APP_SERVER_PORT in .env
+```
+
+### Token expired (401 errors)
+
+JWT tokens expire after 60 minutes. Re-run the [login steps](#testing-the-api) to get a new token.
+
+### Database keyspace not found
+
+**Error:** `Keyspace 'maplefile' does not exist` or `failed to create user`
+
+**Cause:** The Cassandra keyspace hasn't been created yet. This is a one-time infrastructure setup.
+
+**Fix:**
+```bash
+# Initialize the keyspace (one-time setup)
+cd cloud/infrastructure/development
+
+# Find Cassandra container
+export CASSANDRA_CONTAINER=$(docker ps --filter "name=cassandra" -q | head -1)
+
+# Create keyspace
+docker exec -it $CASSANDRA_CONTAINER cqlsh -e "
+CREATE KEYSPACE IF NOT EXISTS maplefile
+WITH replication = {
+    'class': 'SimpleStrategy',
+    'replication_factor': 3
+};"
+
+# Verify keyspace exists
+docker exec -it $CASSANDRA_CONTAINER cqlsh -e "DESCRIBE KEYSPACE maplefile;"
+
+# Restart backend to retry migrations
+cd ../../maplefile-backend
+task dev:restart
+```
+
+**Note:** The backend auto-migrates tables on startup, but expects the keyspace to already exist. This is standard practice - keyspaces are infrastructure setup, not application migrations.
+
+## 🛠️ Technology Stack
+
+- **Go 1.23+** - Programming language
+- **Clean Architecture** - Code organization
+- **Wire** - Dependency injection (Google's code generation)
+- **Cassandra 5.0.4** - Distributed database (3-node cluster)
+- **Redis 7** - Caching layer
+- **SeaweedFS** - S3-compatible object storage
+- **JWT** - User authentication
+- **ChaCha20-Poly1305** - Authenticated encryption (client-side)
+- **Argon2id** - Password hashing / KDF
+
+## 🌐 Services
+
+When you run MapleFile, these services are available:
+
+| Service | Port | Purpose | Access |
+|---------|------|---------|--------|
+| MapleFile Backend | 8000 | HTTP API | http://localhost:8000 |
+| MapleFile Frontend | 5173 | Web UI | http://localhost:5173 |
+| Cassandra | 9042 | Database | `task cql` (from infrastructure dir) |
+| Redis | 6379 | Cache | `task redis` (from infrastructure dir) |
+| SeaweedFS S3 | 8333 | Object storage | http://localhost:8333 |
+| SeaweedFS UI | 9333 | Storage admin | http://localhost:9333 |
+
+## 🏗️ Architecture
+
+### Project Structure
+
+```
+maplefile-backend/
+├── cmd/              # CLI commands (daemon, migrate, version)
+├── config/           # Configuration loading
+├── internal/         # Application code
+│   ├── app/          # Wire application wiring
+│   ├── domain/       # Domain entities
+│   │   ├── collection/  # Collections (folders)
+│   │   ├── crypto/      # Encryption types
+│   │   ├── file/        # File metadata
+│   │   ├── user/        # User accounts
+│   │   └── ...
+│   ├── repo/         # Repository implementations (Cassandra)
+│   ├── usecase/      # Use cases / business logic
+│   ├── service/      # Service layer
+│   └── interface/    # HTTP handlers
+│       └── http/     # REST API endpoints
+├── pkg/              # Shared infrastructure
+│   ├── storage/      # Database, cache, S3, memory
+│   ├── security/     # JWT, encryption, password hashing
+│   └── emailer/      # Email sending
+├── migrations/       # Cassandra schema migrations
+└── docs/             # Documentation
+```
+
+### Key Features
+
+- **🔐 Zero-Knowledge Architecture**: Files encrypted on client, server never sees plaintext
+- **🔑 E2EE Key Chain**: User Password → KEK → Master Key → Collection Keys → File Keys
+- **📦 Storage Modes**: `encrypted_only`, `hybrid`, `decrypted_only`
+- **🤝 Collection Sharing**: `read_only`, `read_write`, `admin` permissions
+- **💾 Two-Tier Caching**: Redis + Cassandra-based cache
+- **📊 Storage Quotas**: 10GB default per user
+- **🔄 File Versioning**: Soft delete with tombstone tracking
+
+### End-to-End Encryption Flow
+
+```
+1. User enters password → Frontend derives KEK (Key Encryption Key)
+2. KEK → Encrypts/decrypts Master Key (stored encrypted on server)
+3. Master Key → Encrypts/decrypts Collection Keys
+4. Collection Key → Encrypts/decrypts File Keys
+5. File Key → Encrypts/decrypts actual file content
+
+Server only stores:
+- Encrypted Master Key (encrypted with KEK from password)
+- Encrypted Collection Keys (encrypted with Master Key)
+- Encrypted File Keys (encrypted with Collection Key)
+- Encrypted file content (encrypted with File Key)
+
+Server NEVER has access to:
+- User's password
+- KEK (derived from password on client)
+- Decrypted Master Key
+- Decrypted Collection Keys
+- Decrypted File Keys
+- Plaintext file content
+```
+
+## 🔗 Links
+
+- **Frontend Application:** [`../../web/maplefile-frontend/README.md`](../../web/maplefile-frontend/README.md)
+- **CLI Tool:** [`../../native/desktop/maplefile/README.md`](../../native/desktop/maplefile/README.md)
+- **Architecture Details:** [`../../CLAUDE.md`](../../CLAUDE.md)
+- **Repository:** [Codeberg - mapleopentech/monorepo](https://codeberg.org/mapleopentech/monorepo)
+
+## 🤝 Contributing
+
+Found a bug? Want a feature to improve MapleFile? Please create an [issue](https://codeberg.org/mapleopentech/monorepo/issues/new).
+
+## 📝 License
+
+This application is licensed under the [**GNU Affero General Public License v3.0**](https://opensource.org/license/agpl-v3). See [LICENSE](../../LICENSE) for more information.