mapleopentech/monorepo
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Initial commit: Open sourcing all of the Maple Open Technologies code.

Browse source
This commit is contained in:
Bartlomiej Mika 2025-12-02 14:33:08 -05:00
commit 755d54a99d
2010 changed files with 448675 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

387
cloud/infrastructure/development/README.md Normal file
View file

@ -0,0 +1,387 @@
# 🏗️ MapleFile (Development) Infrastructure
> Shared development infrastructure for all MapleFile projects. Start once, use everywhere.
## 📖 What is this?
Think of this as your **local cloud environment**. Instead of each MapleFile project (maplefile-backend, maplepress-backend, etc.) running its own database, cache, and storage, they all share this common infrastructure - just like production apps share AWS/cloud services.
**What you get:**
- Database (Cassandra) - stores your data
- Cache (Redis) - makes things fast
- Search (Meilisearch) - powers search features
- File Storage (SeaweedFS) - stores uploaded files
- WordPress (for plugin testing)
**Why shared?**
- Start infrastructure once, restart your apps quickly (seconds vs minutes)
- Closer to real production setup
- Learn proper microservices architecture
**No environment variables needed here** - this project is already configured for local development. The apps that connect to it will have their own `.env` files.
## ⚡ TL;DR
```bash
task dev:start # Start everything (takes 2-3 minutes first time)
task dev:status # Verify all services show "healthy"
```
**Then:** Navigate to a backend project (`../maplepress-backend/` or `../maplefile-backend/`) and follow its README to set up and start the backend. See [What's Next?](#whats-next) section below.
## 📋 Prerequisites
You need these tools installed before starting. Don't worry - they're free and easy to install.
### 1. Docker Desktop
**What is Docker?** A tool that runs software in isolated containers. Think of it as lightweight virtual machines that start instantly.
**Download & Install:**
- **macOS:** [Docker Desktop for Mac](https://www.docker.com/products/docker-desktop/) (includes docker-compose)
- **Windows:** [Docker Desktop for Windows](https://www.docker.com/products/docker-desktop/)
- **Linux:** Follow instructions at [docs.docker.com/engine/install](https://docs.docker.com/engine/install/)
**Verify installation:**
```bash
docker --version # Should show: Docker version 20.x or higher
docker compose version # Should show: Docker Compose version 2.x or higher
```
**What is Docker Compose?** A tool for running multiple Docker containers together. It's **included with Docker Desktop** - you don't need to install it separately! When you install Docker Desktop, you automatically get Docker Compose.
**Note on Docker Compose versions:**
- **Docker Compose v1** (older): Uses `docker-compose` command (hyphen)
- **Docker Compose v2** (current): Uses `docker compose` command (space)
- Our Taskfile **automatically detects** which version you have and uses the correct command
- If you're on Linux with Docker Compose v2, use `docker compose version` (not `docker-compose --version`)
### 2. Task (Task Runner)
**What is Task?** A simple command runner (like `make` but better). We use it instead of typing long docker commands.
**Install:**
- **macOS:** `brew install go-task`
- **Windows:** `choco install go-task` (using [Chocolatey](https://chocolatey.org/))
- **Linux:** `snap install task --classic`
- **Manual install:** Download from [taskfile.dev](https://taskfile.dev/installation/)
**Verify installation:**
```bash
task --version # Should show: Task version 3.x or higher
```
### 3. All other services (Cassandra, Redis, etc.)
**Do I need to install them?** **NO!** Docker will automatically download and run everything. You don't install Cassandra, Redis, or any database directly on your computer.
**What happens when you run `task dev:start`:**
1. Docker downloads required images (first time only - takes a few minutes)
2. Starts all services in containers
3. That's it - everything is ready to use!
## ❓ Common Questions
**Q: Do I need to configure environment variables or create a `.env` file?**
A: **No!** This infrastructure project is pre-configured for local development. However, the application projects that connect to it (like `maplefile-backend`) will need their own `.env` files - check their READMEs.
**Q: Do I need to install Cassandra, Redis, or other databases?**
A: **No!** Docker handles everything. You only install Docker and Task, nothing else.
**Q: Will this mess up my computer or conflict with other projects?**
A: **No!** Everything runs in isolated Docker containers. You can safely remove it all with `task dev:clean` and `docker system prune`.
**Q: How much disk space does this use?**
A: Initial download: ~2-3 GB. Running services + data: ~5-10 GB depending on usage.
**Q: Can I use this on Windows?**
A: **Yes!** Docker Desktop works on Windows. Just make sure to use PowerShell or Git Bash for commands.
**Q: What is Docker Compose? Do I need to install it separately?**
A: **No!** Docker Compose is included with Docker Desktop automatically. When you install Docker Desktop, you get both `docker` and `docker compose` commands.
**Q: I'm getting "docker-compose: command not found" on Linux. What should I do?**
A: You likely have Docker Compose v2, which uses `docker compose` (space) instead of `docker-compose` (hyphen). Our Taskfile automatically detects and uses the correct command. Just run `task dev:start` and it will work on both Mac and Linux.
## 🚀 Quick Start
### 1. Start Infrastructure
```bash
task dev:start
```
Wait for: `✅ Infrastructure ready!`
### 2. Verify Everything Works
```bash
task dev:status
```
**Expected output:** All services show `Up X minutes (healthy)`
```
NAMES STATUS PORTS
maple-cassandra-1-dev Up 2 minutes (healthy) 0.0.0.0:9042->9042/tcp
maple-redis-dev Up 2 minutes (healthy) 0.0.0.0:6379->6379/tcp
maple-wordpress-dev Up 2 minutes (healthy) 0.0.0.0:8081->80/tcp
...
```
### 3. Start Your App
Now navigate to your app directory (e.g., `maplefile-backend`) and run its `task dev` command. Your app will automatically connect to this infrastructure.
### 4. Stop Infrastructure (End of Day)
```bash
task dev:stop # Stops services, keeps data
```
## 🎯 What's Next?
🎉 **Infrastructure is running!** Now set up a backend:
- **MaplePress Backend:** [`../maplepress-backend/README.md`](../maplepress-backend/README.md)
- **MapleFile Backend:** [`../maplefile-backend/README.md`](../maplefile-backend/README.md)
Pick one, navigate to its directory, and follow its setup instructions.
## 📅 Daily Commands
```bash
# Morning - start infrastructure
task dev:start
# Check if everything is running
task dev:status
# Evening - stop infrastructure (keeps data)
task dev:stop
# Nuclear option - delete everything and start fresh
task dev:clean # ⚠️ DELETES ALL DATA
```
## 🔍 Troubleshooting
### Service shows unhealthy or won't start
```bash
# Check logs for specific service
task dev:logs -- cassandra-1
task dev:logs -- redis
task dev:logs -- wordpress
# Or follow logs in real-time
task dev:logs -- cassandra-1
```
**Service names:** `cassandra-1`, `cassandra-2`, `cassandra-3`, `redis`, `meilisearch`, `seaweedfs`, `mariadb`, `wordpress`
### Port already in use
Another service is using the required ports. Check:
- Port 9042 (Cassandra)
- Port 6379 (Redis)
- Port 8081 (WordPress)
- Port 3306 (MariaDB)
Find and stop the conflicting service:
```bash
lsof -i :9042 # macOS/Linux
```
### Want to reset everything
```bash
task dev:clean # Removes all containers and data
task dev:start # Fresh start
```
## 🌐 What's Running?
When you start infrastructure, you get these services:
| Service | Port | Purpose | Access |
|---------|------|---------|--------|
| Cassandra Cluster | 9042 | Database (3-node cluster) | `task cql` |
| Redis | 6379 | Cache & sessions | `task redis` |
| Meilisearch | 7700 | Search engine | http://localhost:7700 |
| SeaweedFS | 8333, 9333 | S3-compatible storage | http://localhost:9333 |
| MariaDB | 3306 | WordPress database | - |
| WordPress | 8081 | Plugin testing | http://localhost:8081 |
## 🔧 Common Operations
### Working with Cassandra
```bash
# Open CQL shell
task cql
# List all keyspaces
task cql:keyspaces
# List tables in a keyspace
task cql:tables -- maplepress
# Check cluster health
task cql:status
```
**Available keyspaces:**
- `maplefile` - MapleFile backend (Redis DB: 1)
- `maplepress` - MaplePress backend (Redis DB: 0)
### Working with Redis
```bash
# Open Redis CLI
task redis
# Then inside Redis CLI:
# SELECT 0 # Switch to maplepress database
# SELECT 1 # Switch to maplefile database
# KEYS * # List all keys
```
### Working with WordPress
**Access:** http://localhost:8081
**First-time setup:**
1. Visit http://localhost:8081
2. Complete WordPress installation wizard
3. Use any credentials (this is a dev site)
**Credentials for WordPress database:**
- Host: `mariadb:3306`
- Database: `wordpress`
- User: `wordpress`
- Password: `wordpress`
**View debug logs:**
```bash
docker exec -it maple-wordpress-dev tail -f /var/www/html/wp-content/debug.log
```
### Working with SeaweedFS (S3 Storage)
**Web UI:** http://localhost:9333
**S3 Configuration for your apps:**
```bash
S3_ENDPOINT=http://seaweedfs:8333
S3_REGION=us-east-1
S3_ACCESS_KEY=any
S3_SECRET_KEY=any
```
## 💻 Development Workflow
**Typical daily flow:**
1. **Morning:** `task dev:start` (in this directory)
2. **Start app:** `cd ../maplefile-backend && task dev`
3. **Work on code** - restart app as needed (fast!)
4. **Infrastructure keeps running** - no need to restart
5. **Evening:** `task dev:stop` (optional - can leave running)
**Why this approach?**
- Infrastructure takes 2-3 minutes to start (Cassandra cluster is slow)
- Your app restarts in seconds
- Start infrastructure once, restart apps freely
## 💾 Data Persistence
All data is stored in Docker volumes and survives restarts:
- `maple-cassandra-1-dev`, `maple-cassandra-2-dev`, `maple-cassandra-3-dev`
- `maple-redis-dev`
- `maple-meilisearch-dev`
- `maple-seaweedfs-dev`
- `maple-mariadb-dev`
- `maple-wordpress-dev`
**To completely reset (deletes all data):**
```bash
task dev:clean
```
## 🎓 Advanced Topics
> **⚠️ SKIP THIS SECTION FOR INITIAL SETUP!**
>
> These topics are for **future use** - after you've successfully set up and used the infrastructure. You don't need to read or do anything here when setting up for the first time.
>
> Come back here only when you need to:
> - Add a new project to the infrastructure (not needed now - mapleopentech and maplepress already configured)
> - Understand Cassandra cluster architecture (curiosity only)
> - Learn why we chose this approach (optional reading)
### Adding a New Project
**When do I need this?** Only if you're creating a brand new project (not maplefile-backend or maplepress-backend - those are already set up).
To add a new project to shared infrastructure:
1. Add keyspace to `cassandra/init-scripts/01-create-keyspaces.cql`:
```cql
CREATE KEYSPACE IF NOT EXISTS mynewproject
WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 3};
```
2. Configure your project's `docker-compose.dev.yml`:
```yaml
networks:
maple-dev:
external: true
services:
app:
environment:
- DATABASE_HOSTS=cassandra-1:9042,cassandra-2:9042,cassandra-3:9042
- DATABASE_KEYSPACE=mynewproject
- DATABASE_CONSISTENCY=QUORUM
- DATABASE_REPLICATION=3
- REDIS_HOST=redis
- REDIS_DB=2 # Use next available: 0=maplepress, 1=maplefile
networks:
- maple-dev
```
3. Restart infrastructure:
```bash
task dev:restart
```
### Cassandra Cluster Details
- **3-node cluster** for high availability
- **Replication factor: 3** (data on all nodes)
- **Consistency level: QUORUM** (2 of 3 nodes must agree)
- **Seed node:** cassandra-1 (other nodes join via this node)
### Architecture Decision: Why Separate Infrastructure?
**Benefits:**
- Faster app restarts (seconds vs minutes)
- Share infrastructure across multiple projects
- Closer to production architecture
- Learn proper service separation
**Trade-off:**
- One extra terminal/directory to manage
- Slightly more complex than monolithic docker-compose
We chose speed and realism over simplicity.
## Contributing
Found a bug? Want a feature to improve the infrastructure? Please create an [issue](https://codeberg.org/mapleopentech/monorepo/issues/new).
## License
This application is licensed under the [**GNU Affero General Public License v3.0**](https://opensource.org/license/agpl-v3). See [LICENSE](../../LICENSE) for more information.