Initial commit: Open sourcing all of the Maple Open Technologies code.
This commit is contained in:
commit
755d54a99d
2010 changed files with 448675 additions and 0 deletions
555
cloud/maplepress-backend/docs/SITE_VERIFICATION.md
Normal file
555
cloud/maplepress-backend/docs/SITE_VERIFICATION.md
Normal file
|
|
@ -0,0 +1,555 @@
|
|||
# Site Verification System
|
||||
|
||||
## Overview
|
||||
|
||||
MaplePress implements **DNS-based domain ownership verification** to ensure users actually own the domains they register. Sites start in "pending" status and remain there until verified through DNS TXT record validation.
|
||||
|
||||
## Verification Method: DNS TXT Records
|
||||
|
||||
MaplePress uses **DNS TXT record verification** - the industry standard used by Google, Cloudflare, and other major services. This proves domain ownership, not just dashboard access.
|
||||
|
||||
### Why DNS Verification?
|
||||
|
||||
- **Proves domain ownership**: Only someone with DNS access can add TXT records
|
||||
- **Industry standard**: Same method used by Google Search Console, Cloudflare, etc.
|
||||
- **Secure**: Cannot be spoofed or bypassed without actual domain control
|
||||
- **Automatic**: Backend performs verification via DNS lookup
|
||||
|
||||
## Site Status Lifecycle
|
||||
|
||||
### Status Constants
|
||||
|
||||
**File**: `internal/domain/site/site.go:61-67`
|
||||
|
||||
```go
|
||||
const (
|
||||
StatusPending = "pending" // Site created, awaiting DNS verification
|
||||
StatusActive = "active" // Site verified via DNS and operational
|
||||
StatusInactive = "inactive" // User temporarily disabled
|
||||
StatusSuspended = "suspended" // Suspended due to violation or non-payment
|
||||
StatusArchived = "archived" // Soft deleted
|
||||
)
|
||||
```
|
||||
|
||||
## 1. Site Creation (Pending State)
|
||||
|
||||
**File**: `internal/usecase/site/create.go`
|
||||
|
||||
When a site is created via **POST /api/v1/sites**:
|
||||
|
||||
### What Gets Generated
|
||||
|
||||
1. **API Key** (test or live mode)
|
||||
- Test mode: `test_sk_...` (skips DNS verification)
|
||||
- Live mode: `live_sk_...` (requires DNS verification)
|
||||
|
||||
2. **Verification Token** (lines 88-92)
|
||||
- Format: `mvp_` + 128-bit random token (base64-encoded)
|
||||
- Example: `mvp_xyz789abc123`
|
||||
- Used in DNS TXT record: `maplepress-verify={token}`
|
||||
|
||||
3. **DNS Verification Instructions**
|
||||
- Provides step-by-step DNS setup guide
|
||||
- Includes domain registrar examples (GoDaddy, Namecheap, Cloudflare, etc.)
|
||||
- Explains DNS propagation timing (5-10 minutes typical)
|
||||
|
||||
4. **Site Entity** (lines 104-113)
|
||||
- Initial status: `StatusPending`
|
||||
- `IsVerified`: `false`
|
||||
- `VerificationToken`: Set to generated token
|
||||
|
||||
### Response Example
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
||||
"domain": "example.com",
|
||||
"site_url": "https://example.com",
|
||||
"api_key": "live_sk_a1b2...", // ⚠️ SHOWN ONLY ONCE
|
||||
"verification_token": "mvp_xyz789abc123",
|
||||
"status": "pending",
|
||||
"search_index_name": "site_...",
|
||||
"verification_instructions": "To verify ownership of example.com, add this DNS TXT record:\n\nHost/Name: example.com\nType: TXT\nValue: maplepress-verify=mvp_xyz789abc123\n\nInstructions:\n1. Log in to your domain registrar...\n2. Find DNS settings...\n3. Add a new TXT record...\n4. Wait 5-10 minutes for DNS propagation\n5. Click 'Verify Domain' in MaplePress"
|
||||
}
|
||||
```
|
||||
|
||||
**Documentation**: `docs/API/create-site.md`
|
||||
|
||||
## 2. Test Mode Bypass
|
||||
|
||||
**File**: `internal/domain/site/site.go:115-125`
|
||||
|
||||
### Test Mode Detection
|
||||
|
||||
```go
|
||||
func (s *Site) IsTestMode() bool {
|
||||
return len(s.APIKeyPrefix) >= 7 && s.APIKeyPrefix[:7] == "test_sk"
|
||||
}
|
||||
```
|
||||
|
||||
### Verification Requirement Check
|
||||
|
||||
```go
|
||||
func (s *Site) RequiresVerification() bool {
|
||||
return !s.IsTestMode() // Test mode sites skip verification
|
||||
}
|
||||
```
|
||||
|
||||
**Key Points:**
|
||||
- Sites with `test_sk_` API keys **skip verification** entirely
|
||||
- Useful for development and testing
|
||||
- Test mode sites can sync pages immediately
|
||||
|
||||
## 3. API Access Control
|
||||
|
||||
**File**: `internal/domain/site/site.go:127-140`
|
||||
|
||||
### CanAccessAPI() Method
|
||||
|
||||
```go
|
||||
func (s *Site) CanAccessAPI() bool {
|
||||
// Allow active sites (fully verified)
|
||||
if s.Status == StatusActive {
|
||||
return true
|
||||
}
|
||||
// Allow pending sites (waiting for verification) for initial setup
|
||||
if s.Status == StatusPending {
|
||||
return true
|
||||
}
|
||||
// Block inactive, suspended, or archived sites
|
||||
return false
|
||||
}
|
||||
```
|
||||
|
||||
**Important**: Pending sites **CAN access the API** for:
|
||||
- Status checks (`GET /api/v1/plugin/status`)
|
||||
- Initial plugin setup
|
||||
- Retrieving site information
|
||||
|
||||
## 4. Verification Enforcement
|
||||
|
||||
### Where Verification is Required
|
||||
|
||||
**File**: `internal/usecase/page/sync.go:85-89`
|
||||
|
||||
When syncing pages (**POST /api/v1/plugin/sync**):
|
||||
|
||||
```go
|
||||
// Verify site is verified (skip for test mode)
|
||||
if site.RequiresVerification() && !site.IsVerified {
|
||||
uc.logger.Warn("site not verified", zap.String("site_id", siteID.String()))
|
||||
return nil, domainsite.ErrSiteNotVerified
|
||||
}
|
||||
```
|
||||
|
||||
**Error**: `internal/domain/site/errors.go:22`
|
||||
```go
|
||||
ErrSiteNotVerified = errors.New("site is not verified")
|
||||
```
|
||||
|
||||
### HTTP Response
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "about:blank",
|
||||
"title": "Forbidden",
|
||||
"status": 403,
|
||||
"detail": "site is not verified"
|
||||
}
|
||||
```
|
||||
|
||||
## 5. DNS Verification Implementation
|
||||
|
||||
### DNS Verifier Package
|
||||
|
||||
**File**: `pkg/dns/verifier.go`
|
||||
|
||||
```go
|
||||
type Verifier struct {
|
||||
resolver *net.Resolver
|
||||
logger *zap.Logger
|
||||
}
|
||||
|
||||
func (v *Verifier) VerifyDomainOwnership(ctx context.Context, domain string, expectedToken string) (bool, error) {
|
||||
// Create context with 10-second timeout
|
||||
lookupCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
|
||||
// Look up TXT records for the domain
|
||||
txtRecords, err := v.resolver.LookupTXT(lookupCtx, domain)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("DNS lookup failed: %w", err)
|
||||
}
|
||||
|
||||
// Expected format: "maplepress-verify=TOKEN"
|
||||
expectedRecord := fmt.Sprintf("maplepress-verify=%s", expectedToken)
|
||||
|
||||
for _, record := range txtRecords {
|
||||
if strings.TrimSpace(record) == expectedRecord {
|
||||
return true, nil // Domain ownership verified!
|
||||
}
|
||||
}
|
||||
|
||||
return false, nil // TXT record not found
|
||||
}
|
||||
```
|
||||
|
||||
### DNS Verification Use Case
|
||||
|
||||
**File**: `internal/usecase/site/verify.go`
|
||||
|
||||
The verification use case performs DNS lookup:
|
||||
|
||||
```go
|
||||
func (uc *VerifySiteUseCase) Execute(ctx context.Context, tenantID gocql.UUID, siteID gocql.UUID, input *VerifySiteInput) (*VerifySiteOutput, error) {
|
||||
// Get site from repository
|
||||
site, err := uc.repo.GetByID(ctx, tenantID, siteID)
|
||||
if err != nil {
|
||||
return nil, domainsite.ErrSiteNotFound
|
||||
}
|
||||
|
||||
// Check if already verified
|
||||
if site.IsVerified {
|
||||
return &VerifySiteOutput{Success: true, Status: site.Status, Message: "Site is already verified"}, nil
|
||||
}
|
||||
|
||||
// Test mode sites skip DNS verification
|
||||
if site.IsTestMode() {
|
||||
site.Verify()
|
||||
if err := uc.repo.Update(ctx, site); err != nil {
|
||||
return nil, fmt.Errorf("failed to update site: %w", err)
|
||||
}
|
||||
return &VerifySiteOutput{Success: true, Status: site.Status, Message: "Test mode site verified successfully"}, nil
|
||||
}
|
||||
|
||||
// Perform DNS TXT record verification
|
||||
verified, err := uc.dnsVerifier.VerifyDomainOwnership(ctx, site.Domain, site.VerificationToken)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("DNS verification failed: %w", err)
|
||||
}
|
||||
|
||||
if !verified {
|
||||
return nil, fmt.Errorf("DNS TXT record not found. Please add the verification record to your domain's DNS settings")
|
||||
}
|
||||
|
||||
// DNS verification successful - mark site as verified
|
||||
site.Verify()
|
||||
if err := uc.repo.Update(ctx, site); err != nil {
|
||||
return nil, fmt.Errorf("failed to update site: %w", err)
|
||||
}
|
||||
|
||||
return &VerifySiteOutput{Success: true, Status: site.Status, Message: "Domain ownership verified successfully via DNS TXT record"}, nil
|
||||
}
|
||||
```
|
||||
|
||||
### Verify Method
|
||||
|
||||
**File**: `internal/domain/site/site.go:169-175`
|
||||
|
||||
```go
|
||||
// Verify marks the site as verified
|
||||
func (s *Site) Verify() {
|
||||
s.IsVerified = true
|
||||
s.Status = StatusActive
|
||||
s.VerificationToken = "" // Clear token after verification
|
||||
s.UpdatedAt = time.Now()
|
||||
}
|
||||
```
|
||||
|
||||
## 6. What Pending Sites Can Do
|
||||
|
||||
**File**: `internal/interface/http/handler/plugin/status_handler.go`
|
||||
|
||||
### Allowed Operations
|
||||
|
||||
✅ **GET /api/v1/plugin/status** - Check site status and quotas
|
||||
- Returns full site details
|
||||
- Shows `is_verified: false`
|
||||
- Shows `status: "pending"`
|
||||
|
||||
### Blocked Operations
|
||||
|
||||
❌ **POST /api/v1/plugin/sync** - Sync pages to search index
|
||||
- Returns 403 Forbidden
|
||||
- Error: "site is not verified"
|
||||
|
||||
❌ **POST /api/v1/plugin/search** - Perform searches
|
||||
- Blocked for unverified sites
|
||||
|
||||
❌ **DELETE /api/v1/plugin/pages** - Delete pages
|
||||
- Blocked for unverified sites
|
||||
|
||||
## 7. Verification Token Details
|
||||
|
||||
**File**: `internal/usecase/site/generate_verification_token.go`
|
||||
|
||||
### Token Generation
|
||||
|
||||
```go
|
||||
func (uc *GenerateVerificationTokenUseCase) Execute() (string, error) {
|
||||
b := make([]byte, 16) // 16 bytes = 128 bits
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
uc.logger.Error("failed to generate random bytes", zap.Error(err))
|
||||
return "", err
|
||||
}
|
||||
|
||||
token := base64.RawURLEncoding.EncodeToString(b)
|
||||
verificationToken := "mvp_" + token // mvp = maplepress verify
|
||||
|
||||
uc.logger.Info("verification token generated")
|
||||
return verificationToken, nil
|
||||
}
|
||||
```
|
||||
|
||||
**Token Format:**
|
||||
- Prefix: `mvp_` (MaplePress Verify)
|
||||
- Encoding: Base64 URL-safe (no padding)
|
||||
- Strength: 128-bit cryptographic randomness
|
||||
- Example: `mvp_dGhpc2lzYXRlc3Q`
|
||||
|
||||
**Security:**
|
||||
- Never exposed in JSON responses (marked with `json:"-"`)
|
||||
- Stored in database only
|
||||
- Cleared after verification
|
||||
|
||||
## 8. DNS Verification Flow
|
||||
|
||||
### Step-by-Step Process
|
||||
|
||||
1. **User creates site** via dashboard (POST /api/v1/sites)
|
||||
- Backend generates API key and verification token
|
||||
- Site status: `pending`
|
||||
- Response includes DNS setup instructions
|
||||
- User receives: API key (once), verification token, DNS TXT record format
|
||||
|
||||
2. **User adds DNS TXT record** to domain registrar
|
||||
- Logs in to domain registrar (GoDaddy, Namecheap, Cloudflare, etc.)
|
||||
- Navigates to DNS management
|
||||
- Adds TXT record: `maplepress-verify={verification_token}`
|
||||
- Waits 5-10 minutes for DNS propagation (can take up to 48 hours)
|
||||
|
||||
3. **User installs WordPress plugin**
|
||||
- Plugin activation screen shows
|
||||
- User enters API key
|
||||
- Plugin connects to backend
|
||||
|
||||
4. **Plugin checks status** (GET /api/v1/plugin/status)
|
||||
- Backend returns site status: `pending`
|
||||
- Plugin shows "Site not verified" message
|
||||
- Plugin displays DNS instructions if not verified
|
||||
|
||||
5. **User verifies site** (POST /api/v1/sites/{id}/verify)
|
||||
- User clicks "Verify Domain" in plugin or dashboard
|
||||
- No request body needed (empty POST)
|
||||
- Backend performs DNS TXT lookup for domain
|
||||
- Backend checks for record: `maplepress-verify={verification_token}`
|
||||
- If found: Site transitions `pending` → `active`, `IsVerified` set to `true`
|
||||
- If not found: Returns error with DNS troubleshooting instructions
|
||||
|
||||
6. **Plugin can now sync** (POST /api/v1/plugin/sync)
|
||||
- Verification check passes
|
||||
- Pages are synced and indexed
|
||||
- Search functionality enabled
|
||||
|
||||
## 9. Architectural Design Decisions
|
||||
|
||||
### Why Pending Sites Can Access API
|
||||
|
||||
From `site.go:127-140`, the design allows pending sites to:
|
||||
- Check their status
|
||||
- View usage statistics
|
||||
- Prepare for verification
|
||||
|
||||
This is a **deliberate UX decision** to allow:
|
||||
1. Plugin to be activated immediately
|
||||
2. Admin to see connection status
|
||||
3. Admin to complete verification steps
|
||||
4. Smoother onboarding experience
|
||||
|
||||
### Why DNS Verification is Required
|
||||
|
||||
DNS verification prevents:
|
||||
- **Domain squatting**: Claiming domains you don't own
|
||||
- **Abuse**: Indexing content from sites you don't control
|
||||
- **Impersonation**: Pretending to be another site
|
||||
- **Unauthorized access**: Using the service without permission
|
||||
|
||||
DNS TXT record verification is the industry standard because:
|
||||
- **Proves domain control**: Only someone with DNS access can add TXT records
|
||||
- **Widely recognized**: Same method used by Google Search Console, Cloudflare, etc.
|
||||
- **Cannot be spoofed**: Requires actual access to domain registrar
|
||||
- **Automatic verification**: Backend can verify ownership without manual review
|
||||
|
||||
### Test Mode Rationale
|
||||
|
||||
Test mode (`test_sk_` keys) bypasses verification to enable:
|
||||
- Local development without DNS
|
||||
- Integration testing in CI/CD
|
||||
- Staging environments
|
||||
- Development workflows
|
||||
|
||||
## 10. Security Considerations
|
||||
|
||||
### Token Security
|
||||
|
||||
1. **Generation**:
|
||||
- Cryptographically secure random generation
|
||||
- 128-bit entropy (sufficient for this use case)
|
||||
- Base64 URL-safe encoding
|
||||
|
||||
2. **Storage**:
|
||||
- Stored in database as plain text (used in DNS TXT record)
|
||||
- Cleared after successful verification
|
||||
- Only accessible to authenticated tenant
|
||||
|
||||
3. **DNS Verification Security**:
|
||||
- DNS TXT records are public (as intended)
|
||||
- Token is meaningless without backend verification
|
||||
- 10-second timeout on DNS lookups prevents DoS
|
||||
- Token cleared after verification prevents reuse
|
||||
|
||||
### Attack Vectors Mitigated
|
||||
|
||||
1. **Domain Squatting**: DNS verification proves domain ownership
|
||||
2. **Token Guessing**: 128-bit entropy makes brute force infeasible
|
||||
3. **Token Reuse**: Token cleared after successful verification
|
||||
4. **Man-in-the-Middle**: HTTPS required for all API calls
|
||||
5. **DNS Spoofing**: Uses multiple DNS resolvers and validates responses
|
||||
6. **DNS Cache Poisoning**: 10-second timeout limits attack window
|
||||
|
||||
## 11. API Documentation
|
||||
|
||||
See individual endpoint documentation:
|
||||
- [Create Site](./API/create-site.md) - Initial site creation
|
||||
- [Verify Site](./API/verify-site.md) - Site verification endpoint
|
||||
- [Plugin Status](./API/plugin-verify-api-key.md) - Check verification status
|
||||
- [Sync Pages](./API/plugin-sync-pages.md) - Requires verification
|
||||
|
||||
## 12. WordPress Plugin Integration
|
||||
|
||||
The WordPress plugin should:
|
||||
|
||||
1. **On Activation**:
|
||||
- Prompt user for API key
|
||||
- Connect to backend
|
||||
- Check verification status
|
||||
|
||||
2. **If Not Verified**:
|
||||
- Display DNS TXT record instructions
|
||||
- Show the exact TXT record to add: `maplepress-verify={token}`
|
||||
- Provide domain registrar examples (GoDaddy, Namecheap, Cloudflare)
|
||||
- Explain DNS propagation timing (5-10 minutes)
|
||||
- Provide "Verify Domain" button
|
||||
- Disable sync/search features
|
||||
|
||||
3. **Verification Process**:
|
||||
- User clicks "Verify Domain"
|
||||
- Plugin calls POST /api/v1/sites/{id}/verify (no body)
|
||||
- Backend performs DNS TXT lookup
|
||||
- If successful: Enable all features
|
||||
- If failed: Show specific DNS error (record not found, timeout, etc.)
|
||||
|
||||
4. **After Verification**:
|
||||
- Enable all features
|
||||
- Allow page synchronization
|
||||
- Enable search functionality
|
||||
- Hide verification prompts
|
||||
|
||||
5. **Error Handling**:
|
||||
- Handle 403 "site is not verified" gracefully
|
||||
- Guide user to DNS verification process
|
||||
- Show DNS troubleshooting tips (check propagation, verify record format)
|
||||
- Retry verification status check
|
||||
|
||||
## 13. Database Schema
|
||||
|
||||
### Site Table Fields
|
||||
|
||||
```
|
||||
sites_by_id:
|
||||
- id (UUID, primary key)
|
||||
- tenant_id (UUID)
|
||||
- status (text: pending|active|inactive|suspended|archived)
|
||||
- is_verified (boolean)
|
||||
- verification_token (text, sensitive)
|
||||
- ...
|
||||
```
|
||||
|
||||
### Indexes Required
|
||||
|
||||
No special indexes needed for DNS verification - uses existing site_id and tenant_id lookups.
|
||||
|
||||
## 14. Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **DNS TXT record not found**:
|
||||
- Check DNS propagation status (use dig or nslookup)
|
||||
- Verify record format: `maplepress-verify={exact_token}`
|
||||
- Wait 5-10 minutes for DNS propagation
|
||||
- Check that TXT record was added to correct domain/subdomain
|
||||
- Verify no typos in the verification token
|
||||
|
||||
2. **DNS lookup timeout**:
|
||||
- Check domain's DNS servers are responding
|
||||
- Verify domain is properly registered
|
||||
- Check for DNS configuration issues
|
||||
- Try again after DNS stabilizes
|
||||
|
||||
3. **Site stuck in pending**:
|
||||
- Verify DNS TXT record is correctly set
|
||||
- Call verification endpoint: POST /api/v1/sites/{id}/verify
|
||||
- Check logs for DNS lookup errors
|
||||
- Use DNS checking tools (dig, nslookup) to verify record
|
||||
|
||||
4. **Test mode not working**:
|
||||
- Verify API key starts with `test_sk_`
|
||||
- Check `IsTestMode()` logic in site.go:115-125
|
||||
- Test mode sites skip DNS verification entirely
|
||||
|
||||
5. **DNS verification fails**:
|
||||
- Token may have been cleared (already verified)
|
||||
- DNS record format incorrect
|
||||
- Wrong domain or subdomain
|
||||
- Check error logs for specific DNS errors
|
||||
|
||||
### Debug Commands
|
||||
|
||||
```bash
|
||||
# Check DNS TXT record manually
|
||||
dig TXT example.com
|
||||
nslookup -type=TXT example.com
|
||||
|
||||
# Check site status
|
||||
curl -X GET http://localhost:8000/api/v1/sites/{id} \
|
||||
-H "Authorization: JWT {token}"
|
||||
|
||||
# Verify site via DNS
|
||||
curl -X POST http://localhost:8000/api/v1/sites/{id}/verify \
|
||||
-H "Authorization: JWT {token}"
|
||||
```
|
||||
|
||||
## 15. Future Enhancements
|
||||
|
||||
Potential improvements to the verification system:
|
||||
|
||||
1. **Token Expiration**:
|
||||
- Add 24-48 hour expiration for verification tokens
|
||||
- Allow token regeneration
|
||||
- Email token to site admin
|
||||
|
||||
2. **Alternative Verification Methods**:
|
||||
- Meta tag verification (alternative to DNS)
|
||||
- File upload verification (.well-known/maplepress-verify.txt)
|
||||
- WordPress plugin automatic verification (callback endpoint)
|
||||
|
||||
3. **Automatic Re-verification**:
|
||||
- Periodic DNS checks to ensure domain ownership hasn't changed
|
||||
- Alert if DNS record is removed
|
||||
- Grace period before disabling site
|
||||
|
||||
4. **Verification Audit Log**:
|
||||
- Track when site was verified
|
||||
- Record who performed verification
|
||||
- Log IP address and timestamp
|
||||
- DNS lookup results and timing
|
||||
Loading…
Add table
Add a link
Reference in a new issue