ClearlyOps Deployment Guide

This document covers deploying ClearlyOps for pilots and production.

Hostname Scheme

ClearlyOps uses explicit subdomains for security isolation:

Environment	UI	Sandbox API	Control-Plane API
dev	app-dev.clearlyops.com	sandbox-api-dev.clearlyops.com	api-dev.clearlyops.com
staging	app-staging.clearlyops.com	sandbox-api-staging.clearlyops.com	api-staging.clearlyops.com
prod	app.clearlyops.com	sandbox-api.clearlyops.com	api.clearlyops.com

• UI (app.*): Pilot web application for uploads, review, and reports
• Sandbox API (sandbox-api.*): Guarded pilot API with auth, quotas, and rate limits
• Control-Plane API (api.*): Future production API surface (stable contract)

CORS is configured per hostname: sandbox API only accepts requests from its corresponding UI hostname.

Architecture Overview

ClearlyOps consists of:

1. API Service - FastAPI application serving /api/v1/* endpoints
2. PostgreSQL Database - Event store, projections, and queue tables
3. Workers - Background processors for reconciliation and projections
4. Reverse Proxy (optional) - nginx for TLS termination and routing

Deployment Paths

Path A (Recommended): Cloud Run + Cloud SQL

This is the recommended path for GCP pilots. Minimal ops overhead with managed infrastructure.

┌─────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   Client    │────▶│  Cloud Run API   │────▶│   Cloud SQL     │
└─────────────┘     └──────────────────┘     │   (Postgres)    │
                            │                └─────────────────┘
                            │                        ▲
                    ┌───────┴────────┐               │
                    │  Cloud Run     │───────────────┘
                    │  Jobs/Workers  │
                    └────────────────┘

Prerequisites

• GCP project with billing enabled
• gcloud CLI installed and authenticated
• Artifact Registry repository created
• Cloud SQL instance provisioned
• Secrets created in GCP Secret Manager (see Secrets Management below)

Automated Deployment (Recommended)

Use the deployment script for consistent deployments:

# Set required environment variables
export GOOGLE_CLOUD_PROJECT=your-project-id

# Deploy to staging
./deploy/scripts/deploy-cloudrun.sh staging

# Deploy to production
./deploy/scripts/deploy-cloudrun.sh prod

# Dry run (show commands without executing)
./deploy/scripts/deploy-cloudrun.sh staging --dry-run

The script handles: - Building and pushing the Docker image - Deploying API service with Secret Manager integration - Deploying reconciliation and projector workers - Environment-specific configuration (CORS, log levels, instance counts)

Retention Cleanup Job (Sandbox)

Deploy the one-shot retention cleanup job and schedule it via Cloud Scheduler:

# Deploy the job
./deploy/scripts/deploy-retention-job.sh staging

# Run on-demand (dry-run)
gcloud run jobs execute clearlyops-retention-staging --region=us-central1 --args="--dry-run"

Schedule in Cloud Scheduler with a daily cadence (e.g., 02:00 UTC) to invoke the job.

Manual Steps

1. Build and push image: ```bash # Configure Docker for Artifact Registry gcloud auth configure-docker us-central1-docker.pkg.dev

# Build and push docker build -t us-central1-docker.pkg.dev/PROJECT/clearlyops/api:latest . docker push us-central1-docker.pkg.dev/PROJECT/clearlyops/api:latest ```

1. Create Cloud SQL instance (if not exists): ```bash gcloud sql instances create clearlyops-db \ --database-version=POSTGRES_15 \ --tier=db-f1-micro \ --region=us-central1 \ --no-assign-ip \ --network=default

gcloud sql databases create clearlyops --instance=clearlyops-db

gcloud sql users create clearlyops \ --instance=clearlyops-db \ --password=SECURE_PASSWORD ```

1. Deploy Cloud Run service: bash gcloud run deploy clearlyops-api \ --image=us-central1-docker.pkg.dev/PROJECT/clearlyops/api:latest \ --region=us-central1 \ --platform=managed \ --no-allow-unauthenticated \ --add-cloudsql-instances=PROJECT:us-central1:clearlyops-db \ --set-env-vars="APP_ENV=prod" \ --set-env-vars="DATABASE_URL=postgresql://clearlyops:PASS@/clearlyops?host=/cloudsql/PROJECT:us-central1:clearlyops-db" \ --set-env-vars="DATA_DIR=/data" \ --set-env-vars="CORS_ORIGINS=https://yourdomain.com" \ --set-env-vars="LOG_LEVEL=INFO" \ --min-instances=0 \ --max-instances=10

Security Note: Cloud Run Authentication

• --allow-unauthenticated: Makes the API publicly accessible without Cloud IAM checks

  • Use this when the API implements its own authentication layer (API keys, JWT tokens, etc.)
  • The API endpoints must enforce authorization at the application level
  • Suitable for public APIs with workspace-scoped access control

• --no-allow-unauthenticated: Requires Cloud IAM authentication to invoke the service

  • Use this for internal services or when clients can authenticate with GCP credentials
  • Clients must include a valid Google ID token in the Authorization: Bearer header
  • Suitable for service-to-service communication within GCP

Current deployment uses --no-allow-unauthenticated as the default secure posture. If your pilot requires public API access, change to --allow-unauthenticated and ensure the API implements proper authentication and authorization checks.

1. Deploy workers (as Cloud Run Jobs or always-on services):

Option A: Cloud Run Jobs (for batch processing): bash gcloud run jobs create projector-worker \ --image=us-central1-docker.pkg.dev/PROJECT/clearlyops/api:latest \ --region=us-central1 \ --add-cloudsql-instances=PROJECT:us-central1:clearlyops-db \ --set-env-vars="DATABASE_URL=..." \ --command="python,scripts/run_worker.py,--kind,projector"

Option B: Always-on Cloud Run service (for real-time processing): bash gcloud run deploy projector-worker \ --image=us-central1-docker.pkg.dev/PROJECT/clearlyops/api:latest \ --region=us-central1 \ --platform=managed \ --no-allow-unauthenticated \ --add-cloudsql-instances=PROJECT:us-central1:clearlyops-db \ --set-env-vars="DATABASE_URL=..." \ --command="python,scripts/run_worker.py,--kind,projector" \ --min-instances=1 \ --max-instances=1

Cloud Run Environment Variables

Variable	Required	Description
APP_ENV	Yes	dev, staging, or prod
DATABASE_URL	Yes	Cloud SQL connection string (inject via Secret Manager)
DATA_DIR	Yes	Data directory (use /data)
CORS_ORIGINS	Yes (prod)	Explicit origins (not *)
LOG_LEVEL	No	DEBUG, INFO, WARNING, ERROR
BLOB_STORE_BACKEND	No	local or gcs
GCS_BUCKET_NAME	If gcs	GCS bucket for blob storage
SANDBOX_MODE	No	Enable sandbox API (true or false)
SANDBOX_API_KEYS	If sandbox	Comma-separated keys (workspace_id=key)
SANDBOX_DEFAULT_WORKSPACE_ID	No	Default workspace for unmapped keys
SANDBOX_FREE_RUNS	No	Free runs per workspace (default: 10)
SANDBOX_MAX_ROWS	No	Max rows per CSV (default: 10000)
SANDBOX_MAX_FILE_BYTES	No	Max upload size (default: 10MB)
SANDBOX_MAX_COLUMNS	No	Max columns per CSV (default: 100)
SANDBOX_MAX_CONCURRENT_RUNS	No	Max in-flight runs per workspace
SANDBOX_RATE_LIMIT_PER_KEY	No	Requests per key per window
SANDBOX_RATE_LIMIT_PER_IP	No	Requests per IP per window
SANDBOX_RATE_LIMIT_WINDOW_SECONDS	No	Rate limit window in seconds
SANDBOX_RETENTION_DAYS	No	Data retention days (default: 30)
SANDBOX_RETENTION_ROLES	No	Artifact roles eligible for cleanup

---

Path B (Fallback): GCE VM + docker-compose

Use this for pilots that need more control, longer worker timeouts, or as a fallback.

┌─────────────┐     ┌─────────────────────────────────────────┐
│   Client    │────▶│  GCE VM                                 │
└─────────────┘     │  ┌─────────┐  ┌─────────┐  ┌─────────┐  │
                    │  │  nginx  │─▶│   API   │─▶│Cloud SQL│  │
                    │  │  :443   │  │  :8000  │  │(external)│  │
                    │  └─────────┘  └─────────┘  └─────────┘  │
                    │               ┌──────────────────────┐  │
                    │               │  Workers (projector, │  │
                    │               │  agent-executor)     │  │
                    │               └──────────────────────┘  │
                    └─────────────────────────────────────────┘

Prerequisites

• GCE VM (e2-small or larger) with Docker installed
• Firewall rules allowing 443 inbound only
• Cloud SQL instance (or local Postgres for dev)
• TLS certificates (certbot or GCP managed certs)

Steps

1. Provision VM: ```bash gcloud compute instances create clearlyops-pilot \ --zone=us-central1-a \ --machine-type=e2-small \ --image-family=cos-stable \ --image-project=cos-cloud \ --boot-disk-size=20GB \ --tags=clearlyops-api

# Allow HTTPS only gcloud compute firewall-rules create allow-clearlyops-https \ --allow=tcp:443 \ --target-tags=clearlyops-api ```

1. SSH into VM and clone repo: ```bash gcloud compute ssh clearlyops-pilot --zone=us-central1-a

git clone https://github.com/your-org/clearlyops.git cd clearlyops ```

1. Configure environment: ```bash cp deploy/config/clearlyops-api.env.example .env

# Edit .env with production values vim .env ```

Required changes for production: - APP_ENV=prod - DATABASE_URL=postgresql://... (Cloud SQL connection) - CORS_ORIGINS=https://yourdomain.com

1. Configure TLS (certbot example): ```bash # Install certbot sudo apt-get update && sudo apt-get install -y certbot

# Get certificate sudo certbot certonly --standalone -d api.yourdomain.com

# Copy certs to nginx dir mkdir -p deploy/nginx/certs sudo cp /etc/letsencrypt/live/api.yourdomain.com/fullchain.pem deploy/nginx/certs/ sudo cp /etc/letsencrypt/live/api.yourdomain.com/privkey.pem deploy/nginx/certs/ ```

1. Enable TLS in nginx config:
2. Edit deploy/nginx/conf.d/api.conf
3. Uncomment the TLS server block

4. Update docker-compose.yml to expose port 443 and mount certs

5. Start services: bash docker compose up -d docker compose logs -f

6. Verify deployment: bash curl -fsS https://api.yourdomain.com/health curl -fsS https://api.yourdomain.com/version curl -fsS https://api.yourdomain.com/api/v1/workspaces

---

Worker Configuration

Workers consume from the Postgres-backed queue and run as separate containers.

Worker	Subscription	Purpose
Projector	projector	Updates materialized views from events
Agent Executor	agent.execute	Executes approved agent plans

Running Workers

# Via docker-compose (recommended for VM)
docker compose up -d projector-worker agent-executor-worker

# Standalone (for testing)
DATABASE_URL=postgresql://... python scripts/run_worker.py --kind projector
DATABASE_URL=postgresql://... python scripts/run_worker.py --kind agent-executor

---

Verification

After deployment, use the automated verification script:

# Verify staging deployment (read-only checks)
./deploy/scripts/verify-staging.sh staging

# Verify with DNS/TLS hostname check
./deploy/scripts/verify-staging.sh staging --hostname-check

# Verify production
./deploy/scripts/verify-staging.sh prod --hostname-check

The verification script checks: - Cloud Run services exist and are running - Health and version endpoints respond - Secrets exist in Secret Manager - Cloud SQL instance exists and is running - Optional: DNS/TLS for configured hostnames

Manual verification (if needed):

# Health check
curl -fsS https://<host>/health
# Expected: {"status": "ok"}

# Version info
curl -fsS https://<host>/version
# Expected: {"version": "..."}

# API routes work
curl -fsS https://<host>/api/v1/workspaces
# Expected: 200 response (or 401 if auth required)

# API isolation (root should 404)
curl -fsS https://<host>/
# Expected: 404 with JSON error

---

Secrets Management

All sensitive configuration is stored in GCP Secret Manager and injected at runtime.

Required Secrets

Secret Name	Description	Example Value
clearlyops-database-url[-staging]	PostgreSQL connection string	postgresql://user:pass@/db?host=/cloudsql/...
clearlyops-sandbox-api-keys[-staging]	Comma-separated API keys	sk_sandbox_key1,sk_sandbox_key2
clearlyops-stripe-api-key[-staging]	Stripe API key (optional)	sk_live_... or sk_test_...
clearlyops-stripe-webhook-secret[-staging]	Stripe webhook secret (optional)	whsec_...

Setup Secrets

Use the setup script to create placeholder secrets:

export GOOGLE_CLOUD_PROJECT=your-project-id

# Create staging secrets
./deploy/scripts/setup-secrets.sh staging

# Create production secrets
./deploy/scripts/setup-secrets.sh prod

Update Secret Values

# Update database URL
echo 'postgresql://user:pass@/db?host=/cloudsql/project:region:instance' | \
  gcloud secrets versions add clearlyops-database-url-staging --data-file=- --project=$GOOGLE_CLOUD_PROJECT

# Update sandbox API keys (comma-separated for rotation support)
echo 'sk_sandbox_key1,sk_sandbox_key2' | \
  gcloud secrets versions add clearlyops-sandbox-api-keys-staging --data-file=- --project=$GOOGLE_CLOUD_PROJECT

Key Rotation

Sandbox API keys support rotation without downtime:

1. Generate a new key and add it to the comma-separated list
2. Update the secret: old_key,new_key
3. Deploy the service (picks up new secret version)
4. Migrate clients to the new key
5. Remove the old key: new_key
6. Update the secret and redeploy

Local Development

For local development, secrets are read from environment variables as a fallback. The infra/adapters/gcp_secrets.py module handles this automatically:

from infra.adapters.gcp_secrets import get_secret

# Tries GCP Secret Manager first, falls back to env var
api_key = get_secret("clearlyops-sandbox-api-keys", fallback_env_var="SANDBOX_API_KEYS")

---

Security Checklist

Before going live, verify:

• [ ] Only port 443 is exposed externally
• [ ] CORS_ORIGINS is set to explicit origins (not *)
• [ ] Secrets are stored in Secret Manager (not in env files or images)
• [ ] SANDBOX_API_KEYS is set (required in production when SANDBOX_MODE=true)
• [ ] Cloud SQL uses private IP (no public access)
• [ ] OS auto-updates are enabled (VM path)
• [ ] Logs are sent to Cloud Logging
• [ ] TLS certificates are valid and auto-renewing
• [ ] Database credentials are rotated periodically
• [ ] Rate limits are configured for sandbox API
• [ ] Sandbox data retention is configured (SANDBOX_RETENTION_DAYS, SANDBOX_RETENTION_ROLES)
• [ ] Retention cleanup job is scheduled (Cloud Scheduler/cron)

---

Troubleshooting

API not responding

# Check service health
docker compose ps
docker compose logs clearlyops-api

# Check database connectivity
docker compose exec clearlyops-api python -c "
import psycopg2
import os
conn = psycopg2.connect(os.environ['DATABASE_URL'])
print('Database connection OK')
conn.close()
"

Workers not processing

# Check worker logs
docker compose logs projector-worker

# Check queue depth
docker compose exec clearlyops-db psql -U clearlyops -c "
SELECT topic, COUNT(*) as pending
FROM queue_messages
WHERE claimed_at IS NULL
GROUP BY topic;
"

Database connection issues

• Verify DATABASE_URL is correct
• For Cloud SQL: ensure Cloud SQL Auth Proxy is running or use Unix socket
• Check firewall rules allow connection from VM to Cloud SQL

---

Local Development

For local development, use docker-compose with the local Postgres:

# Start all services
docker compose up -d

# Rebuild after code changes
docker compose build clearlyops-api
docker compose up -d clearlyops-api

# View logs
docker compose logs -f

# Stop all services
docker compose down

# Reset database (WARNING: destroys data)
docker compose down -v
docker compose up -d