Run Workers on CircleCI Cloud

Platform: CircleCI Cloud (managed SaaS) | Database: PostgreSQL | Difficulty: Easy-Medium

This guide is specifically for CircleCI Cloud (the managed service at circleci.com), NOT self-hosted runners. If you want to use self-hosted runners, see CircleCI Self-Hosted Runner Guide.

---

How CircleCI Cloud Works for Workers

CircleCI Cloud runs jobs in ephemeral Docker containers that are destroyed after the job ends. This is fundamentally different from a traditional background worker:

Feature	Traditional Worker	CircleCI Cloud
Lifetime	Runs forever	Max 5 hours per job
Storage	Persistent disk	Ephemeral (lost after job)
Database	Local or network	Must be external (network-accessible)
Redis	Local or network	Must be external
Cost	Server rental	Compute minutes

Bottom line: CircleCI Cloud workers run in short bursts and connect to external services. This is actually perfect for OpenCodeHub’s architecture because the worker code is stateless — all state lives in PostgreSQL + Redis.

---

Prerequisites

Before you start, you need:

1. CircleCI Account (free tier works)
2. External PostgreSQL database (required — local DB won’t persist)
3. External Redis (required — local Redis won’t persist)
4. Git repository pushed to GitHub/GitLab/Bitbucket (CircleCI connects to it)

Recommended PostgreSQL Providers

Provider	Free Tier	Best For
Supabase	500MB, 2GB egress	Full-featured, good dashboard
Neon	500MB, 3 databases	Serverless, scales to zero
Railway	$5 credit	Simple, pay-as-you-go
Render	1GB	Easy setup
AWS RDS	None	Enterprise

Recommended Redis Providers

Provider	Free Tier	Best For
Upstash	10,000 commands/day	Serverless, HTTP API
Redis Cloud	30MB	Managed Redis
Railway	$5 credit	Easy setup
KeyDB	Self-host	Performance

---

Step 1: Set Up External PostgreSQL

Using Supabase (Recommended)

1. Go to supabase.com and create a project
2. In Project Settings → Database, copy the connection string:

   postgresql://postgres:[password]@db.xxxxxxxxxx.supabase.co:5432/postgres

3. Create the opencodehub database:

   CREATE DATABASE opencodehub;

Using Neon

1. Go to neon.tech and create a project
2. Copy the connection string from the dashboard
3. Create the database:

   CREATE DATABASE opencodehub;

---

Step 2: Set Up External Redis

Using Upstash (Recommended for Serverless)

1. Go to upstash.com and create a Redis database
2. Copy the Redis URL:

   rediss://default:[password]@xxxx.upstash.io:6379

3. Enable TLS (Upstash requires it)

---

Step 3: Configure CircleCI Project

3.1 Connect Repository

1. Go to app.circleci.com
2. Click Projects → Set Up Project
3. Select your OpenCodeHub repository
4. CircleCI will detect .circleci/config.yml

3.2 Create a Context (Recommended)

Contexts let you share environment variables across jobs securely.

1. Go to Organization Settings → Contexts
2. Click Create Context
3. Name it: opencodehub-production
4. Add these environment variables:

Variable	Value	Description
DATABASE_URL	postgresql://...	Your PostgreSQL connection string
DATABASE_DRIVER	postgres	Explicitly set driver
REDIS_URL	redis://... or rediss://...	Your Redis URL
JWT_SECRET	openssl rand -hex 32	Generate this
SESSION_SECRET	openssl rand -hex 32	Generate this
INTERNAL_HOOK_SECRET	openssl rand -hex 32	Generate this
CRON_SECRET	openssl rand -hex 32	Generate this
RUNNER_SECRET	openssl rand -hex 32	Generate this
WORKFLOW_SECRET_ENCRYPTION_KEY	openssl rand -hex 32	Generate this
SITE_URL	https://git.yourdomain.com	Your OpenCodeHub URL
SKIP_REDIS_CHECK	1	Skip Redis check during build

Security: Never commit these values to git. Always use CircleCI contexts or project environment variables.

3.3 Alternative: Project-Level Environment Variables

If you don’t want to use contexts:

1. Go to Project Settings → Environment Variables
2. Add each variable individually

---

Step 4: Initialize Database Schema

Before the worker can run, the database schema must exist. Run this once:

# On your local machine with DATABASE_URL set
export DATABASE_URL="postgresql://postgres:[password]@db.xxxxxxxxxx.supabase.co:5432/opencodehub"

# Install dependencies
bun install

# Push schema to database
bun run db:push

# Or generate and run migrations
bun run db:generate
bun run db:migrate

# Create admin user
bun run scripts/seed-admin.ts

Important: Do this from your local machine or a one-time job. The CircleCI worker job won’t create the schema automatically.

---

Step 5: Understanding the CircleCI Cloud Config

The .circleci/config.yml in your repo has multiple approaches:

Option A: Scheduled Tasks (Recommended for Free Plan)

Jobs run on a cron schedule. Each job starts, does its work, and exits.

workflows:
  scheduled-merge-queue:
    triggers:
      - schedule:
          cron: "*/15 * * * *"  # Every 15 minutes
    jobs:
      - merge-queue:
          context: opencodehub-production

What happens:

1. CircleCI starts a Docker container every 15 minutes
2. Installs Bun and dependencies
3. Runs bun run scripts/worker.ts for 30 minutes
4. Processes all pending merge queue items
5. Container is destroyed

Pros: Works on free plan, reliable, no timeouts Cons: 15-minute delay between processing (jobs only run on schedule)

Option B: Continuous Worker (Performance Plan)

One long-running job that stays active until timeout.

workflows:
  worker-continuous:
    jobs:
      - worker:
          context: opencodehub-production

What happens:

1. CircleCI starts a container
2. Worker runs continuously, processing items every 5 seconds
3. Job runs until the 5-hour timeout
4. CircleCI restarts the job automatically

Pros: Near real-time processing Cons: Requires Performance plan (5h jobs), uses more compute minutes

Option C: API-Triggered (Event-Driven)

OpenCodeHub calls CircleCI API when events happen.

In your OpenCodeHub webhook handler:

// When merge queue needs processing
async function triggerCircleCIWorker(eventType: string, repoId: string) {
  const response = await fetch(
    `https://circleci.com/api/v2/project/gh/${org}/${repo}/pipeline`,
    {
      method: "POST",
      headers: {
        "Circle-Token": process.env.CIRCLECI_API_TOKEN!,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        branch: "main",
        parameters: {
          event_type: eventType,
          repo_id: repoId,
        },
      }),
    }
  );
  return response.json();
}

Pros: Only runs when needed, most efficient Cons: Requires code changes in OpenCodeHub, more complex setup

---

Step 6: Choose Your Plan

Free Plan (Recommended to Start)

workflows:
  scheduled-merge-queue:
    triggers:
      - schedule:
          cron: "*/15 * * * *"  # Every 15 min
    jobs:
      - merge-queue:
          context: opencodehub-production

Monthly usage: ~2,880 minutes (48 runs/day × 30 days × 2 min avg) Free tier: 6,000 minutes/month ✅

Performance Plan (For Production)

workflows:
  worker-continuous:
    jobs:
      - worker:
          context: opencodehub-production

Monthly usage: ~7,200 minutes (5h × 24/day × 30 days) Performance plan: Included in price

---

Step 7: Database Connection from CircleCI Cloud

Since CircleCI Cloud containers are ephemeral, the database must be network-accessible.

Connection String Format

# Supabase
DATABASE_URL=postgresql://postgres:[password]@db.xxxxxxxxxx.supabase.co:5432/opencodehub

# Neon (with SSL)
DATABASE_URL=postgresql://[user]:[password]@ep-xxxxx.us-east-1.aws.neon.tech:5432/opencodehub?sslmode=require

# Railway
DATABASE_URL=postgresql://postgres:[password]@containers-xxxxx.railway.app:5432/railway

# AWS RDS (with SSL)
DATABASE_URL=postgresql://opencodehub:[password]@my-db.xxxxx.us-east-1.rds.amazonaws.com:5432/opencodehub?sslmode=require

SSL/TLS Configuration

Most managed PostgreSQL require SSL. The worker handles this automatically when sslmode=require is in the URL.

For Supabase, add this to your environment variables:

DATABASE_SSL=true
DATABASE_SSL_REJECT_UNAUTHORIZED=false  # Supabase uses self-signed certs

---

Step 8: Redis Connection from CircleCI Cloud

Upstash Redis URL

REDIS_URL=rediss://default:[password]@xxxx.upstash.io:6379

Note: rediss:// (with double s) = TLS enabled. Upstash requires TLS.

Redis Cloud

REDIS_URL=redis://default:[password]@redis-xxxxx.redis-cloud.com:12345

Testing Connection

You can test the connection from CircleCI by running the health-check job:

# Trigger manually from CircleCI dashboard
# or wait for the scheduled health check

---

Step 9: Monitoring

View Job Logs

1. Go to app.circleci.com
2. Select your project
3. Click on any job to see real-time logs

Set Up Notifications

1. Project Settings → Notifications
2. Connect Slack, email, or webhooks
3. Get alerts when jobs fail

Track Compute Usage

1. Plan → Usage
2. Monitor minutes used per month
3. Set up usage alerts

---

Step 10: Troubleshooting

Problem	Solution
”Database connection refused”	Your PostgreSQL doesn’t allow external connections. Check provider’s “Connection Pooling” or “Network” settings. For Supabase, go to Database → Connection Pooling.
”Redis connection timeout”	Redis provider blocks unknown IPs. Add CircleCI’s IP ranges to allowlist. Or use Upstash which doesn’t require IP allowlisting.
Job killed after 1 hour	Free plan default timeout is 1 hour. Add no_output_timeout: 5h or upgrade to Performance plan.
”Out of memory”	Use resource_class: large instead of medium. Or optimize the worker memory usage.
”bun install fails”	Network issues. The config has retry logic, but you can also try npm install as fallback.
”Schema doesn’t exist”	Run bun run db:push from your local machine first to create tables.
Worker starts but does nothing	Check that DATABASE_URL points to the right database and that the mergeQueueItems table has rows with status = 'queued'.
”SSL certificate verify failed”	Add DATABASE_SSL_REJECT_UNAUTHORIZED=false for providers with self-signed certs (Supabase).

---

Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                    CircleCI Cloud                           │
│  ┌─────────────────┐  ┌─────────────────┐                  │
│  │ Docker Container │  │ Docker Container │  ...            │
│  │ (every 15 min)   │  │ (every 15 min)   │                  │
│  │                  │  │                  │                  │
│  │ bun run worker.ts│  │ bun run worker.ts│                  │
│  │                  │  │                  │                  │
│  │ ┌─────────────┐  │  │ ┌─────────────┐  │                  │
│  │ │ Node.js/Bun │  │  │ │ Node.js/Bun │  │                  │
│  │ │ Worker Loop │  │  │ │ Worker Loop │  │                  │
│  │ └─────────────┘  │  │ └─────────────┘  │                  │
│  └────────┬─────────┘  └────────┬─────────┘                  │
└───────────┼─────────────────────┼────────────────────────────┘
            │                     │
            │  TCP/IP (Internet)  │
            ▼                     ▼
    ┌──────────────┐      ┌──────────────┐
    │  PostgreSQL   │      │    Redis     │
    │  (Supabase)   │      │  (Upstash)   │
    │               │      │              │
    │ merge_queue   │      │ sessions     │
    │ repositories  │      │ rate limits  │
    │ pull_requests │      │ job queues   │
    └──────────────┘      └──────────────┘

---

Quick Start Checklist

• Create PostgreSQL database (Supabase/Neon/Railway)
• Create Redis database (Upstash/Redis Cloud)
• Run bun run db:push from local machine
• Push OpenCodeHub code to GitHub
• Connect repo to CircleCI
• Create Context opencodehub-production with all env vars
• Verify .circleci/config.yml is in repo root
• Trigger first job manually from CircleCI dashboard
• Check job logs for ”✅ Worker started”
• Create a test PR and verify merge queue processes it

---

Next Steps

• CircleCI Integration Guide — For CI/CD pipelines (separate from worker)
• Deployment Guide — Full production setup
• Self-Hosted Runner Guide — If you prefer self-hosted over Cloud
• Merge Queue — How merge queue works