Project Auto-Provisioning System

Overview

This system automatically provisions Integration Hub tokens and Analytics DB credentials for all projects in the system. Projects are stored in an external read-only pepcore database, so when new projects are created there, they need to be provisioned in this application.

Problem Solved

Before: Projects created in the pepcore database would exist without: - API tokens (Integration Hub tokens) - Analytics DB credentials

This caused errors when accessing integrations pages and prevented analytics data from being collected.

After: Every project is automatically provisioned within 5 minutes of creation, with daily reconciliation to catch any gaps.

Architecture

External Pepcore DB (read-only)
         ↓
   Polling Job (every 5 min) ← Optimized to skip if no changes
         ↓
   Provisioning Service
    ├─→ Generate API Token
    ├─→ Create Analytics DB
    └─→ Initialize Tables
         ↓
   Fully provisioned ✓

Components

1. ProjectProvisioningService (app/services/project_provisioning_service.rb)

Core service that handles all provisioning logic.

Key Methods: - provision_project(project) - Provisions a single project - provision_all_missing - Finds and provisions all incomplete projects - check_project_status(project) - Returns what's missing - audit_all_projects - Generates comprehensive status report

Features: - Idempotent (safe to run multiple times) - Individual project failures don't stop others - Comprehensive error handling and logging - Sentry integration for failures

2. ProjectProvisioningJob (app/jobs/project_provisioning_job.rb)

Background job that runs every 5 minutes to check for new projects.

Optimization: - Caches project count between runs - Skips provisioning if count unchanged - Runs safety check every 30 minutes even if count unchanged - Uses dedicated project_provisioning queue

Failsafes: - 5-minute timeout - Cleans up database connections - Continues on individual failures - Always schedules next run

3. ProjectReconciliationJob (app/jobs/project_reconciliation_job.rb)

Daily job (1 AM UTC) that performs deep scan of all projects.

Purpose: - Safety net to catch any gaps - Generates comprehensive daily report - Auto-fixes missing credentials - Alerts via Sentry if persistent issues

4. Rake Tasks (lib/tasks/project_provisioning.rake)

Command-line tools for management and monitoring.

Available Commands:

# Schedule the 5-minute polling job
rake projects:schedule_provisioning

# Schedule the daily reconciliation job
rake projects:schedule_reconciliation

# Manually provision all missing credentials
rake projects:provision_missing

# Audit status without making changes
rake projects:audit_status

# Provision a specific project
rake projects:provision_project[PROJECT_ID]

# Check job status
rake projects:job_status

# Clear scheduled jobs (use with caution)
rake projects:clear_scheduled_jobs

5. ActiveAdmin Interface

Batch Actions (select multiple projects): - "Provision Integration Hub + Analytics" - Manually trigger provisioning - "Check Provisioning Status" - See what's missing

Project Index Page: - "Provisioning Status" column with visual indicators - ✅/❌ indicators for API Token and Analytics DB

Project Show Page: - Provisioning status panel at top - "Re-provision Now" button if missing credentials

Setup Instructions

🚀 Cloud66 Job Scheduling (Recommended Approach)

Approach: Cloud66 Jobs (Instead of Self-Scheduling Rails Jobs)

We use Cloud66 Jobs to directly schedule the provisioning tasks instead of Rails job self-scheduling. This provides better visibility, control, and reliability.

Why Cloud66 Jobs Instead of Self-Scheduling: - ✅ Centralized Control - All scheduling visible in Cloud66 dashboard - ✅ Easy Management - Enable/disable via Cloud66 without code changes - ✅ Better Reliability - Cloud66 is more reliable than Rails job scheduling - ✅ Simpler Code - No complex self-scheduling logic needed - ✅ Better Monitoring - Execution history and success/failure tracking

Cloud66 Jobs Configuration Required:

Create two recurring jobs in Cloud66:

Job 1: Project Provisioning - Name: Project Provisioning - Type: Rake Task - Task: projects:provision_missing - Schedule: Every 5 minutes UTC - Server: Your main application server

Job 2: Project Reconciliation - Name: Project Reconciliation - Type: Rake Task - Task: projects:reconcile_daily - Schedule: Daily at 1:00 AM UTC - Server: Your main application server

How It Works: 1. Cloud66 runs rake tasks on schedule → Direct provisioning work 2. Rails jobs do the actual work → No self-scheduling complexity 3. Cloud66 handles all scheduling → Reliable and visible 4. Easy pause/resume → Control via Cloud66 dashboard

Benefits: - ✅ Simpler architecture - Cloud66 handles scheduling, Rails handles work - ✅ Better monitoring - All job status in Cloud66 dashboard - ✅ Easier debugging - Clear separation of concerns - ✅ No job queue dependency - Works even if delayed_job has issues

Initial Backfill (One-time)

After deploying, provision any existing projects with missing credentials:

bundle exec rake projects:provision_missing

Delayed Job Scheduling (No Cron Needed!)

The system uses self-scheduling Delayed Jobs (like the heartbeat system):

• Each job automatically schedules the next one
• No external cron required
• Managed by your existing Delayed Job workers
• Self-sustaining system

Monitoring

Logs

All provisioning activity is logged with clear prefixes:

🔍 [ProjectProvisioning] Starting provisioning poll...
✅ [ProjectProvisioning] Poll complete in 2.3s: Scanned=42, Missing=3, Provisioned=3, Failed=0
📊 [ProjectReconciliation] Starting daily reconciliation...

Sentry Integration

Failures are automatically reported to Sentry with full context: - Project ID and name - Error type and message - Operation that failed (provisionapitoken or provisionanalyticsdb)

Check Job Status

rake projects:job_status

Shows: - Polling job status and next run time - Daily job status and next run time - Any failed jobs

Audit Projects

rake projects:audit_status

Shows: - Total projects - Fully provisioned count and percentage - Projects missing API tokens - Projects missing DB credentials - Projects missing both - Detailed list of projects with issues

Usage Examples

Check if a specific project is provisioned

rake projects:provision_project[87]

Output:

```

🔧 PROVISIONING PROJECT 87

Name: Team ML09x Project Key: ML09X

Current Status: API Token: ❌ Missing Analytics DB: ❌ Missing

🔧 Provisioning...

✅ SUCCESS API Token: created

Analytics DB: created

### Audit all projects

```bash
rake projects:audit_status

Manually provision missing projects

rake projects:provision_missing

Via ActiveAdmin

1. Go to Projects page
2. Select projects needing provisioning
3. Choose "Provision Integration Hub + Analytics" from batch actions
4. Click "Submit"

Error Handling

Level 1: Individual Project Failures

• Each project provision is wrapped in rescue block
• One failure doesn't stop others
• Error logged with project ID
• Reported to Sentry

Level 2: Service Failures

• Service returns success/failure status
• Job logs failure but continues
• Job always schedules next run

Level 3: Job Failures

• Delayed Job allows 1 retry
• 20-minute timeout prevents hanging
• Failed jobs kept in DB for inspection
• Can retry via ActiveAdmin

Critical Failsafes

• Never runs in main request cycle
• Always in background jobs
• Database connection pool management
• PepcoreBase.clear_active_connections! cleanup

Troubleshooting

Jobs not running

# Check job status
rake projects:job_status

# If not scheduled, schedule them
rake projects:schedule_provisioning
rake projects:schedule_reconciliation

Projects still missing credentials

# Run audit to see what's missing
rake projects:audit_status

# Manually provision missing
rake projects:provision_missing

# Check logs for errors
tail -f log/production.log | grep ProjectProvisioning

Database connection issues

The jobs automatically clean up connections, but if you see connection pool exhaustion:

# In Rails console
PepcoreBase.clear_active_connections!

Provision a specific project manually

# Via rake task
rake projects:provision_project[PROJECT_ID]

# Or via Rails console
service = ProjectProvisioningService.new
result = service.provision_project(Project.find(PROJECT_ID))
puts result.inspect

Performance

Optimization Results: - Polling job runs in < 1 second when no changes detected - Full provisioning of 100 projects: ~15-30 seconds - Single project provision: ~2-3 seconds - Daily reconciliation: ~30-60 seconds (depends on project count)

Resource Usage: - Minimal DB load (cached count check) - Dedicated queue prevents blocking other jobs - Connection cleanup prevents pool exhaustion

Testing

Run the audit first to see current state: bash rake projects:audit_status

Then test provisioning a single project: bash rake projects:provision_project[PROJECT_ID]

Finally, test the full system: bash rake projects:provision_missing

Success Metrics

✅ 100% of projects have API tokens within 5 minutes of creation
✅ 100% of projects have Analytics DB within 5 minutes
✅ Zero provision failures (or < 0.1% failure rate)
✅ Job execution time < 30 seconds
✅ Zero impact on main application performance
✅ Full visibility via ActiveAdmin

Support

For issues or questions: 1. Check logs: tail -f log/production.log | grep ProjectProvisioning 2. Run audit: rake projects:audit_status 3. Check Sentry for errors 4. Review job status: rake projects:job_status