Hypertext Rails

Documentation

Getting Started

Communication Center

Procore / Project Groups

Other Features

Twilio SMS Integration - Complete Guide

Complete documentation for Twilio SMS integration in PepTalk, including setup, configuration, testing, and troubleshooting.

Table of Contents

  1. Overview
  2. Architecture
  3. Installation & Setup
  4. Configuration
  5. Phone Number Handling
  6. Testing Guide
  7. Usage & API
  8. Sandbox Modes
  9. Troubleshooting
  10. Compliance & Security

Overview

PepTalk integrates with Twilio to send SMS notifications to stakeholders. The system handles: - Phone number normalization to E.164 format - SMS consent management with audit trails - Cadence limit enforcement - Delivery status tracking - Opt-out handling (STOP/START keywords) - Webhook processing for status updates

Key Features

  • ✅ Automatic phone number normalization
  • ✅ SMS consent management (ISO27001 compliant)
  • ✅ Cadence rule enforcement
  • ✅ Delivery status tracking via webhooks
  • ✅ Automatic opt-out handling
  • ✅ Sandbox modes for testing
  • ✅ Fallback channel support

Architecture

Components

  1. PhoneNormalizationService - Normalizes phone numbers to E.164 format (+{country_code}{number})
  2. TwilioService - Wrapper for Twilio API (send SMS, validate webhooks)
  3. SmsDeliverySendJob - Background job to send SMS messages
  4. Twilio::WebhooksController - Handles delivery status and opt-out webhooks
  5. ConsentAuditLog - Immutable audit trail for consent changes (ISO27001 compliance)
  6. PhoneCodeHelper - Shared utilities for phone code extraction and country lookup

Data Flow

1. Template with SMS variant created
2. DeliveryScheduleService creates CommsDelivery (channel: 'sms')
3. TemplateVariableReplacementJob resolves variables → triggers SmsDeliverySendJob
4. SmsDeliverySendJob:
   - Checks consent & opt-out status
   - Validates cadence limits
   - Normalizes phone number
   - Sends via TwilioService
   - Updates delivery with external_message_id
5. Twilio webhooks update delivery status asynchronously
6. Inbound STOP/START handled via webhooks

Installation & Setup

Step 1: Install Dependencies

bundle install

This installs: - twilio-ruby (~> 6.0) - Twilio API client - Note: phonelib was removed - we use COUNTRIES data for phone normalization

Step 2: Run Database Migrations

rails db:migrate

Required migrations: - add_sms_fields_to_stakeholders - Adds phone_normalized, sms_consent_given_at, sms_consent_revoked_at, country_code - add_external_tracking_to_comms_deliveries - Adds external_message_id, external_status, error_code, error_message - create_consent_audit_logs - Creates audit log table for compliance

Step 3: Configure Twilio Credentials

Add to config/application.yml:

# Production credentials
TWILIO_ACCOUNT_SID: ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN: your_auth_token_here
TWILIO_FROM_NUMBER: +13349663045

# Optional: Use Messaging Service instead of single number
# TWILIO_MESSAGING_SERVICE_SID: MGxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Sandbox configuration (see Sandbox Modes section)
TWILIO_SANDBOX: 1  # Set to 0 for production
# TWILIO_SANDBOX_FROM_NUMBER: "+14155238886"  # For Messaging Sandbox
TWILIO_TEST_ACCOUNT_SID: ACxx...  # For test credentials sandbox
TWILIO_TEST_AUTH_TOKEN: xxx

Step 4: Configure Webhooks

Option A: Configure via API (Recommended)

# For production
rails twilio:configure_webhooks[https://app.peptalk.com]

# For local testing with ngrok
# 1. Start ngrok: ngrok http 3000
# 2. Configure webhooks:
rails twilio:configure_webhooks[https://YOUR_ID.ngrok.io]

What this does: - Automatically finds your phone number in Twilio - Configures status callback: {base_url}/webhooks/twilio/status - Configures inbound message: {base_url}/webhooks/twilio/inbound - All done via API - no console access needed!

View current configuration: bash rails twilio:show_config

Option B: Configure via Twilio Console

  1. Log in to Twilio Console
  2. Navigate to Phone Numbers → Manage → Active Numbers
  3. Click on your phone number
  4. Set webhook URLs:
    • Status Callback URL: https://app.peptalk.com/webhooks/twilio/status
    • A Message Comes In: https://app.peptalk.com/webhooks/twilio/inbound
  5. Click Save

Configuration

Environment Variables

Variable Required Description
TWILIO_ACCOUNT_SID Yes Your Twilio Account SID
TWILIO_AUTH_TOKEN Yes Your Twilio Auth Token
TWILIO_FROM_NUMBER Yes* Phone number to send from (E.164 format)
TWILIO_MESSAGING_SERVICE_SID No* Messaging Service SID (alternative to FROM_NUMBER)
TWILIO_SANDBOX No Set to 1 to enable sandbox mode
TWILIO_SANDBOX_FROM_NUMBER No Sandbox number for Messaging Sandbox mode
TWILIO_TEST_ACCOUNT_SID No Test Account SID for test credentials sandbox
TWILIO_TEST_AUTH_TOKEN No Test Auth Token for test credentials sandbox

*Either TWILIO_FROM_NUMBER or TWILIO_MESSAGING_SERVICE_SID must be set.

Verify Configuration

rails twilio:verify

This checks: - ✅ Account SID configured - ✅ Auth Token configured - ✅ From Number or Messaging Service configured - ✅ TwilioService can initialize

Phone Number Handling

Phone Number Storage

The system stores phone numbers in two formats:

  1. phone - E.164 formatted phone number (e.g., +639621731584)
  2. phone_normalized - Same as phone (for consistency)
  3. country_code - Stores phone code (e.g., 63, 971, 1) NOT ISO2 code

Important: country_code field stores the phone code (numeric), not the ISO2 country code (e.g., "PH", "AE").

Phone Normalization

All phone numbers are normalized to E.164 format (+{country_code}{number}) using PhoneNormalizationService.

Supported input formats: - +639621731584 (E.164 - already normalized) - 639621731584 (without +) - +63 962 173 1584 (with spaces) - (334) 966-3045 (US format) - 09123456789 (local format with leading zero)

Normalization process: 1. Removes all non-digit characters except + 2. If starts with +, validates and returns 3. If no +, uses country_code field to build E.164 format 4. Removes leading zeros 5. Validates length (7-15 digits after country code)

Example: ```ruby

Input: "09123456789" with country_code: "63"

Output: "+639123456789"

PhoneNormalizationService.normalize("09123456789", country: "PH")

=> "+639123456789"


### Country Code Selection

Users select country from a dropdown in stakeholder forms. The form:
- Displays: `🇵🇭 Philippines (+63)`
- Sends: Phone code `63` (not ISO2 "PH")
- Stores: Phone code `63` in `country_code` field

**Helper methods:**
- `stakeholder.country_phone_code` - Returns phone code directly
- `stakeholder.country_iso2` - Converts phone code to ISO2 (for display)
- `stakeholder.country_name` - Gets country name from phone code

---

## Testing Guide

### Quick Start (No Webhooks Required)

You can test SMS sending immediately without configuring webhooks. Webhooks are only needed for status updates and inbound messages.

### Step 1: Verify Credentials

```bash
rails twilio:verify

Step 2: Test Phone Normalization

# Test with various formats
rails twilio:test_normalize["(334) 966-3045"]
rails twilio:test_normalize[3349663045]
rails twilio:test_normalize[+13349663045]

# Or use environment variable (recommended for special characters)
TEST_PHONE="(334) 966-3045" rails twilio:test_normalize

Expected output: ✅ Normalized: +13349663045 Valid: Yes

Step 3: Send Test SMS

# Replace with your test phone number
rails twilio:test_send[+13349663045]
rails twilio:test_send["+63 962 173 1584"]

# Or use environment variable
TEST_PHONE_NUMBER="+63 962 173 1584" rails twilio:test_send

Expected output: ✅ SMS sent successfully! Message SID: SMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Status: queued

Step 4: Test Full Workflow

# Test complete workflow (creates delivery, sends SMS)
rails twilio:test_workflow_sms[+13349663045]

Testing in Rails Console

# Test TwilioService
service = TwilioService.new
result = service.send_sms(
  to: '+13349663045',
  body: 'Hello from PepTalk!'
)
# => { success: true, message_sid: "SM...", status: "queued", ... }

# Test phone normalization
PhoneNormalizationService.normalize('(334) 966-3045')
# => "+13349663045"

PhoneNormalizationService.valid?('invalid')
# => false

Testing with Webhooks (Optional)

For local development:

# Terminal 1: Start Rails
rails server

# Terminal 2: Start ngrok
ngrok http 3000

# Terminal 3: Configure webhooks with ngrok URL
rails twilio:configure_webhooks[https://abc123.ngrok.io]

# Send test SMS
rails twilio:test_send[+13349663045]

# Check webhook logs
tail -f log/development.log | grep TwilioWebhook

Usage & API

Sending SMS

SMS deliveries are created automatically by workflows when: 1. Template has sms_text variant 2. Stakeholder has preferred_channel: 'sms' or fallback to SMS 3. Stakeholder has valid phone and consent

# SMS is sent automatically via workflow
workflow = AutomationWorkflow.find(id)
service = DeliveryScheduleService.new(workflow)
result = service.create_deliveries
# → Creates deliveries with channel: 'sms'
# → TemplateVariableReplacementJob → SmsDeliverySendJob → sends via Twilio

Managing Consent

stakeholder = Stakeholder.find(id)

# Grant consent
stakeholder.grant_sms_consent!(
  performed_by: current_admin_user,
  source: 'admin_ui',
  metadata: { ip_address: request.remote_ip }
)

# Revoke consent
stakeholder.revoke_sms_consent!(
  performed_by: current_admin_user,
  source: 'admin_ui',
  metadata: { reason: 'User request' }
)

# Check consent
stakeholder.has_sms_consent? # => true/false

Opt-Out Handling

Automatic via inbound webhooks: - STOP/UNSUBSCRIBE/CANCEL/END/QUIT → Opt out - START/UNSTOP/YES → Opt in - HELP/INFO → Show help message

All opt-out actions are logged in consent_audit_logs.

Sandbox Modes

The system supports three modes for testing and development:

Mode 1: Production (Real SMS)

Configuration: yaml TWILIO_SANDBOX: 0 # or remove TWILIO_SANDBOX TWILIO_ACCOUNT_SID: ACxx... # Live credentials TWILIO_AUTH_TOKEN: xxx TWILIO_FROM_NUMBER: +13349663045

Behavior: - ✅ Real SMS sent to any valid phone number - ✅ Uses Twilio credits - ✅ Full delivery tracking - ⚠️ Trial accounts can only send to verified numbers

Mode 2: Messaging Sandbox (Real SMS to Joined Numbers)

Use case: Can't add your number in Console (e.g., trial) but want real SMS on your phone.

Configuration: yaml TWILIO_SANDBOX: 1 TWILIO_SANDBOX_FROM_NUMBER: "+14155238886" # Sandbox number from Console TWILIO_ACCOUNT_SID: ACxx... # Live credentials TWILIO_AUTH_TOKEN: xxx

Setup: 1. Twilio Console → Messaging → Try it out → Send an SMS → Sandbox 2. Note sandbox number (e.g., +14155238886) and join code (e.g., join abcd-efgh) 3. Text join <your-code> to sandbox number 4. Configure TWILIO_SANDBOX_FROM_NUMBER in application.yml

Behavior: - ✅ Real SMS sent to numbers that joined sandbox - ✅ Uses Twilio credits - ✅ No Console verification needed - ❌ Only works for numbers that joined sandbox - ⚠️ Error 63015 = recipient hasn't joined

Mode 3: Test Credentials Sandbox (No Real SMS)

Use case: Verify app flow only - no real SMS, no charges.

Configuration: ```yaml TWILIO_SANDBOX: 1

DO NOT set TWILIOSANDBOXFROM_NUMBER

TWILIOTESTACCOUNTSID: ACxx... # Test credentials TWILIOTESTAUTHTOKEN: xxx ```

Setup: 1. Twilio Console → Account → API keys & tokens → Test credentials 2. Copy Test Account SID and Test Auth Token 3. Configure in application.yml

Behavior: - ✅ API accepts messages (returns success) - ✅ No real SMS delivered - ✅ No charges - ✅ Good for CI/local testing - ⚠️ Uses magic From number: +15005550006

Which Mode to Use?

Goal Mode Config
Real SMS from live number Production TWILIO_SANDBOX: 0
Real SMS, no Console verification Messaging Sandbox TWILIO_SANDBOX: 1 + TWILIO_SANDBOX_FROM_NUMBER
No real SMS, no charges Test Credentials TWILIO_SANDBOX: 1 + TWILIO_TEST_* (no sandbox From)

For Automation Workflows: - Sandbox enabled (TWILIO_SANDBOX=1): All SMS uses sandbox (Messaging Sandbox or Test Credentials) - Sandbox disabled (TWILIO_SANDBOX=0): All SMS uses production credentials

Troubleshooting

Common Issues

"Twilio credentials missing"

  • Check TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN are set
  • Verify credentials are correct in Twilio Console
  • Restart Rails server after adding credentials

"Invalid or missing phone number"

  • Ensure stakeholder has phone field populated
  • Phone must be valid E.164 or normalizable format
  • Check phone_normalized field after save
  • Verify country_code is set (stores phone code, not ISO2)

"Stakeholder has not given SMS consent"

  • Call stakeholder.grant_sms_consent! before sending
  • Check sms_consent_given_at and sms_consent_revoked_at timestamps
  • Verify consent in Communication Center UI

"Webhook signature validation failed"

  • Ensure webhook URL is exactly as configured (including https://)
  • Check TWILIO_AUTH_TOKEN matches Twilio Console
  • Verify no proxies/CDNs are modifying request

"SMS not sending"

  • Check job queue: Delayed::Job.where("handler LIKE '%SmsDeliverySendJob%'")
  • Verify merged_payload has body field
  • Check cadence limits aren't blocking send
  • Review logs: tail -f log/development.log | grep -E 'Twilio|SMS'
  • Verify stakeholder has consent: stakeholder.has_sms_consent?
  • Check phone is normalized: stakeholder.phone_normalized

"Trial account restriction"

"Error 63015" (Messaging Sandbox)

  • Recipient hasn't joined the sandbox
  • Have recipient text join <code> to sandbox number
  • Get join code from Twilio Console → Messaging → Sandbox

Debugging Commands

# Verify configuration
rails twilio:verify

# Check account type
rails twilio:account_info

# Test normalization
rails twilio:test_normalize[PHONE_NUMBER]

# Test send
rails twilio:test_send[PHONE_NUMBER]

# Show webhook config
rails twilio:show_config

# Configure webhooks
rails twilio:configure_webhooks[BASE_URL]

Logs

All SMS operations are logged with prefixes: - [TwilioService] - Twilio API calls - [SmsDeliverySendJob] - SMS job processing - [TwilioWebhook] - Webhook processing

# View SMS-related logs
tail -f log/development.log | grep -E '\[Twilio|SMS\]'

Compliance & Security

ISO27001 Compliance

The system meets ISO27001 requirements for SMS consent management:

Requirement Implementation
FR-4 Process STOP keyword
FR-5 Audit trail
FR-7 Respect opt-out
NFR-3 Encrypt credentials
NFR-4 TLS 1.2+
NFR-5 Validate signatures

Audit Logs

All consent changes are logged in consent_audit_logs with: - Stakeholder ID - Channel (sms/email/teams) - Action (consentgiven/consentrevoked/optoutreceived) - Source (admin_ui/webhook/api/import) - Metadata (IP, user agent, etc) - Performed by (admin user, nullable) - Timestamp

Records are immutable (cannot be updated or deleted).

Security Best Practices

  1. Never expose credentials - Use environment variables or Rails credentials
  2. Validate all webhooks - Signature validation is mandatory
  3. Rate limit webhooks - Prevent abuse of opt-out endpoints
  4. Audit all actions - Use ConsentAuditLog for compliance
  5. Encrypt at rest - Use Rails encrypted credentials
  6. Use TLS - All Twilio communication uses HTTPS

Webhook Security

Webhooks validate Twilio signatures using: - X-Twilio-Signature header - Full webhook URL (including query params) - POST body parameters - TWILIO_AUTH_TOKEN

Invalid signatures return HTTP 403.

Available Rake Tasks

# Verify credentials
rails twilio:verify

# Check account type (trial vs paid)
rails twilio:account_info

# Test phone normalization
rails twilio:test_normalize[PHONE_NUMBER]
# OR: TEST_PHONE="+63 962 173 1584" rails twilio:test_normalize

# Verify phone with Twilio
rails twilio:verify_phone[PHONE_NUMBER]

# Send test SMS
rails twilio:test_send[PHONE_NUMBER]
# OR: TEST_PHONE_NUMBER="+63 962 173 1584" rails twilio:test_send

# Test full workflow
rails twilio:test_workflow_sms[PHONE_NUMBER]

# Configure webhooks
rails twilio:configure_webhooks[BASE_URL]

# Show webhook config
rails twilio:show_config

# Test sandbox (test credentials only)
rails twilio:test_sandbox

Pro Tips: - Use environment variables for numbers with spaces/special chars: TEST_PHONE="(334) 966-3045" rails twilio:test_normalize - Normalize first to get E.164 format (no spaces) - Use normalized format for subsequent commands (no quotes needed)

Files & Components

Services

  • app/services/phone_normalization_service.rb - Phone number normalization
  • app/services/twilio_service.rb - Twilio API wrapper

Jobs

  • app/jobs/sms_delivery_send_job.rb - SMS delivery job

Controllers

  • app/controllers/twilio/webhooks_controller.rb - Webhook handling

Models

  • app/models/consent_audit_log.rb - Audit log model
  • app/models/concerns/phone_code_helper.rb - Phone code utilities

Rake Tasks

  • lib/tasks/twilio.rake - Testing and configuration tasks

Migrations

  • add_sms_fields_to_stakeholders - SMS fields and country_code
  • add_external_tracking_to_comms_deliveries - External tracking fields
  • create_consent_audit_logs - Audit log table

Support & Resources

For issues: 1. Check logs: tail -f log/development.log | grep -E 'Twilio|SMS' 2. Review comms_deliveries for failed deliveries 3. Check Twilio Console for API errors 4. Verify webhook configuration 5. Test phone normalization and consent status

Last Updated: January 2026
Version: 1.0