Hypertext Rails

Documentation

System Documentation

Quick Links

Communication Center Documentation

Overview

The Communication Center is a comprehensive system for managing automated and manual communications with stakeholders across multiple channels (Email, SMS, Microsoft Teams). It provides template-based messaging, workflow automation, cadence management, and delivery tracking.

The interface is organized into five main tabs, each with its own folder structure in app/views/communication_center/: - Overview - Dashboard with key metrics and analytics - Delivery Schedule - View and manage scheduled deliveries - Stakeholders - Manage stakeholder information and associations - Communications - Create and manage communication templates - Automation - Configure automated workflows

Core Concepts

Communication Flow

  1. Template Creation → Define reusable communication templates with channel-specific content
  2. Workflow Setup (Optional) → Configure automated workflows for scheduled or triggered communications
  3. Instance Creation → Create a communication instance (manual or automated)
  4. Delivery Generation → System generates individual deliveries for each stakeholder
  5. Event Tracking → Track delivery events (sent, delivered, opened, clicked, bounced, failed)

Tab 1: Overview

The Overview tab provides a comprehensive dashboard with key metrics, engagement analytics, channel distribution, recent activities, and upcoming automations.

Features

Key Metrics

Displays four primary metrics with month-over-month percentage changes: - Total Contacts: Count of unique projects with post destinations - Communications Sent: Total count of sent post attempts - Active Reports: Count of scheduled/processing scheduled postings - Pending Alerts: Count of failed post attempts from the last 7 days

Engagement Overview

Shows 7 weeks of communication data broken down by channel (Email, SMS, Teams).

Channel Distribution

Displays the distribution of communications across channels (Email, SMS, Teams) with counts and percentages.

Recent Activities

Lists the 5 most recent sent communications with details about the post, project, and channel.

Upcoming Automations

Shows the next 5 scheduled postings with their scheduled dates and project information.

Data Loading

The load_dashboard_data method in CommunicationCenterDashboard concern loads all dashboard data:

def load_dashboard_data
  # Key Metrics
  @total_contacts = calculate_total_contacts
  @communications_sent = calculate_communications_sent
  @active_reports = calculate_active_reports
  @pending_alerts = calculate_pending_alerts

  # Calculate percentage changes from last month
  @contacts_change = calculate_contacts_change
  @communications_change = calculate_communications_change
  @reports_change = calculate_reports_change
  @alerts_change = calculate_alerts_change

  # Engagement Overview - 7 weeks of data
  @engagement_data = load_engagement_overview

  # Channel Distribution
  @channel_distribution = load_channel_distribution

  # Recent Activities
  @recent_activities = load_recent_activities

  # Upcoming Automations
  @upcoming_automations = load_upcoming_automations
end

Key Metrics Calculations

Total Contacts: - Queries: Project.joins(:post_destinations).distinct.count - Change calculation compares current month vs last month based on post_destinations.created_at

Communications Sent: - Queries: PostAttempt.where(status: 'sent').count - Change calculation compares current month vs last month of sent attempts

Active Reports: - Queries: ScheduledPosting.where(status: ['scheduled', 'processing']).count - Change calculation compares current month vs last month of scheduled/processing postings

Pending Alerts: - Queries: PostAttempt.where(status: 'failed').where('created_at > ?', 7.days.ago).count - Change calculation compares current month vs last month of failed attempts

Models and Tables Used

Model/Table Purpose Key Queries
Project Count unique projects with post destinations Project.joins(:post_destinations).distinct
PostDestination Join table for projects and channels Used in joins for contact counting
PostAttempt Track sent and failed communication attempts where(status: 'sent'), where(status: 'failed')
ScheduledPosting Track scheduled and processing reports where(status: ['scheduled', 'processing'])
Channel Identify channel types (email, sms, teams) where("LOWER(name) LIKE ?", "%email%")
Post Post information for recent activities Post.where(id: post_ids)
PostV2 Alternative post model for scheduled postings PostV2.where(id: post_v2_ids)

Note: All metrics are calculated on-demand with no caching. Performance may degrade as data volume grows.

Email Sending System

The Communication Center includes a complete email sending system for automation workflows. This section documents how emails are sent from delivery creation to final delivery.

⚠️ Timezone Information

Important: All send_at times in the automation system are stored and executed in UTC+00 (Coordinated Universal Time).

  • The server uses UTC timezone for all scheduled executions
  • send_at values stored in CommsDelivery are in UTC
  • run_at values in Delayed::Job are in UTC
  • When creating deliveries, ensure send_at times are converted to UTC

Example: - If you want to send at 8:00 PM in UTC+8 timezone - Convert to UTC: 8:00 PM - 8 hours = 12:00 PM UTC - Set send_at to 12:00 UTC (or use Rails timezone helpers to convert automatically)

This ensures consistent scheduling regardless of server location or user timezone.

Email Sending Flow

1. Delivery Created
   ↓
2. Variables Resolved Immediately (for preview)
   ↓
3. TemplateVariableReplacementJob Scheduled (for send_at time)
   ↓
4. At send_at time: TemplateVariableReplacementJob Runs
   - Re-resolves variables for fresh data
   - Triggers CommsDeliverySendJob
   ↓
5. CommsDeliverySendJob Runs
   - Validates delivery
   - Calls ApplicationMailer.delivery_email
   ↓
6. ApplicationMailer Sends Email
   - Extracts subject/body from merged_payload
   - Renders HTML and plain text formats
   - Sends via ActionMailer (Postmark/SMTP)
   ↓
7. Delivery Status Updated
   - Status: sent
   - CommsEvent created (type: sent)

Key Components

1. CommsDeliverySendJob

  • Background job that orchestrates email sending
  • Triggered by TemplateVariableReplacementJob after variable resolution
  • Handles validation, error handling, and status updates
  • See "Services" section above for detailed documentation

2. ApplicationMailer (delivery_email method)

  • ActionMailer method that renders and sends emails
  • Extracts content from merged_payload JSON
  • Supports HTML and plain text formats
  • See "Services" section above for detailed documentation

3. Email Template

  • Located at app/views/application_mailer/delivery_email.html.erb
  • Renders body content as raw HTML (supports embedded images)
  • Includes footer with project information

Email Content Structure

The merged_payload JSON structure used for emails:

{
  "subject": "Weekly Report for Team Madrid",
  "body": "This month's morale: 45\n\n<img src=\"https://...\" alt=\"Chart\" />\n\nBaseline morale: 59",
  "recipient": {
    "name": "John Doe",
    "email": "john@example.com",
    "stakeholder_id": 38,
    "project_id": 74
  }
}

Email Features

  • HTML Support: Full HTML rendering with embedded images, charts, and formatting
  • Plain Text Fallback: Automatically generated for text-only email clients
  • Embedded Images: Chart images embedded as <img> tags with public ActiveStorage URLs
  • Reply-To Configuration: Configurable via MAIL_REPLY_TO_ADDRESS environment variable
  • Error Handling: Comprehensive error handling with retry logic and status tracking

Configuration

Development (Postmark): - Configured in config/environments/development.rb - Uses Postmark API for email delivery

Production/Staging (SMTP): - Configured in config/environments/production.rb and config/environments/staging.rb - Uses SMTP settings from config/initializers/smtp_settings.rb - Supports standard SMTP servers

Environment Variables: - MAIL_FROM_ADDRESS: Default sender email address - MAIL_REPLY_TO_ADDRESS: Reply-to email address (defaults to MAIL_FROM_ADDRESS)

Tab 2: Delivery Schedule

The Delivery Schedule tab displays all communication deliveries with filtering capabilities by channel, status, persona, and search terms.

Features

  • View all deliveries with details (template name, receiver, project, channel, scheduled date/time, status, persona)
  • Filter by channel (email, sms, teams, all)
  • Filter by status (scheduled, pending, paused, sent, delivered, failed, cancelled, all)
  • Filter by persona (executive, project team user, csm and admin, all)
  • Search by stakeholder name or template name
  • Cancel individual deliveries
  • Retry failed deliveries
  • Preview deliveries with real-time variable resolution (shows resolved content, not {{variables}})

Delivery Preview

Real-Time Variable Resolution: - When you click "Preview" on a delivery, the system resolves template variables on-the-fly - Shows actual values (e.g., "Team Madrid", "45", "59") instead of placeholders (e.g., {{project_name}}, {{this_months_morale}}) - Uses TemplateVariableResolverService with 10-second timeout - Gracefully falls back to unresolved payload if resolution fails or times out

Preview Features: - Shows delivery information (recipient, project, channel, scheduled time) - Displays resolved email/SMS/Teams content - Handles both HTML and plain text content - Shows raw payload in collapsible debug section

Error Handling: - If resolution times out (10 seconds), shows unresolved payload - If connection fails, shows unresolved payload - Logs warnings but doesn't fail the preview - User can still see the delivery structure even if variables aren't resolved

Data Loading

The load_deliveries method in DeliveriesConcern loads delivery data:

def load_deliveries
  @deliveries = CommsDelivery.includes(:comms_instance => :template, :stakeholder => [], :project => [])
                             .order(send_at: :desc)

  # Apply filters based on params
  # - Search by stakeholder name or template name
  # - Filter by channel
  # - Filter by status
  # - Filter by persona

  @deliveries_count = @deliveries.count
  @deliveries = @deliveries.limit(50)
end

Models and Tables Used

Model/Table Purpose Key Relationships
CommsDelivery Individual delivery records Primary table for this tab
CommsInstance Parent communication instance belongs_to :comms_instance
CommunicationTemplate Template used for delivery comms_instance.template
Stakeholder Delivery recipient belongs_to :stakeholder
Project Associated project context belongs_to :project (optional)

Key Filters: - Channel filter: where(channel: params[:channel]) - Status filter: where(status: params[:status]) - Persona filter: joins(:stakeholder).where(stakeholders: { persona: params[:persona] }) - Search: joins(:stakeholder, comms_instance: :template).where("LOWER(stakeholders.name) LIKE ... OR LOWER(communication_templates.name) LIKE ...")

Tab 3: Stakeholders

The Stakeholders tab manages stakeholder information, their project associations, and cadence rules.

Features

  • View all stakeholders with their contact information, persona, preferred channel, and project associations
  • Create new stakeholders
  • Edit existing stakeholders
  • Delete stakeholders (if no associated deliveries exist)
  • Filter by persona
  • Search by name or email
  • Manage stakeholder-project associations
  • Configure stakeholder-specific cadence rules

Data Loading

The load_stakeholders method in StakeholdersConcern loads stakeholder data:

def load_stakeholders
  @stakeholders = Stakeholder.includes(stakeholder_projects: :project).order(created_at: :desc)

  # Apply filters
  if params[:search].present?
    @stakeholders = @stakeholders.where("LOWER(name) LIKE LOWER(?) OR LOWER(email) LIKE LOWER(?)", search_term, search_term)
  end

  if params[:persona].present? && params[:persona] != 'all'
    @stakeholders = @stakeholders.where(persona: params[:persona])
  end
end

Models and Tables Used

Model/Table Purpose Key Relationships
Stakeholder Stakeholder records Primary table for this tab
StakeholderProject Many-to-many join table has_many :stakeholder_projects
Project Associated projects has_many :projects, through: :stakeholder_projects
CadenceRule Stakeholder-specific cadence rules Polymorphic: scope_type: 'stakeholder'
Persona Persona classification Referenced via persona field (string)

Key Attributes: - name, email, phone, teams_id - persona: String field (executive, project team user, csm and admin) - preferred_channel: Preferred communication channel - channels_opt_out: JSON array of opted-out channels

Tab 4: Communications (Templates)

The Communications tab manages communication templates with multi-channel support.

Features

  • View all communication templates
  • Create new templates (with draft support)
  • Edit existing templates
  • Archive/activate templates
  • Duplicate templates
  • Filter by template type (report, alert, storysofar, custom, all)
  • Filter by status (active, archived, all)
  • Search by template name
  • Manage template variables
  • Preview templates across channels (email, sms, teams)
  • Line break normalization for better editing experience

Template Editing Improvements

Line Break Handling

When Saving Email Content: - Normalizes line breaks for consistent storage: - Converts \r\n (Windows) to \n - Converts \r (Mac) to \n - Converts \n to <br> HTML tags - Ensures email content displays correctly with proper line breaks

When Loading Email Content for Editing: - Converts <br> tags back to newlines (\n) for natural editing - Users can edit with normal line breaks instead of HTML tags - Makes template editing more intuitive

Example: ``` User types in textarea: Line 1 Line 2 Line 3

Stored in database: Line 1
Line 2
Line 3

Displayed in email: Line 1 Line 2 Line 3 ```

Variable Picker

  • Searchable variable picker with categories:
    • Project Variables (e.g., {{project_name}}, {{this_months_morale}})
    • Dynamic Components
    • Dynamic Charts
  • Filter variables by name, value, or description
  • Insert variables into subject or body with proper spacing
  • Preview shows highlighted variables in gray background

Data Loading

The load_templates method in TemplatesConcern loads template data:

def load_templates
  @templates = CommunicationTemplate.with_names.order(created_at: :desc)

  # Apply filters
  if params[:search].present?
    @templates = @templates.where("LOWER(name) LIKE LOWER(?)", search_term)
  end

  if params[:template_type].present? && params[:template_type] != 'all'
    @templates = @templates.where(template_type: params[:template_type])
  end

  if params[:status].present? && params[:status] != 'all'
    @templates = @templates.where(is_active: params[:status] == 'active')
  end
end

Models and Tables Used

Model/Table Purpose Key Relationships
CommunicationTemplate Template records Primary table for this tab
AdminUser Template creator belongs_to :created_by, class_name: 'AdminUser'
CommsInstance Instances using this template has_many :comms_instances
AutomationWorkflow Workflows using this template has_many :automation_workflows
TemplateVariable Available template variables Referenced for variable picker

Key Attributes: - name: Template identifier - template_type: Type (report, alert, storysofar, custom) - channel_variants: JSON with channel-specific content (emailhtml, smstext, teamstext) - variables: JSON array of template variables - `isactive`: Boolean for active/archived status

Tab 5: Automation (Workflows)

The Automation tab manages automated communication workflows.

Features

  • View all automation workflows
  • Create new workflows (scheduled, trigger-based, persona-based)
  • Edit existing workflows
  • Pause/resume workflows
  • Archive workflows
  • Filter by workflow type (scheduled, trigger, persona, all)
  • Search by workflow name
  • Preview deliveries before creation
  • Create deliveries from workflows
  • Smart action buttons based on workflow state

Workflow Action Buttons

The automation table shows different action buttons based on workflow state:

  1. Never Run Before (last_run_at.nil?)

    • Shows: "Activate" button (green)
    • Action:
      • Scheduled workflows: Creates deliveries immediately and activates workflow
      • Trigger workflows: Only activates workflow (no deliveries created until triggers fire)
    • Confirmation: Varies by workflow type

  2. Active Workflow (status == 'active' and has run before)

    • Shows: "Pause" button (yellow)
    • Action: Pauses the workflow (sets status to 'paused')
    • Note: Workflow will not create new deliveries until resumed

  3. Paused Workflow (status == 'paused' and has run before)

    • Shows: "Resume" button (green)
    • Action: Resumes the workflow (sets status to 'active')
    • Note: Workflow will resume creating deliveries on schedule

Button Logic: - Checks last_run_at.nil? first to handle never-run workflows - Then checks status to determine pause/resume - Clear visual distinction between states - Prevents confusion about workflow state

Data Loading

The load_workflows method in WorkflowsConcern loads workflow data:

def load_workflows
  @workflows = AutomationWorkflow.includes(:template, :created_by).order(created_at: :desc)

  # Apply filters
  if params[:search].present?
    @workflows = @workflows.where("LOWER(name) LIKE LOWER(?)", search_term)
  end

  if params[:workflow_type].present? && params[:workflow_type] != 'all'
    @workflows = @workflows.where(workflow_type: params[:workflow_type])
  end

  @workflows_count = @workflows.count
end

Models and Tables Used

Model/Table Purpose Key Relationships
AutomationWorkflow Workflow records Primary table for this tab
CommunicationTemplate Template used by workflow belongs_to :template
AdminUser Workflow creator belongs_to :created_by, class_name: 'AdminUser'
CommsInstance Instances created by workflow has_many :comms_instances
CommsDelivery Deliveries created from workflow Through CommsInstance
Stakeholder Target audience for workflows Resolved via audience_config
Project Project filters in workflows Resolved via audience_config
Persona Persona filters in workflows Resolved via audience_config

Key Attributes: - name: Workflow identifier - workflow_type: Type (scheduled, trigger, persona) - schedule_config: JSON configuration for scheduled workflows (frequency, time, dayofweek, etc.) - trigger_config: JSON configuration for trigger-based workflows (events array, last_trigger_check_at) - audience_config: JSON defining target audience - channels: JSON array of channels (email, sms, teams) - status: Workflow status (active, paused, archived) - last_run_at: Timestamp of last execution - next_run_at: Timestamp of next scheduled execution (scheduled workflows only)

For complete workflow documentation, see: docs/AUTOMATION_WORKFLOW_COMPLETE_FLOW.md

Models and Their Purposes

1. CommunicationTemplate

Purpose: Reusable message templates with multi-channel support

Key Attributes: - name: Template identifier - template_type: Type of communication (report, alert, story_so_far, custom) - channel_variants: JSON containing channel-specific content (emailhtml, smstext, teamstext) - variables: JSON defining template variables for dynamic content - `isactive`: Whether the template is currently active

Relationships: - Belongs to AdminUser (created_by) - Has many CommsInstance - Has many AutomationWorkflow

2. AutomationWorkflow

Purpose: Automated communication scheduling and triggering

Key Attributes: - name: Workflow identifier - workflow_type: Type of automation (scheduled, trigger, persona) - template_id: Associated template - schedule_config: JSON configuration for scheduled workflows (frequency, time, dayofweek, etc.) - trigger_config: JSON configuration for trigger-based workflows (events array, last_trigger_check_at timestamp) - audience_config: JSON defining target audience (personas, projects, filter modes) - channels: JSON array of channels to use (email, sms, teams) - status: Workflow status (active, paused, archived) - last_run_at: Timestamp of last execution - next_run_at: Timestamp of next scheduled execution (scheduled workflows only)

For complete workflow documentation including trigger events and deployment requirements, see: docs/AUTOMATION_WORKFLOW_COMPLETE_FLOW.md

Relationships: - Belongs to CommunicationTemplate (template) - Belongs to AdminUser (created_by) - Has many CommsInstance

3. CommsInstance

Purpose: Represents a single communication execution (one-time send or workflow execution)

Key Attributes: - template_id: Template used for this instance - automation_workflow_id: Workflow that created this instance (if automated) - audience_scope: JSON defining the audience for this instance - scheduled_at: When the communication should be sent - status: Instance status (draft, scheduled, paused, sent, failed) - created_by_id: Admin user who created it

Relationships: - Belongs to CommunicationTemplate (template) - Belongs to AdminUser (created_by) - Belongs to AutomationWorkflow (optional) - Has many CommsDelivery

4. CommsDelivery

Purpose: Individual delivery to a specific stakeholder via a specific channel

Key Attributes: - comms_instance_id: Parent communication instance - stakeholder_id: Target stakeholder - project_id: Associated project (for project-level tracking) - channel: Delivery channel (email, sms, microsoft teams) - merged_payload: JSON containing the final rendered message content - send_at: When the delivery should be sent (always in UTC+00 timezone) - status: Delivery status (scheduled, pending, paused, sent, delivered, failed, cancelled) - failure_reason: Error message if delivery failed

Relationships: - Belongs to CommsInstance - Belongs to Stakeholder - Belongs to Project (optional) - Has many CommsEvent

5. CommsEvent

Purpose: Tracks events related to a delivery (sent, delivered, opened, clicked, bounced, failed)

Key Attributes: - delivery_id: Parent delivery - event_type: Type of event (sent, delivered, opened, clicked, bounced, failed) - metadata: JSON containing event-specific data - occurred_at: When the event occurred

Relationships: - Belongs to CommsDelivery (delivery)

6. Stakeholder

Purpose: Represents a person who receives communications

Key Attributes: - name: Stakeholder name - email: Email address - phone: Phone number (for SMS) - teams_id: Microsoft Teams user ID - persona: Persona classification (executive, project team user, csm and admin) - preferred_channel: Preferred communication channel - channels_opt_out: JSON array of opted-out channels

Relationships: - Has many StakeholderProject (many-to-many with Projects) - Has many Projects (through StakeholderProject) - Has many CommsDelivery - Has method cadence_rules (polymorphic via CadenceRule)

7. StakeholderProject

Purpose: Join table linking Stakeholders to Projects (many-to-many)

Key Attributes: - stakeholder_id: Stakeholder reference - project_id: Project reference - role: Stakeholder's role in the project (optional)

Relationships: - Belongs to Stakeholder - Belongs to Project

8. Project

Purpose: Represents a project that stakeholders are associated with

Key Attributes: (Inherited from PepcoreBase) - Standard project attributes

Relationships: - Has many StakeholderProject - Has many Stakeholders (through StakeholderProject) - Has many CommsDelivery - Has many AlertTrigger - Has method for cadence_rules (polymorphic via CadenceRule)

9. CadenceRule

Purpose: Controls communication frequency limits to prevent over-communication

Key Attributes: - scope_type: Scope of the rule (persona, stakeholder, project, multi_project) - scope_ref: ID of the referenced entity (as string) - rules: JSON containing frequency limits: - max_per_day: Maximum communications per day - max_per_week: Maximum communications per week - max_per_month: Maximum communications per month - cooldown_hours: Minimum hours between communications - type_limits: Per-type limits (report, alert, storysofar, custom) - default_types: JSON array of default communication types allowed

Polymorphic Relationships: - Can belong to Persona (scopetype='persona') - Can belong to Stakeholder (scopetype='stakeholder') - Can belong to Project (scope_type='project')

10. Persona

Purpose: Groups stakeholders by role/type with shared cadence rules

Key Attributes: - name: Persona name (must be unique) - Automatically creates a default CadenceRule on creation

Relationships: - Has many Stakeholders (via persona field) - Has one CadenceRule (scope_type='persona')

Default Personas: - executive - project team user - csm and admin

11. AlertTrigger

Purpose: Represents events that can trigger automated communications

Key Attributes: - project_id: Associated project - trigger_type: Type of trigger event - payload: JSON containing trigger-specific data - triggered_at: When the trigger occurred

Relationships: - Belongs to Project

12. Import

Purpose: Tracks bulk import operations (e.g., stakeholder imports)

Key Attributes: - file_name: Name of imported file - status: Import status - result_counts: JSON containing import statistics

Relationships: None (standalone)

Services

CommsDeliverySendJob

Purpose: Background job that sends email deliveries after variable resolution is complete

Queue: default

When it runs: Triggered by TemplateVariableReplacementJob after successful variable resolution

Key Features:

Validation and Safety Checks

  • Verifies delivery exists and hasn't been processed (sent/delivered/cancelled)
  • Only processes email deliveries (skips SMS/Teams)
  • Checks send_at time has passed (with 5-second buffer for timing)
  • Validates stakeholder and payload content exist
  • Reschedules job if send_at time hasn't been reached yet

Email Sending Process

  1. Updates delivery status to pending before sending
  2. Calls ApplicationMailer.delivery_email(delivery) to send email
  3. Updates delivery status to sent on success
  4. Creates CommsEvent record with event type sent and metadata

Error Handling

  • Retry Logic: 3 attempts with exponential backoff for transient errors
  • Discards on: ActiveJob::DeserializationError (delivery no longer exists)
  • On Failure: Sets delivery status to failed, records failure_reason, creates CommsEvent with type failed
  • Logging: Comprehensive logging for debugging and monitoring

Status Transitions

  • scheduled or pending → pending (before send) → sent (after successful send)
  • On failure: Status → failed, failure_reason set, CommsEvent created

ApplicationMailer (delivery_email method)

Purpose: Sends email deliveries for automation workflows

Key Methods: - delivery_email(delivery): Sends an email for a CommsDelivery record

Process: 1. Extracts email content from delivery.merged_payload_hash: - Subject from payload['subject'] or payload['title'] (defaults to "Communication from Peptalk") - Body from payload['body'], payload['content'], payload['message'], or payload['html'] 2. Gets recipient email from stakeholder.email or payload['recipient']['email'] 3. Sets reply-to address from MAIL_REPLY_TO_ADDRESS environment variable (defaults to MAIL_FROM_ADDRESS) 4. Renders email in both HTML and plain text formats: - HTML: Uses application_mailer/delivery_email.html.erb template with raw HTML body - Plain text: Strips HTML tags from body for text-only clients 5. Sends email via ActionMailer (configured for Postmark in development, SMTP in production/staging)

Email Template: - Located at app/views/application_mailer/delivery_email.html.erb - Renders body content as raw HTML (supports embedded images, charts, formatting) - Includes footer with project information if available

Error Handling: - Validates delivery and stakeholder exist before sending - Falls back to default subject/body if content is missing - Logs errors but doesn't raise exceptions (handled by CommsDeliverySendJob)

DeliveryScheduleService

Purpose: Creates deliveries from workflows or instances with intelligent project filtering and immediate variable resolution

Key Methods: - create_deliveries(send_at:, created_by:): Creates CommsInstance and CommsDeliveries - preview: Returns preview of what would be created (stakeholders, channels, count)

Process: 1. Validates workflow status (must be active or paused) 2. Resolves audience based on workflow/instance configuration 3. Applies project filtering (include/exclude modes) 4. Checks template has available channels 5. Creates CommsInstance with audience snapshot 6. Creates CommsDelivery for each stakeholder/channel/project combination 7. Immediately resolves template variables for preview purposes 8. Schedules TemplateVariableReplacementJob to re-resolve at send_at time 9. Respects cadence rules and opt-outs 10. Updates workflow status and last_run_at

Project Filtering Modes

The service supports three project filtering modes in audience_config:

  1. 'include' or 'specific' (Backward Compatible)

    • Only includes stakeholders who are associated with specified projects
    • Uses intersection: stakeholder_projects & workflow_projects
    • Example: Only send to stakeholders in Project A and Project B

  2. 'exclude' (New)

    • Excludes stakeholders who are associated with specified projects
    • Uses difference: stakeholder_projects - workflow_projects
    • Example: Send to all stakeholders EXCEPT those in Project A

  3. Default (No Filter)

    • Includes all stakeholders (if no project_ids specified)
    • Uses all stakeholder's projects as contexts

Project Context Creation: - Creates one delivery per project context for each stakeholder - If no projects remain after filtering, creates delivery with project_id: nil - Allows project-specific variable resolution (e.g., {{project_name}})

Immediate Variable Resolution

When it happens: - Right after creating each CommsDelivery record - Before scheduling the background job

Why it matters: - Enables instant preview of deliveries - Users can see resolved content immediately - Preview doesn't need to wait for background job

Process: 1. Creates delivery with initial payload (may contain {{variables}}) 2. Calls TemplateVariableResolverService to resolve variables 3. Updates merged_payload with resolved content 4. Logs success or warnings (doesn't fail delivery creation if resolution fails) 5. Schedules job to re-resolve at send_at time for fresh data

Error Handling: - If resolution fails, delivery is still created - Background job will retry resolution at send time - Logs errors but doesn't block delivery creation

TemplateVariableResolverService

Purpose: Resolves template variables in delivery payloads with performance optimizations

Key Methods: - resolve_and_render: Main method that resolves all variables and returns JSON payload

Features: - Variable Extraction: Only resolves variables actually used in template - Memoization: Prevents N+1 queries by caching fetched data - Two-Tier Fetching: ActiveRecord model first, analytics DB fallback - Timeout Protection: 5-second timeout for analytics DB queries - Error Handling: Graceful fallback for unresolved variables

Performance Optimizations: - Extracts variables using regex /\{\{([^}]+)\}\}/ - Only resolves variables found in template content (performance optimization) - Extracts variables from dynamic component tokens (e.g., {{GAUGE:label:baseline_morale:engagement_rate}}) - For charts, extracts variables from config parameters (x, y, data) - Memoizes submission data to prevent duplicate queries - Uses pluck for efficient data fetching

See "Template Variables" section above for detailed documentation.

TemplateVariableReplacementJob

Purpose: Background job that re-resolves template variables at send time to ensure fresh data, then triggers email sending

Queue: default

When it runs: Scheduled to run at delivery's send_at time

Key Features:

Timeout Protection

  • 30-second timeout wrapper around variable resolution
  • Prevents job from hanging indefinitely
  • Discards timeout errors (won't benefit from retry)

Retry Logic

  • Retries: 3 attempts with exponential backoff
  • Retries on: StandardError (transient errors)
  • Discards on: Timeout::Error (won't benefit from retry)
  • Re-raises: ActiveRecord::ConnectionTimeoutError (transient, should retry)

Smart Detection

  • Detects unresolved variables by checking for {{ pattern in payload
  • Distinguishes between "resolved" and "re-resolved" actions in logs
  • Only fails delivery if variables were never resolved
  • Allows graceful handling if re-resolution fails but payload already resolved

Error Handling

# Timeout Error
- Logs error
- Sets failure_reason: "Template variable resolution timeout: operation took too long"
- Sets status: 'failed'
- Discards job (no retry)

# Connection Timeout Error
- Logs error
- Sets failure_reason: "Database connection timeout: {message}"
- Sets status: 'failed'
- Re-raises to retry when connections are available

# Standard Error
- Logs error with backtrace
- Sets failure_reason: "Template variable resolution error: {message}"
- Sets status: 'failed'
- Re-raises to retry (up to 3 attempts)

Job Flow

  1. Finds delivery by ID
  2. Checks if already processed (sent/delivered/cancelled) → skip
  3. Detects if variables need resolution (checks for {{ pattern)
  4. If no resolution needed, triggers email send job directly (for already-resolved payloads)
  5. If resolution needed:
    • Wraps resolution in 30-second timeout
    • Calls TemplateVariableResolverService.resolve_and_render
    • Updates merged_payload with resolved content
  6. Triggers CommsDeliverySendJob if delivery channel is email and status is scheduled or pending
  7. Logs success or failure
  8. Handles errors appropriately (retry vs discard)

Workflows

Manual Communication Flow

  1. Admin creates/selects a CommunicationTemplate
  2. Admin creates a CommsInstance with audience scope
  3. System generates CommsDelivery records for each stakeholder
  4. Deliveries are sent via appropriate channels
  5. CommsEvent records track delivery status

Automated Workflow Flow

For detailed documentation on automation workflows (both scheduled and trigger-based), see: docs/AUTOMATION_WORKFLOW_COMPLETE_FLOW.md

Scheduled Workflows: 1. Admin creates AutomationWorkflow with schedule configuration 2. System checks next_run_at and creates CommsInstance when due 3. DeliveryScheduleService creates deliveries 4. Deliveries are sent and tracked 5. Workflow last_run_at and next_run_at are updated

Trigger-Based Workflows: 1. Admin creates AutomationWorkflow with trigger configuration 2. Cron job (rake automation_workflows:process_triggers) evaluates triggers every 5 minutes 3. When trigger fires, DeliveryScheduleService creates deliveries immediately 4. Deliveries are sent immediately (status: pending, send_at: Time.current) 5. Workflow last_trigger_check_at is updated

Cadence Enforcement Flow

  1. Before creating a delivery, system checks relevant CadenceRule
  2. Checks if stakeholder has received too many communications in time period
  3. Checks type-specific limits
  4. Checks cooldown period
  5. If limits exceeded, delivery is skipped or queued

Best Practices

  1. Template Design:

    • Use clear variable names
    • Test templates across all channels
    • Keep SMS messages concise

  2. Workflow Configuration:

    • Set appropriate schedules to avoid off-hours sends
    • Test workflows with preview before activating
    • Monitor delivery success rates

  3. Cadence Management:

    • Set persona-level defaults
    • Use stakeholder overrides sparingly
    • Monitor cadence rule effectiveness

  4. Stakeholder Management:

    • Keep contact information up-to-date
    • Respect opt-out preferences
    • Use personas for consistent targeting

  5. Delivery Monitoring:

    • Regularly review failed deliveries
    • Monitor event tracking for engagement
    • Adjust templates based on performance

Common Use Cases

Weekly Executive Report

  1. Create template with report content
  2. Create scheduled workflow (weekly, Monday 9 AM)
  3. Target "executive" persona
  4. Use email and Teams channels
  5. System automatically sends weekly reports

Project Alert Notifications (Trigger-Based)

  1. Create alert template
  2. Create trigger-based workflow (workflow_type: trigger)
  3. Configure trigger events (e.g., morale drop, kiosk offline, spotcheck low completion)
  4. Target stakeholders for specific project
  5. System evaluates triggers every 5 minutes via cron job
  6. System sends alerts immediately when triggers fire

For detailed trigger-based workflow documentation, see: docs/AUTOMATION_WORKFLOW_COMPLETE_FLOW.md

Custom One-Time Communication

  1. Select or create template
  2. Create CommsInstance manually
  3. Define audience scope
  4. Schedule send time
  5. System creates and sends deliveries

Complete Data Flow: Automation Workflow to Delivery

This section documents the complete flow from when a user clicks "Activate" on an automation workflow to when the email is sent.

Overview

The system uses an immediate generation approach: - Deliveries are created and variables resolved immediately - Chart images are generated immediately (not deferred to background) - Preview shows whatever is already in the payload - Final send re-resolves data to ensure freshness

Step 1: User Clicks "Activate"

Controller: CommunicationCenter::DeliveriesConcern#create_deliveries

What Happens: 1. For scheduled workflows: DeliveryScheduleService.create_deliveries builds CommsInstance + CommsDelivery records 2. For trigger workflows: Only workflow status is updated to active (no deliveries created until triggers fire) 3. For scheduled workflows, for each delivery: - Initial payload created with template content - Immediate variable resolution via TemplateVariableResolverService - Simple variables resolved (e.g., {{project_name}} → "Team Madrid") - Dynamic components (charts, gauges) generate images immediately - merged_payload updated with resolved content (including <img> tags for charts) 4. TemplateVariableReplacementJob scheduled to run at send_at time (for data freshness) 5. Workflow last_run_at updated 6. User redirected to Delivery Schedule tab

Result: Fast response (~1-2 seconds for simple templates, longer if many charts)

Step 2: Variable Resolution During Delivery Creation

When: Immediately after each CommsDelivery is created

Service: TemplateVariableResolverService.resolve_and_render

Process: 1. Extracts only variables actually used in template (performance optimization) 2. Uses memoization to prevent N+1 queries 3. For simple variables: - Fetches data (prefers CheckInSubmission model, falls back to analytics DB with 5s timeout) - Calculates metrics (morale, engagement, etc.) - Replaces {{variable}} with actual value 4. For dynamic components (GAUGE, THEME, LIST, BAR, LINE, PIE, CHART): - Detects component token format - For CHART type, extracts actual chart type (BAR/LINE/PIE) from label using ChartTypeExtractor - Resolves any variables within the component config (e.g., x, y, data parameters for charts) - Generates image immediately via DynamicComponentImageService - Replaces token with <img> tag 5. Updates merged_payload column with fully resolved content 6. Logs success or warnings (doesn't fail delivery creation if resolution fails)

Example Resolved Payload (with chart): json { "subject": "Weekly Report for Team Madrid", "body": "This month's morale: 45\n\n<img src=\"https://integrations.peptalk.com/rails/active_storage/blobs/.../chart.png\" alt=\"Weekly Morale Trend\" style=\"max-width: 100%; height: auto;\" />\n\nBaseline morale: 59", "recipient": { "name": "John Doe", "email": "john@example.com", "stakeholder_id": 38, "project_id": 74 } }

Step 3: Chart Image Generation (If Templates Contain Charts)

When: During variable resolution (Step 2)

Process: 1. TemplateVariableResolverService detects chart token (e.g., {{BAR:...}}) 2. Calls DynamicComponentImageService to generate image 3. ChartDataService fetches time-series data based on x-axis config (daily/weekly/monthly) 4. DynamicComponentRendererService builds Chart.js HTML with real data 5. Grover converts HTML to PNG (800x600 viewport, 2x scale) 6. PNG stored in ActiveStorage 7. Returns public URL for image 8. Token replaced with <img> tag in payload

Performance: ~2-5 seconds per chart

Step 4: Background Job Scheduling

Job: TemplateVariableReplacementJob

When Scheduled: To run at delivery's send_at time

Purpose: Re-resolve variables at send time to ensure fresh data

Timezone: The job is scheduled using the send_at time, which is always in UTC+00. The delayedjob system stores this as `runat` in UTC.

# In DeliveryScheduleService (line 192)
delivery.schedule_variable_replacement_job
# This creates a Delayed::Job with run_at = delivery.send_at (UTC)

Note: The send_at value must be in UTC. If scheduling based on user's local time, convert to UTC first.

Step 5: Preview (Anytime Before Send)

What Happens: 1. User clicks "Preview" button on a delivery 2. Loads delivery's merged_payload from database 3. No image generation in preview - shows whatever is already in payload 4. If payload has <img> tags → shows chart images 5. If payload still has unresolved variables (fallback): - Calls TemplateVariableResolverService on-the-fly - 10-second timeout for preview resolution - Gracefully falls back to unresolved payload if timeout/failure

Result: Preview shows final email content with charts as images

Step 6: Send Time (Background Job)

Job: TemplateVariableReplacementJob runs at send_at time (UTC+00)

Purpose: Re-resolve simple variables to ensure fresh data (metrics may have changed since creation)

Timezone: The job executes when server UTC time reaches the send_at UTC time stored in the delivery record.

Process: 1. Checks if delivery already processed (sent/delivered/cancelled) → skip 2. Detects if variables need resolution (checks for {{ pattern in subject/body) 3. Wraps resolution in 30-second timeout 4. Calls TemplateVariableResolverService to re-resolve 5. Does NOT regenerate chart images (they're already in payload as <img> tags from initial resolution) 6. Updates merged_payload with latest simple variable values 7. Handles errors appropriately (retry transient errors, discard timeout errors)

Why Re-resolve Simple Variables? - Data may have changed between creation and send time - Ensures stakeholders receive up-to-date metrics - Example: If delivery created Monday but sent Friday, morale may have changed

Step 7: Email Sending (Background Job)

Job: CommsDeliverySendJob (triggered by TemplateVariableReplacementJob)

When: Immediately after variable resolution completes at send_at time

Process: 1. TemplateVariableReplacementJob completes variable resolution 2. If delivery channel is email and status is scheduled or pending, triggers CommsDeliverySendJob 3. CommsDeliverySendJob validates delivery: - Checks delivery exists and hasn't been processed - Verifies send_at time has passed (with 5-second buffer) - Validates stakeholder and payload content exist 4. Updates delivery status to pending 5. Calls ApplicationMailer.delivery_email(delivery) to send email 6. Updates delivery status to sent 7. Creates CommsEvent record with event type sent

Email Content: - Subject extracted from merged_payload['subject'] - Body extracted from merged_payload['body'] (supports HTML with embedded images) - Recipient email from stakeholder.email or merged_payload['recipient']['email'] - Reply-to address from MAIL_REPLY_TO_ADDRESS environment variable (defaults to MAIL_FROM_ADDRESS)

Email Rendering: - HTML format: Renders delivery_email.html.erb template with raw HTML body content - Plain text format: Strips HTML tags from body for text-only email clients - Images: Chart images embedded as <img> tags with public URLs from ActiveStorage

Error Handling: - If email send fails: Delivery status set to failed, failure_reason recorded, CommsEvent with type failed created - Retry logic: 3 attempts with exponential backoff for transient errors - Discards on deserialization errors (delivery no longer exists)

Key Design Decisions

Why Immediate Image Generation?

Benefits: - Fast preview: Images already generated when user views preview - Consistent experience: What you preview is what gets sent - No background job delays for viewing deliveries - Simpler debugging: Can verify chart immediately after creation

Trade-offs: - Slightly longer "Activate" response time if many charts (for scheduled workflows) - Images may become stale if data changes significantly before send

Why Re-resolve at Send Time?

  • Simple variables (morale, engagement) might change
  • Ensures recipients get fresh metrics
  • Chart images remain as-is (regenerating would be expensive and usually unnecessary)

Error Handling

Stage Error Type Handling
Delivery Creation Variable resolution fails Delivery still created, logs warning
Delivery Creation Chart generation fails Falls back to text placeholder
Preview Resolution times out (10s) Shows unresolved payload
Send Time Job Timeout (30s) Status 'failed', discards job
Send Time Job Connection timeout Status 'failed', retries
Send Time Job Standard error Status 'failed', retries up to 3 times
Email Send Job Email send fails Status 'failed', failure_reason recorded, retries up to 3 times
Email Send Job Deserialization error Discards job (delivery no longer exists)
Email Send Job Send time not reached Reschedules job for correct time

Component Types Supported

The system supports these dynamic component types:

Type Format Description
GAUGE {{GAUGE:Label:value_var:delta_var}} Numeric value with delta indicator
THEME {{THEME:Label:value_var:percentage}} Theme with progress bar
LIST {{LIST:Label:items_var}} Bulleted list
BAR `{{BAR:Label\ title=...\
LINE `{{LINE:Label\ title=...\
PIE `{{PIE:Label\ title=...\
CHART `{{CHART:Label\ title=...\

Chart Type Detection: - The CHART type is a generic chart token that automatically determines the chart type (BAR, LINE, or PIE) from the label using ChartTypeExtractor - If the label contains "BAR", "LINE", or "PIE" (case-insensitive), the corresponding chart type is used - Falls back to BAR if no type is detected in the label - Example: {{CHART:Bar Chart|title=...|x=...|y=...}} → Extracted as BAR chart

All component types are recognized by both TemplateVariableResolverService and TemplateVariableReplacementJob.

Template Variables

Available Variables

The Communication Center supports 7 built-in template variables that can be used in email subjects and body content:

Variable Description Calculation Returns
{{project_name}} Project name project.name String (e.g., "Team Madrid")
{{this_months_morale}} Current month morale score MoraleCalculationService.calculate_team_morale_from_submissions for current month String (0-100)
{{baseline_morale}} Baseline morale using project's baseline range MoraleCalculationService.calculate_baseline_morale_from_submissions with project's default baseline range String (0-100)
{{engagement_rate}} Engagement rate as percentage (avg_score / 5.0 * 100).round for current month submissions String (0-100)
{{action_closure_rate}} Action closure rate as percentage (avg_score / 5.0 * 100).round for current month submissions String (0-100)
{{responses_count}} Number of check-in submissions Count of submissions for current month String (count)
{{multi_project_average_morale}} Average morale across all stakeholder's projects Average of this_months_morale across all projects associated with stakeholder String (0-100)

Dynamic Chart Components

The Communication Center supports dynamic chart generation that can be embedded in templates. Charts are rendered using Chart.js, converted to PNG images via Grover, and automatically inserted into emails.

Supported Chart Types

  • BAR - Vertical bar charts for comparing values
  • LINE - Line charts for showing trends over time
  • PIE - Pie charts for showing distributions
  • CHART - Generic chart type that automatically extracts the chart type (BAR/LINE/PIE) from the label using ChartTypeExtractor

Chart Token Format

Chart Types: You can use specific chart types (BAR, LINE, PIE) or the generic CHART type:

{{CHART_TYPE:Label|parameter1=value1|parameter2=value2|...}}

Examples: - ✅ {{BAR:Performance|title=Metrics|...}} - Specific bar chart - ✅ {{LINE:Trend|title=Trend|...}} - Specific line chart - ✅ {{PIE:Distribution|title=Breakdown|...}} - Specific pie chart - ✅ {{CHART:Bar Chart|title=Metrics|...}} - Generic chart (extracted as BAR from label) - ✅ {{CHART:Line Graph|title=Trend|...}} - Generic chart (extracted as LINE from label)

Generic CHART Type: - The CHART type automatically extracts the chart type from the label using ChartTypeExtractor - Searches for "BAR", "LINE", or "PIE" keywords in the label (case-insensitive) - Falls back to BAR if no type is detected - Useful when the chart type is determined dynamically or from user input

Chart Parameters

Parameter Description Required Example Values
title Chart title displayed at the top No title=Monthly Trends
x X-axis date period (daily, weekly, monthly) Yes x=daily or x=weekly or x=monthly
y Y-axis metric (variable name) Yes y=baseline_morale
data Data values for pie charts (variable names) Yes (for pie) data=morale,engagement,closure
labels Custom labels (comma-separated) No labels=Jan,Feb,Mar
legend Legend position No legend=bottom (options: top, bottom, left, right)
color Primary color name No color=indigo (options: indigo, blue, green, red, purple, pink, yellow, gray)

X-Axis Date Period Options

Value Description Data Range
daily Daily data points Current month (Day 1 to Day 30/31)
weekly Weekly data points Last 3 months (week by week)
monthly Monthly data points Last 6 months (month by month)

Y-Axis Metric Options

Any template variable can be used as a Y-axis metric: - baseline_morale - Baseline morale score - this_months_morale - Current month morale - engagement_rate - Engagement percentage - action_closure_rate - Action closure percentage - responses_count - Number of responses

Chart Examples

1. Bar Chart - Weekly Morale Trend

{{BAR:Team Performance|title=Weekly Morale Trend|x=weekly|y=baseline_morale|legend=bottom|color=indigo}}

What it does: - Creates a bar chart showing morale over the last 3 months - X-axis: Week of Oct 7, Week of Oct 14, etc. - Y-axis: Morale score for each week - Legend at bottom, Indigo colored bars

2. Line Chart - Monthly Engagement

{{LINE:Engagement Trend|title=6-Month Engagement|x=monthly|y=engagement_rate|color=blue|legend=top}}

What it does: - Creates a line chart showing engagement over 6 months - Smooth curve with blue color - Legend at top

3. Pie Chart - Metrics Distribution

{{PIE:Metrics Distribution|title=Key Metrics|data=engagement_rate,action_closure_rate,responses_count|labels=Engagement,Actions,Responses|color=purple}}

What it does: - Creates a pie chart with 3 slices - Each slice represents: Engagement Rate, Action Closure Rate, and Response Count - Custom labels for each slice - Color palette based on purple theme

Template Integration Example

<h2>Weekly Report for {{project_name}}</h2>

<p>Here's your performance overview:</p>

{{BAR:Performance|title=This Month's Metrics|x=weekly|y=baseline_morale|color=indigo|legend=bottom}}

<p>Engagement trend:</p>

{{LINE:Engagement|title=Engagement Rate|x=monthly|y=engagement_rate|color=blue}}

<p>Distribution breakdown:</p>

{{PIE:Distribution|title=Metrics Breakdown|data=engagement_rate,action_closure_rate|labels=Engagement,Actions|color=purple}}

Color Options

Predefined color palette:

Color Name Hex Code
indigo #6366f1 (default)
blue #3b82f6
green #10b981
red #ef4444
purple #9333ea
pink #ec4899
yellow #f59e0b
gray #6b7280

For pie charts, a multi-color palette is automatically generated using all available colors.

How Chart Generation Works

1. Template Creation

  • User creates a communication template
  • Adds chart tokens like {{BAR:...}} in the email body
  • Saves the template

2. Delivery Creation (via "Activate" for scheduled workflows)

  • When "Activate" is clicked on a scheduled workflow
  • System creates CommsDelivery records
  • Chart tokens are detected during variable resolution
  • Chart images are generated immediately (not deferred)
  • For trigger workflows, deliveries are only created when triggers fire (via cron job)

3. Chart Image Generation Pipeline

When: During immediate variable resolution (Step 2 of delivery creation)

Process: - Step 1: Parse chart token and extract configuration (type, x-axis, y-axis, color, etc.) - For CHART type, use ChartTypeExtractor to determine actual chart type (BAR/LINE/PIE) from label - Step 2: Resolve variables in chart config (x, y, data parameters) if they contain variable names - Step 3: ChartDataService fetches real time-series data from CheckInSubmission records - Step 4: DynamicComponentRendererService generates Chart.js HTML with the data - Step 5: DynamicComponentImageService uses Grover to convert HTML to PNG (800x600 viewport, 2x scale) - Step 6: PNG stored in ActiveStorage with public URL - Step 7: Token in merged_payload replaced with <img> tag pointing to stored image

Performance: ~2-5 seconds per chart (done synchronously during delivery creation)

Note: Chart images are generated immediately during delivery creation, not deferred to background jobs. This ensures images are available for preview and email sending.

4. Delivery Sending

  • At send time, delivery contains fully rendered images
  • Charts display as embedded PNG images
  • No JavaScript required in email client
  • Email sending is fully active: TemplateVariableReplacementJob triggers CommsDeliverySendJob after variable resolution
  • ApplicationMailer.delivery_email sends HTML emails with embedded chart images

Technical Implementation

Services and Utilities Involved

Service/Utility Purpose
ChartDataService Fetches time-series data (daily/weekly/monthly) from CheckInSubmission
DynamicComponentRendererService Generates Chart.js HTML with data and styling
DynamicComponentImageService Converts HTML to PNG via Grover, stores in ActiveStorage
DynamicComponentImageGenerationJob Background job for async image generation
ChartTypeExtractor Utility module that extracts chart type (BAR/LINE/PIE) from labels for generic CHART tokens

ChartTypeExtractor Utility

Purpose: Extracts the actual chart type (BAR, LINE, or PIE) from a label string for generic CHART tokens.

How it works: - Searches for chart type keywords in the label (case-insensitive) - Looks for "BAR", "LINE", or "PIE" in the label text - Returns the first matching type found - Falls back to "BAR" if no type is detected

Usage: ruby ChartTypeExtractor.extract_chart_type_from_label("Bar Chart") # => "BAR" ChartTypeExtractor.extract_chart_type_from_label("Line Graph") # => "LINE" ChartTypeExtractor.extract_chart_type_from_label("Pie Chart") # => "PIE" ChartTypeExtractor.extract_chart_type_from_label("My Chart") # => "BAR" (fallback)

Integration: - Used by TemplateVariableResolverService when parsing CHART tokens - Used by DynamicComponentImageGenerationJob for chart type detection - Used by DynamicComponentPreviewHelper for preview rendering

ChartDataService Details

# X-axis options
X_AXIS_TYPES = %w[day weekly monthly].freeze

# Y-axis metrics mapped to calculations
METRIC_MAPPINGS = {
  'baseline_morale' => :morale,
  'this_months_morale' => :morale,
  'engagement_rate' => :engagement,
  'action_closure_rate' => :engagement,
  'responses_count' => :responses
}

Data Fetching: - daily: Current month, day by day (Dec 01, Dec 02, etc.) - weekly: Last 3 months, week by week (Week of Oct 7, Week of Oct 14, etc.) - monthly: Last 6 months, month by month (July 2025, August 2025, etc.)

Image Generation Settings

Setting Value
Viewport 800x600 pixels
Format PNG
Scale 2x (for high resolution)
Timeout 30 seconds
Chart.js Version 4.4.1 (CDN)

Chart Limitations

  1. Data Source: Charts use data from CheckInSubmission records only
  2. Static Images: Charts are rendered as static PNG images (no interactivity in emails)
  3. Size: Charts are rendered at 800x600px (optimal for email)
  4. Complexity: Keep charts simple for better email rendering
  5. Performance: Chart generation adds ~2-5 seconds per chart (done immediately during delivery creation)

Best Practices

  1. Keep It Simple: Use 1-3 data points per chart for clarity
  2. Meaningful Titles: Always provide clear titles
  3. Color Consistency: Use consistent colors across your templates
  4. Legend Placement: Bottom or top works best for email rendering
  5. Test First: Preview deliveries before sending to verify chart appearance
  6. Date Period Selection: Choose appropriate granularity (daily for detail, monthly for trends)

Debugging Chart Issues

Chart not showing in preview? 1. Check that chart token syntax is correct (use BAR/LINE/PIE/CHART) 2. If using CHART type, ensure label contains chart type keyword (Bar, Line, Pie) 3. Verify the delivery has been created and processed 4. Check logs for DynamicComponentImageGenerationJob errors 5. Ensure Grover/Node.js is working: which node

Wrong data displayed? 1. Verify x-axis period is correct (daily, weekly, monthly) 2. Check y-axis variable name matches available metrics 3. Ensure project has CheckInSubmission data for the selected period 4. For CHART type, verify ChartTypeExtractor correctly identified the chart type from label

Image quality issues? 1. Charts render at 800x600px with 2x scaling for high quality 2. Consider simplifying complex charts with many data points

Usage Example (Full Template): ``` Subject: Weekly Report for {{project_name}}

Body: This month's morale: {{thismonthsmorale}} Baseline morale: {{baselinemorale}} Engagement rate: {{engagementrate}}% Responses received: {{responses_count}} ```

Variable Resolution Process

The TemplateVariableResolverService handles all variable resolution with several performance optimizations:

1. Variable Extraction (Performance Optimization)

What it does: - Scans template content (subject and body) for {{variable_name}} patterns using regex /\{\{([^}]+)\}\}/ - Extracts only variables that are actually used in the template - Uses Ruby Set to automatically deduplicate variable names

Why it matters: - If a template only uses {{project_name}}, the system won't calculate {{multi_project_average_morale}} - Prevents unnecessary database queries and calculations - Significantly improves performance for templates with few variables

Example: ```ruby

Template content: "Hello {{projectname}}, your morale is {{thismonths_morale}}"

Extracted variables: ['projectname', 'thismonths_morale']

System only resolves these two variables, not all 7


#### 2. Memoization (N+1 Query Prevention)

**What it is:**
- Stores fetched data in instance variables (`@memoized_data`) during resolution
- Prevents fetching the same data multiple times
- No external cache system required (Cloud66 compatible)

**How it works:**
```ruby
# First call: Fetches from DB, stores in @memoized_data
submissions = memoized_submissions_for_month  # DB query

# Subsequent calls: Returns stored data
submissions = memoized_submissions_for_month  # No DB query!

Benefits: - Solves N+1 query problem: Fetch once, use many times - Faster resolution: Avoids repeated database queries - Memory efficient: Data cleared after resolution completes - No external dependencies: Just instance variables

Memoized Data Types: - Submission data for current month - Submission data for specific date ranges - Submission data for specific projects - Database credentials and server connections

3. Two-Tier Data Fetching Strategy

Priority Order:

  1. CheckInSubmission Model (Preferred)

    • Uses ActiveRecord connection pool
    • Faster and more efficient
    • Better connection management
    • Used first for all queries
    • Uses pluck(:created_at, :score) for performance

  2. Analytics DB (Fallback)

    • Only used if CheckInSubmission model has no data
    • Uses Sequel connection with with_analytics_connection
    • 5-second timeout to prevent hanging on slow/failed connections
    • Properly closes connections after use
    • Limits results to 10,000 records to prevent memory issues

Why Two-Tier? - CheckInSubmission model may be syncing or empty - Analytics DB has historical data but is slower - Fallback ensures variables can still be resolved - Timeout prevents hanging on failed connections

4. Date Range Handling

Current Month Calculations: - All morale/engagement calculations use current month (1st to last day) - Date range: Date.current.beginning_of_month..Date.current.end_of_month

Baseline Morale: - Uses project's default baseline range (configured per project) - Options: last_3_months, last_6_months, last_12_months, etc. - Calculated via BaselineRangeCalculator.default_for_project(project)

5. Error Handling

Unresolved Variables: - If a variable cannot be resolved, it's replaced with: [no template found for {{variable_name}}] - System continues processing other variables - Delivery is not failed due to unresolved variables (handled gracefully)

Timeout Protection: - 5-second timeout for analytics DB queries - Prevents hanging on slow/failed connections - Logs warnings but doesn't fail the resolution

6. Variable Resolution Flow

1. Extract variables from template → ['project_name', 'this_months_morale']
2. For each variable:
   a. Check memoized data (if exists, use it)
   b. If not memoized:
      - Try CheckInSubmission model first
      - If no data, try Analytics DB (with timeout)
      - Store result in memoized data
   c. Calculate value (morale, engagement, etc.)
3. Replace {{variables}} in template with calculated values
4. Return resolved JSON payload

Variable Resolution Timing

Variables are resolved at three different times for different purposes:

  1. Immediate Resolution (Delivery Creation)

    • When: Right after CommsDelivery is created
    • Purpose: Store resolved content for preview
    • Service: TemplateVariableResolverService
    • Result: merged_payload updated with resolved content

  2. Preview Resolution (On-Demand)

    • When: User clicks "Preview" button
    • Purpose: Show real-time preview with current data
    • Service: TemplateVariableResolverService (with 10-second timeout)
    • Result: Display resolved content in preview modal
    • Fallback: If resolution fails, shows unresolved payload

  3. Send-Time Resolution (Background Job)

    • When: At delivery's send_at time
    • Purpose: Ensure fresh data at send time, then trigger email sending
    • Service: TemplateVariableReplacementJob (with 30-second timeout)
    • Result: merged_payload updated with latest metrics
    • Triggers: CommsDeliverySendJob for email deliveries after successful resolution
    • Retry: 3 attempts with exponential backoff for transient errors

Performance Optimizations

1. Variable Extraction (Only Resolve What's Used)

What it is: - Scans template content to find only variables actually used - Uses regex /\{\{([^}]+)\}\}/ to extract variable names - Only resolves variables found in template, not all available variables

Performance Impact: - If template only uses {{project_name}}, system doesn't calculate {{multi_project_average_morale}} - Prevents unnecessary database queries and calculations - Significantly faster for templates with few variables

Example: ```ruby

Template: "Hello {{project_name}}"

Extracted: ['project_name']

Resolved: Only project_name (1 query)

NOT resolved: thismonthsmorale, baseline_morale, etc. (saves 6+ queries)


### 2. Memoization (Instance Variables)

**What it is:**
- Storing computed values in instance variables (`@memoized_data`) to avoid recalculating them
- Like solving an N+1 query problem - fetch once, store in variable, reuse
- No external cache system - just instance variables
- Cloud66 compatible - no cache configuration needed

**How it works:**
```ruby
# First call: Fetches from DB, stores in @memoized_data
submissions = memoized_submissions_for_month  # DB query

# Subsequent calls: Returns stored data
submissions = memoized_submissions_for_month  # No DB query!

Memoized Data Types: - Submission data for current month (keyed by project + date range) - Submission data for specific date ranges (keyed by project + date range) - Submission data for specific projects (keyed by project + date range) - Database credentials (keyed by project ID) - Database server connections (keyed by credential ID)

Benefits: - Solves N+1 queries: Fetch once, use many times - No external dependencies: Just instance variables - Cloud66 compatible: No cache configuration needed - Faster: Avoids repeated database queries - Memory efficient: Data cleared after resolution completes

3. Two-Tier Data Fetching

Strategy: 1. Try ActiveRecord model first (faster, uses connection pool) 2. Fallback to analytics DB only if model has no data

Performance Impact: - Most queries use ActiveRecord (fast) - Only slow analytics DB queries when necessary - 5-second timeout prevents hanging

4. Efficient Database Queries

Techniques: - Uses pluck(:created_at, :score) instead of loading full records - Only fetches required columns - Limits analytics DB queries to 10,000 records - Uses date range queries instead of scanning all data

5. Storing Resolved Content

Implementation: - Resolved email content is stored in comms_deliveries.merged_payload column - Variables are resolved immediately when delivery is created - Preview reads from database (instant lookup, no recalculation) - Background job re-resolves at send time for fresh data

What's stored: json { "subject": "Weekly Report for Team Madrid", "body": "This month's morale: 45\nBaseline morale: 59\nEngagement: 80%\nResponses: 12", "recipient": { "name": "John Doe", "email": "john@example.com", "stakeholder_id": 38, "project_id": 74 } }

Benefits: - Preview is instant (reads from DB, no calculation) - No recalculation needed for preview - All variables already replaced - no {{variables}} left in stored content - Background job only re-resolves at send time (for fresh data)

6. Timeout Protection

Where it's used: - Preview resolution: 10-second timeout - Background job resolution: 30-second timeout - Analytics DB queries: 5-second timeout

Benefits: - Prevents hanging operations - Graceful fallback if timeout occurs - Better user experience (doesn't freeze UI) - Prevents job queue from backing up

Technical Notes

Data Storage

  • All JSON fields use Rails' native JSON support
  • Timestamps are stored in UTC
  • Resolved template content is stored in comms_deliveries.merged_payload (JSON column)

Relationships

  • Polymorphic relationships use scope_type and scope_ref pattern (e.g., CadenceRule)
  • Many-to-many relationships use join tables (e.g., StakeholderProject)

Delivery Management

  • Delivery status transitions are managed by the system
  • Failed deliveries can be retried if status is failed
  • Cancelled deliveries cannot be retried
  • Delivery statuses: scheduled, pending, paused, sent, delivered, failed, cancelled

Cadence Rules

  • Cadence rules are checked synchronously before delivery creation
  • Rules can be set at persona, stakeholder, or project level
  • Polymorphic relationship via scope_type and scope_ref

Performance

  • Overview tab metrics are calculated on-demand - performance may degrade with large datasets
  • Variable resolution uses memoization (instance variables) to avoid repeated queries
  • Only variables actually used in templates are resolved (performance optimization)
  • Resolved content is stored in merged_payload for fast preview lookups
  • Two-tier data fetching: ActiveRecord model first, analytics DB fallback

Template Variable Resolution

  • Variables are resolved at three times:
    1. Immediately when delivery is created (for preview)
    2. On-demand when user clicks preview (with 10s timeout)
    3. At send time via background job (with 30s timeout, retry logic)
  • Uses memoization to prevent N+1 queries
  • Extracts only variables actually used in template
  • Extracts variables from dynamic component tokens (GAUGE, THEME, LIST, charts)
  • For charts, resolves variables in configuration parameters (x, y, data)
  • Two-tier fetching: CheckInSubmission model first, analytics DB fallback
  • Timeout protection at all levels (5s for analytics DB, 10s for preview, 30s for job)
  • Supports generic CHART type with automatic type extraction via ChartTypeExtractor

Background Jobs

  • TemplateVariableReplacementJob runs at delivery's send_at time
    • Retry logic: 3 attempts with exponential backoff for transient errors
    • Discards timeout errors (won't benefit from retry)
    • Re-raises connection timeout errors (transient, should retry)
    • 30-second timeout wrapper around variable resolution
    • Detects unresolved variables by checking for {{ pattern
    • Triggers CommsDeliverySendJob after successful variable resolution for email deliveries
  • CommsDeliverySendJob sends email deliveries after variable resolution
    • Retry logic: 3 attempts with exponential backoff for transient errors
    • Validates delivery and payload before sending
    • Checks send_at time has passed (with 5-second buffer)
    • Calls ApplicationMailer.delivery_email to send email
    • Updates delivery status and creates CommsEvent records
    • Discards on deserialization errors
  • DynamicComponentImageGenerationJob handles async component image generation
    • Retry logic: 4 attempts for StandardError, 3 attempts for Timeout::Error
    • Supports all component types including charts
    • Handles both token-based and text-pattern-based component detection
    • Falls back to text placeholder on final retry failure

Project Filtering

  • Supports three modes: 'include'/'specific', 'exclude', and default (all)
  • 'specific' mode is kept for backward compatibility (treated as 'include')
  • Creates one delivery per project context for each stakeholder
  • Creates delivery with project_id: nil if no projects remain after filtering

Template Editing

  • Line breaks normalized when saving: \r\n → \n → <br>
  • Line breaks converted back when editing: <br> → \n
  • Ensures consistent display and natural editing experience

Error Handling

  • Variable resolution failures don't block delivery creation
  • Preview gracefully falls back to unresolved payload if resolution fails
  • Background job handles errors appropriately (retry vs discard)
  • All errors are logged for debugging

Recent Improvements

Email Sending Functionality (Latest Update)

Email Automation: - ✅ CommsDeliverySendJob - Background job that sends email deliveries - ✅ ApplicationMailer.delivery_email - Method that renders and sends HTML emails - ✅ Email template at app/views/application_mailer/delivery_email.html.erb - ✅ TemplateVariableReplacementJob now triggers CommsDeliverySendJob after variable resolution - ✅ Full email sending flow: Variable resolution → Email send job → Mailer → Email delivery - ✅ Error handling with retry logic and status tracking - ✅ Support for HTML emails with embedded chart images

Email Features: - ✅ Extracts subject and body from merged_payload - ✅ Supports HTML content with embedded images - ✅ Plain text fallback for text-only email clients - ✅ Reply-to address configuration via environment variables - ✅ Delivery status tracking (pending → sent → delivered) - ✅ CommsEvent records for email tracking

Image Rendering Improvements: - ✅ Chart images generated immediately during delivery creation (not deferred) - ✅ Images stored in ActiveStorage with public URLs - ✅ <img> tags embedded directly in email HTML - ✅ Improved chart data handling in DynamicComponentRendererService - ✅ Better error handling with text placeholder fallbacks - ✅ Support for both immediate and async image generation

Template Variable System Enhancements

Performance Optimizations: - ✅ Variable extraction: Only resolves variables actually used in templates (prevents unnecessary queries) - ✅ Memoization: Prevents N+1 queries by caching fetched data in instance variables - ✅ Two-tier data fetching: ActiveRecord model first, analytics DB fallback with timeout protection - ✅ Efficient queries: Uses pluck for fetching only required columns

New Template Variables: - ✅ {{this_months_morale}} - Current month's team morale score - ✅ {{baseline_morale}} - Baseline morale using project's baseline range - ✅ {{engagement_rate}} - Engagement rate as percentage - ✅ {{action_closure_rate}} - Action closure rate as percentage - ✅ {{responses_count}} - Count of check-in submissions - ✅ {{multi_project_average_morale}} - Average morale across all stakeholder's projects

Reliability Improvements: - ✅ Timeout protection: 5s for analytics DB, 10s for preview, 30s for background job - ✅ Retry logic: 3 attempts with exponential backoff for transient errors - ✅ Smart error handling: Discards timeout errors, retries connection errors - ✅ Graceful fallbacks: Preview shows unresolved payload if resolution fails

User Experience: - ✅ Real-time variable resolution in delivery previews (shows actual values, not {{variables}}) - ✅ Immediate variable resolution when creating deliveries (enables instant preview) - ✅ Line break normalization for better template editing experience - ✅ Smart workflow action buttons (Activate/Pause/Resume based on state)

Project Filtering: - ✅ Support for 'exclude' project filter mode (in addition to 'include') - ✅ Better project context handling (creates deliveries per project context) - ✅ Backward compatible (supports legacy 'specific' mode)

Code Quality: - ✅ Better error logging and debugging information - ✅ Comprehensive timeout protection at all levels - ✅ Improved code organization and documentation

Chart System Enhancements: - ✅ Generic CHART type support with automatic type extraction from labels - ✅ ChartTypeExtractor utility for intelligent chart type detection - ✅ Variable resolution in chart configuration parameters (x, y, data) - ✅ Support for all chart types (BAR, LINE, PIE) via specific or generic tokens - ✅ Improved chart token parsing for both specific and generic formats

Future Enhancements

Potential areas for expansion: - A/B testing for templates - Advanced scheduling (timezone-aware) - Template versioning - Delivery analytics dashboard - Webhook integrations for external systems - Template preview/testing interface - Bulk operations for stakeholders - Additional template variables (custom calculations) - Template variable validation and testing