timebank-cc-public/references/BOUNCE_SETUP.md

Universal Email Bounce Handling System

Overview

This system provides universal bounce handling for ALL mailables that works with any SMTP server by:

1. Automatically intercepting all outgoing emails via Laravel's MessageSending event
2. Checking for suppressed recipients before emails are sent
3. Adding bounce tracking headers to all outgoing emails
4. Processing bounce emails from a dedicated mailbox
5. Using configurable thresholds to suppress emails and reset verification status
6. Providing conservative bounce counting to prevent false positives

🎯 Key Feature: This system works automatically with ALL existing and future mailables without requiring code changes!

Setup Steps

1. Configure Bounce Email Address

Add these to your .env file:

# Bounce handling configuration
BOUNCE_PROCESSING_ENABLED=true  # Set to false on local/staging environments without IMAP
MAIL_BOUNCE_ADDRESS=bounces@yourdomain.org
BOUNCE_MAILBOX=bounces@yourdomain.org
BOUNCE_HOST=imap.yourdomain.org
BOUNCE_PORT=993
BOUNCE_PROTOCOL=imap
BOUNCE_USERNAME=bounces@yourdomain.org
BOUNCE_PASSWORD=your-bounce-mailbox-password
BOUNCE_SSL=true

Important: Set BOUNCE_PROCESSING_ENABLED=false on local development and staging environments that don't have access to the bounce mailbox to prevent IMAP connection errors.

2. Create Bounce Email Address

Create a dedicated email address (e.g., bounces@yourdomain.org) that will receive bounce notifications:

• Set up the email account on your email server
• Configure IMAP access for programmatic reading

3. Configure Your SMTP Server

Most SMTP servers will respect the Return-Path header and send bounces to that address automatically.

4. Process Bounces

Run the bounce processing command periodically:

# Process bounces (dry run first to test)
php artisan mailings:process-bounces --dry-run

# Process bounces for real
php artisan mailings:process-bounces --delete

# Or use command options instead of config file
php artisan mailings:process-bounces \
    --mailbox=bounces@yourdomain.org \
    --host=imap.yourdomain.org \
    --username=bounces@yourdomain.org \
    --password=your-password \
    --ssl \
    --delete

5. Schedule Automatic Processing

Add to your app/Console/Kernel.php:

protected function schedule(Schedule $schedule)
{
    // Process bounces every hour
    $schedule->command('mailings:process-bounces --delete')
             ->hourly()
             ->withoutOverlapping();
}

Threshold Configuration

Configure bounce thresholds in config/timebank-cc.php:

'bounce_thresholds' => [
    // Number of hard bounces before email is suppressed from future mailings
    'suppression_threshold' => 3,

    // Number of hard bounces before email_verified_at is set to null
    'verification_reset_threshold' => 2,

    // Time window in days to count bounces (prevents old bounces from accumulating)
    'counting_window_days' => 30,

    // Only count these bounce types toward thresholds
    'counted_bounce_types' => ['hard'],

    // Specific bounce reasons that count as definitive hard bounces
    'definitive_hard_bounce_patterns' => [
        'user unknown', 'no such user', 'mailbox unavailable',
        'does not exist', 'invalid recipient', 'address rejected',
        '5.1.1', '5.1.2', '5.1.3', '550', '551',
    ],
]

Universal System Architecture

Automatic Email Interception

The system uses Laravel's MessageSending event to automatically:

• Intercept ALL outgoing emails from any mailable class
• Check recipients against the suppression list
• Block emails to suppressed addresses (logs the action)
• Add bounce tracking headers to all outgoing emails
• Remove suppressed recipients from multi-recipient emails

No Code Changes Required

• Works with existing mailables: ContactFormMailable, TransferReceived, NewMessageMail, etc.
• Works with future mailables automatically
• Works with Mail::to(), Mail::queue(), and Mail::later() methods
• Works with both sync and queued emails

Enhanced Mailables (Optional)

For enhanced bounce tracking, mailables can optionally:

• Extend BounceTrackingMailable base class
• Use TracksBounces trait for additional tracking features

Commands Available

Test Universal System

# Test with normal (non-suppressed) email
php artisan test:universal-bounce --scenario=normal --email=test@example.com

# Test with suppressed email (should be blocked)
php artisan test:universal-bounce --scenario=suppressed --email=test@example.com

# Test with mixed recipients
php artisan test:universal-bounce --scenario=mixed

Process Bounce Emails

php artisan mailings:process-bounces [options]

Manage Bounced Emails

# Show comprehensive bounce statistics with threshold info
php artisan mailings:manage-bounces stats

# List bounced emails
php artisan mailings:manage-bounces list

# Check bounce counts for a specific email
php artisan mailings:manage-bounces check-thresholds --email=user@example.com

# Check all emails against current thresholds
php artisan mailings:manage-bounces check-thresholds

# Suppress a specific email
php artisan mailings:manage-bounces suppress --email=problem@example.com

# Unsuppress an email
php artisan mailings:manage-bounces unsuppress --email=fixed@example.com

# Cleanup old soft bounces (older than 90 days)
php artisan mailings:manage-bounces cleanup --days=90

How It Works

1. Outgoing Emails: Each newsletter email gets:

   • Return-Path header set to your bounce address
   • X-Mailing-ID header for tracking
   • X-Recipient-Email header for identification

2. Bounce Detection: When an email bounces:

   • SMTP server sends bounce notification to Return-Path address
   • Bounce processor reads the dedicated mailbox
   • Extracts recipient email and bounce type from bounce message
   • Records bounce in database

3. Threshold-Based Actions:

   • System counts definitive hard bounces within a time window (default: 30 days)
   • After 2 hard bounces (default): email_verified_at is set to null for all profiles
   • After 3 hard bounces (default): Email is suppressed from future mailings
   • Only specific bounce patterns count toward thresholds (prevents false positives)

4. Conservative Approach:

   • Only "definitive" hard bounces count (user unknown, domain invalid, etc.)
   • Time window prevents old bounces from accumulating indefinitely
   • Configurable thresholds allow fine-tuning for your use case

5. Integration:

   • SendBulkMailJob checks for suppressed emails before sending
   • Mailing.getRecipientsQuery() excludes suppressed emails
   • Bounce detection works with any SMTP server
   • Multi-profile support: affects Users, Organizations, Banks, and Admins

Bounce Types

• Hard Bounce: Permanent delivery failure (user doesn't exist, domain invalid)

  • Counts toward suppression and verification reset thresholds
  • Only "definitive" patterns count (configured in bounce_thresholds)

• Soft Bounce: Temporary failure (mailbox full, server temporarily unavailable)

  • Recorded but doesn't count toward thresholds
  • Not automatically suppressed

• Unknown: Could not determine bounce type from message content

  • Recorded but doesn't count toward thresholds

Threshold Benefits

• Prevents False Positives: Temporary server issues won't immediately suppress emails
• Gradual Response: Reset verification before full suppression
• Time-Based: Old bounces don't accumulate indefinitely
• Conservative: Only definitive bounce patterns count
• Configurable: Adjust thresholds for your sending patterns

Testing

Automated Testing with Local Development

For testing with Mailpit (local SMTP server):

# Test the threshold system with simulated bounces
php artisan test:bounce-system --scenario=single                    # Test single bounce (no action)
php artisan test:bounce-system --scenario=threshold-verification    # Test verification reset (2 bounces)
php artisan test:bounce-system --scenario=threshold-suppression     # Test suppression (3 bounces)
php artisan test:bounce-system --scenario=multiple                  # Test all scenarios

# Test email sending integration
php artisan test:mailpit-integration --send-test                    # Send test mailing email via Mailpit
php artisan test:mailpit-integration --test-suppression             # Verify suppressed emails are blocked

# View results
php artisan mailings:manage-bounces stats                           # Show bounce statistics
php artisan mailings:manage-bounces check-thresholds               # Check all emails against thresholds

Production Testing

1. Set up a test bounce mailbox
2. Send a test mailing to a non-existent address
3. Check that bounce appears in the bounce mailbox
4. Run php artisan mailings:process-bounces --dry-run to test parsing
5. Verify the bounce is correctly detected and categorized

What to Expect

• 1 Hard Bounce: Recorded but no action taken
• 2 Hard Bounces: email_verified_at set to null for all profiles
• 3 Hard Bounces: Email suppressed from future mailings
• Email Sending: Suppressed emails are automatically skipped during bulk sends

Troubleshooting

• Check IMAP/POP3 credentials and server settings
• Verify Return-Path is being set correctly on outgoing emails
• Test bounce mailbox connection manually
• Check Laravel logs for bounce processing errors
• Use --dry-run flag to test without making changes