# Universal Email Bounce Handling System

## Overview

This system provides **universal bounce handling for ALL mailables** that works with any SMTP server by:
1. **Automatically intercepting all outgoing emails** via Laravel's MessageSending event
2. **Checking for suppressed recipients** before emails are sent
3. **Adding bounce tracking headers** to all outgoing emails
4. Processing bounce emails from a dedicated mailbox
5. Using configurable thresholds to suppress emails and reset verification status
6. Providing conservative bounce counting to prevent false positives

**🎯 Key Feature**: This system works automatically with **ALL existing and future mailables** without requiring code changes!

## Setup Steps

### 1. Configure Bounce Email Address

Add these to your `.env` file:

```env
# Bounce handling configuration
BOUNCE_PROCESSING_ENABLED=true  # Set to false on local/staging environments without IMAP
MAIL_BOUNCE_ADDRESS=bounces@yourdomain.org
BOUNCE_MAILBOX=bounces@yourdomain.org
BOUNCE_HOST=imap.yourdomain.org
BOUNCE_PORT=993
BOUNCE_PROTOCOL=imap
BOUNCE_USERNAME=bounces@yourdomain.org
BOUNCE_PASSWORD=your-bounce-mailbox-password
BOUNCE_SSL=true
```

**Important**: Set `BOUNCE_PROCESSING_ENABLED=false` on local development and staging environments that don't have access to the bounce mailbox to prevent IMAP connection errors.

### 2. Create Bounce Email Address

Create a dedicated email address (e.g., `bounces@yourdomain.org`) that will receive bounce notifications:
- Set up the email account on your email server
- Configure IMAP access for programmatic reading

### 3. Configure Your SMTP Server

Most SMTP servers will respect the Return-Path header and send bounces to that address automatically.

### 4. Process Bounces

Run the bounce processing command periodically:

```bash
# Process bounces (dry run first to test)
php artisan mailings:process-bounces --dry-run

# Process bounces for real
php artisan mailings:process-bounces --delete

# Or use command options instead of config file
php artisan mailings:process-bounces \
    --mailbox=bounces@yourdomain.org \
    --host=imap.yourdomain.org \
    --username=bounces@yourdomain.org \
    --password=your-password \
    --ssl \
    --delete
```

### 5. Schedule Automatic Processing

Add to your `app/Console/Kernel.php`:

```php
protected function schedule(Schedule $schedule)
{
    // Process bounces every hour
    $schedule->command('mailings:process-bounces --delete')
             ->hourly()
             ->withoutOverlapping();
}
```

## Threshold Configuration

Configure bounce thresholds in `config/timebank-cc.php`:

```php
'bounce_thresholds' => [
    // Number of hard bounces before email is suppressed from future mailings
    'suppression_threshold' => 3,

    // Number of hard bounces before email_verified_at is set to null
    'verification_reset_threshold' => 2,

    // Time window in days to count bounces (prevents old bounces from accumulating)
    'counting_window_days' => 30,

    // Only count these bounce types toward thresholds
    'counted_bounce_types' => ['hard'],

    // Specific bounce reasons that count as definitive hard bounces
    'definitive_hard_bounce_patterns' => [
        'user unknown', 'no such user', 'mailbox unavailable',
        'does not exist', 'invalid recipient', 'address rejected',
        '5.1.1', '5.1.2', '5.1.3', '550', '551',
    ],
]
```

## Universal System Architecture

### Automatic Email Interception
The system uses Laravel's `MessageSending` event to automatically:
- **Intercept ALL outgoing emails** from any mailable class
- **Check recipients against the suppression list**
- **Block emails** to suppressed addresses (logs the action)
- **Add bounce tracking headers** to all outgoing emails
- **Remove suppressed recipients** from multi-recipient emails

### No Code Changes Required
- Works with **existing mailables**: `ContactFormMailable`, `TransferReceived`, `NewMessageMail`, etc.
- Works with **future mailables** automatically
- Works with **Mail::to()**, **Mail::queue()**, and **Mail::later()** methods
- Works with both **sync and queued** emails

### Enhanced Mailables (Optional)
For enhanced bounce tracking, mailables can optionally:
- Extend `BounceTrackingMailable` base class
- Use `TracksBounces` trait for additional tracking features

## Commands Available

### Test Universal System
```bash
# Test with normal (non-suppressed) email
php artisan test:universal-bounce --scenario=normal --email=test@example.com

# Test with suppressed email (should be blocked)
php artisan test:universal-bounce --scenario=suppressed --email=test@example.com

# Test with mixed recipients
php artisan test:universal-bounce --scenario=mixed
```

### Process Bounce Emails
```bash
php artisan mailings:process-bounces [options]
```

### Manage Bounced Emails
```bash
# Show comprehensive bounce statistics with threshold info
php artisan mailings:manage-bounces stats

# List bounced emails
php artisan mailings:manage-bounces list

# Check bounce counts for a specific email
php artisan mailings:manage-bounces check-thresholds --email=user@example.com

# Check all emails against current thresholds
php artisan mailings:manage-bounces check-thresholds

# Suppress a specific email
php artisan mailings:manage-bounces suppress --email=problem@example.com

# Unsuppress an email
php artisan mailings:manage-bounces unsuppress --email=fixed@example.com

# Cleanup old soft bounces (older than 90 days)
php artisan mailings:manage-bounces cleanup --days=90
```

## How It Works

1. **Outgoing Emails**: Each newsletter email gets:
   - Return-Path header set to your bounce address
   - X-Mailing-ID header for tracking
   - X-Recipient-Email header for identification

2. **Bounce Detection**: When an email bounces:
   - SMTP server sends bounce notification to Return-Path address
   - Bounce processor reads the dedicated mailbox
   - Extracts recipient email and bounce type from bounce message
   - Records bounce in database

3. **Threshold-Based Actions**:
   - System counts definitive hard bounces within a time window (default: 30 days)
   - After 2 hard bounces (default): `email_verified_at` is set to `null` for all profiles
   - After 3 hard bounces (default): Email is suppressed from future mailings
   - Only specific bounce patterns count toward thresholds (prevents false positives)

4. **Conservative Approach**:
   - Only "definitive" hard bounces count (user unknown, domain invalid, etc.)
   - Time window prevents old bounces from accumulating indefinitely
   - Configurable thresholds allow fine-tuning for your use case

5. **Integration**:
   - `SendBulkMailJob` checks for suppressed emails before sending
   - `Mailing.getRecipientsQuery()` excludes suppressed emails
   - Bounce detection works with any SMTP server
   - Multi-profile support: affects Users, Organizations, Banks, and Admins

## Bounce Types

- **Hard Bounce**: Permanent delivery failure (user doesn't exist, domain invalid)
  - Counts toward suppression and verification reset thresholds
  - Only "definitive" patterns count (configured in bounce_thresholds)

- **Soft Bounce**: Temporary failure (mailbox full, server temporarily unavailable)
  - Recorded but doesn't count toward thresholds
  - Not automatically suppressed

- **Unknown**: Could not determine bounce type from message content
  - Recorded but doesn't count toward thresholds

## Threshold Benefits

- **Prevents False Positives**: Temporary server issues won't immediately suppress emails
- **Gradual Response**: Reset verification before full suppression
- **Time-Based**: Old bounces don't accumulate indefinitely
- **Conservative**: Only definitive bounce patterns count
- **Configurable**: Adjust thresholds for your sending patterns

## Testing

### Automated Testing with Local Development

For testing with Mailpit (local SMTP server):

```bash
# Test the threshold system with simulated bounces
php artisan test:bounce-system --scenario=single                    # Test single bounce (no action)
php artisan test:bounce-system --scenario=threshold-verification    # Test verification reset (2 bounces)
php artisan test:bounce-system --scenario=threshold-suppression     # Test suppression (3 bounces)
php artisan test:bounce-system --scenario=multiple                  # Test all scenarios

# Test email sending integration
php artisan test:mailpit-integration --send-test                    # Send test mailing email via Mailpit
php artisan test:mailpit-integration --test-suppression             # Verify suppressed emails are blocked

# View results
php artisan mailings:manage-bounces stats                           # Show bounce statistics
php artisan mailings:manage-bounces check-thresholds               # Check all emails against thresholds
```

### Production Testing

1. Set up a test bounce mailbox
2. Send a test mailing to a non-existent address
3. Check that bounce appears in the bounce mailbox
4. Run `php artisan mailings:process-bounces --dry-run` to test parsing
5. Verify the bounce is correctly detected and categorized

### What to Expect

- **1 Hard Bounce**: Recorded but no action taken
- **2 Hard Bounces**: `email_verified_at` set to `null` for all profiles
- **3 Hard Bounces**: Email suppressed from future mailings
- **Email Sending**: Suppressed emails are automatically skipped during bulk sends

## Troubleshooting

- Check IMAP/POP3 credentials and server settings
- Verify Return-Path is being set correctly on outgoing emails
- Test bounce mailbox connection manually
- Check Laravel logs for bounce processing errors
- Use `--dry-run` flag to test without making changes