Initial commit

2026-03-23 21:37:59 +01:00
commit 2547717edb
2193 changed files with 972171 additions and 0 deletions
--- a/references/BOUNCE_SETUP.md
+++ b/references/BOUNCE_SETUP.md
@@ -0,0 +1,266 @@
+# Universal Email Bounce Handling System
+
+## Overview
+
+This system provides **universal bounce handling for ALL mailables** that works with any SMTP server by:
+1. **Automatically intercepting all outgoing emails** via Laravel's MessageSending event
+2. **Checking for suppressed recipients** before emails are sent
+3. **Adding bounce tracking headers** to all outgoing emails
+4. Processing bounce emails from a dedicated mailbox
+5. Using configurable thresholds to suppress emails and reset verification status
+6. Providing conservative bounce counting to prevent false positives
+
+**🎯 Key Feature**: This system works automatically with **ALL existing and future mailables** without requiring code changes!
+
+## Setup Steps
+
+### 1. Configure Bounce Email Address
+
+Add these to your `.env` file:
+
+```env
+# Bounce handling configuration
+BOUNCE_PROCESSING_ENABLED=true  # Set to false on local/staging environments without IMAP
+MAIL_BOUNCE_ADDRESS=bounces@yourdomain.org
+BOUNCE_MAILBOX=bounces@yourdomain.org
+BOUNCE_HOST=imap.yourdomain.org
+BOUNCE_PORT=993
+BOUNCE_PROTOCOL=imap
+BOUNCE_USERNAME=bounces@yourdomain.org
+BOUNCE_PASSWORD=your-bounce-mailbox-password
+BOUNCE_SSL=true
+```
+
+**Important**: Set `BOUNCE_PROCESSING_ENABLED=false` on local development and staging environments that don't have access to the bounce mailbox to prevent IMAP connection errors.
+
+### 2. Create Bounce Email Address
+
+Create a dedicated email address (e.g., `bounces@yourdomain.org`) that will receive bounce notifications:
+- Set up the email account on your email server
+- Configure IMAP access for programmatic reading
+
+### 3. Configure Your SMTP Server
+
+Most SMTP servers will respect the Return-Path header and send bounces to that address automatically.
+
+### 4. Process Bounces
+
+Run the bounce processing command periodically:
+
+```bash
+# Process bounces (dry run first to test)
+php artisan mailings:process-bounces --dry-run
+
+# Process bounces for real
+php artisan mailings:process-bounces --delete
+
+# Or use command options instead of config file
+php artisan mailings:process-bounces \
+    --mailbox=bounces@yourdomain.org \
+    --host=imap.yourdomain.org \
+    --username=bounces@yourdomain.org \
+    --password=your-password \
+    --ssl \
+    --delete
+```
+
+### 5. Schedule Automatic Processing
+
+Add to your `app/Console/Kernel.php`:
+
+```php
+protected function schedule(Schedule $schedule)
+{
+    // Process bounces every hour
+    $schedule->command('mailings:process-bounces --delete')
+             ->hourly()
+             ->withoutOverlapping();
+}
+```
+
+## Threshold Configuration
+
+Configure bounce thresholds in `config/timebank-cc.php`:
+
+```php
+'bounce_thresholds' => [
+    // Number of hard bounces before email is suppressed from future mailings
+    'suppression_threshold' => 3,
+
+    // Number of hard bounces before email_verified_at is set to null
+    'verification_reset_threshold' => 2,
+
+    // Time window in days to count bounces (prevents old bounces from accumulating)
+    'counting_window_days' => 30,
+
+    // Only count these bounce types toward thresholds
+    'counted_bounce_types' => ['hard'],
+
+    // Specific bounce reasons that count as definitive hard bounces
+    'definitive_hard_bounce_patterns' => [
+        'user unknown', 'no such user', 'mailbox unavailable',
+        'does not exist', 'invalid recipient', 'address rejected',
+        '5.1.1', '5.1.2', '5.1.3', '550', '551',
+    ],
+]
+```
+
+## Universal System Architecture
+
+### Automatic Email Interception
+The system uses Laravel's `MessageSending` event to automatically:
+- **Intercept ALL outgoing emails** from any mailable class
+- **Check recipients against the suppression list**
+- **Block emails** to suppressed addresses (logs the action)
+- **Add bounce tracking headers** to all outgoing emails
+- **Remove suppressed recipients** from multi-recipient emails
+
+### No Code Changes Required
+- Works with **existing mailables**: `ContactFormMailable`, `TransferReceived`, `NewMessageMail`, etc.
+- Works with **future mailables** automatically
+- Works with **Mail::to()**, **Mail::queue()**, and **Mail::later()** methods
+- Works with both **sync and queued** emails
+
+### Enhanced Mailables (Optional)
+For enhanced bounce tracking, mailables can optionally:
+- Extend `BounceTrackingMailable` base class
+- Use `TracksBounces` trait for additional tracking features
+
+## Commands Available
+
+### Test Universal System
+```bash
+# Test with normal (non-suppressed) email
+php artisan test:universal-bounce --scenario=normal --email=test@example.com
+
+# Test with suppressed email (should be blocked)
+php artisan test:universal-bounce --scenario=suppressed --email=test@example.com
+
+# Test with mixed recipients
+php artisan test:universal-bounce --scenario=mixed
+```
+
+### Process Bounce Emails
+```bash
+php artisan mailings:process-bounces [options]
+```
+
+### Manage Bounced Emails
+```bash
+# Show comprehensive bounce statistics with threshold info
+php artisan mailings:manage-bounces stats
+
+# List bounced emails
+php artisan mailings:manage-bounces list
+
+# Check bounce counts for a specific email
+php artisan mailings:manage-bounces check-thresholds --email=user@example.com
+
+# Check all emails against current thresholds
+php artisan mailings:manage-bounces check-thresholds
+
+# Suppress a specific email
+php artisan mailings:manage-bounces suppress --email=problem@example.com
+
+# Unsuppress an email
+php artisan mailings:manage-bounces unsuppress --email=fixed@example.com
+
+# Cleanup old soft bounces (older than 90 days)
+php artisan mailings:manage-bounces cleanup --days=90
+```
+
+## How It Works
+
+1. **Outgoing Emails**: Each newsletter email gets:
+   - Return-Path header set to your bounce address
+   - X-Mailing-ID header for tracking
+   - X-Recipient-Email header for identification
+
+2. **Bounce Detection**: When an email bounces:
+   - SMTP server sends bounce notification to Return-Path address
+   - Bounce processor reads the dedicated mailbox
+   - Extracts recipient email and bounce type from bounce message
+   - Records bounce in database
+
+3. **Threshold-Based Actions**:
+   - System counts definitive hard bounces within a time window (default: 30 days)
+   - After 2 hard bounces (default): `email_verified_at` is set to `null` for all profiles
+   - After 3 hard bounces (default): Email is suppressed from future mailings
+   - Only specific bounce patterns count toward thresholds (prevents false positives)
+
+4. **Conservative Approach**:
+   - Only "definitive" hard bounces count (user unknown, domain invalid, etc.)
+   - Time window prevents old bounces from accumulating indefinitely
+   - Configurable thresholds allow fine-tuning for your use case
+
+5. **Integration**:
+   - `SendBulkMailJob` checks for suppressed emails before sending
+   - `Mailing.getRecipientsQuery()` excludes suppressed emails
+   - Bounce detection works with any SMTP server
+   - Multi-profile support: affects Users, Organizations, Banks, and Admins
+
+## Bounce Types
+
+- **Hard Bounce**: Permanent delivery failure (user doesn't exist, domain invalid)
+  - Counts toward suppression and verification reset thresholds
+  - Only "definitive" patterns count (configured in bounce_thresholds)
+
+- **Soft Bounce**: Temporary failure (mailbox full, server temporarily unavailable)
+  - Recorded but doesn't count toward thresholds
+  - Not automatically suppressed
+
+- **Unknown**: Could not determine bounce type from message content
+  - Recorded but doesn't count toward thresholds
+
+## Threshold Benefits
+
+- **Prevents False Positives**: Temporary server issues won't immediately suppress emails
+- **Gradual Response**: Reset verification before full suppression
+- **Time-Based**: Old bounces don't accumulate indefinitely
+- **Conservative**: Only definitive bounce patterns count
+- **Configurable**: Adjust thresholds for your sending patterns
+
+## Testing
+
+### Automated Testing with Local Development
+
+For testing with Mailpit (local SMTP server):
+
+```bash
+# Test the threshold system with simulated bounces
+php artisan test:bounce-system --scenario=single                    # Test single bounce (no action)
+php artisan test:bounce-system --scenario=threshold-verification    # Test verification reset (2 bounces)
+php artisan test:bounce-system --scenario=threshold-suppression     # Test suppression (3 bounces)
+php artisan test:bounce-system --scenario=multiple                  # Test all scenarios
+
+# Test email sending integration
+php artisan test:mailpit-integration --send-test                    # Send test mailing email via Mailpit
+php artisan test:mailpit-integration --test-suppression             # Verify suppressed emails are blocked
+
+# View results
+php artisan mailings:manage-bounces stats                           # Show bounce statistics
+php artisan mailings:manage-bounces check-thresholds               # Check all emails against thresholds
+```
+
+### Production Testing
+
+1. Set up a test bounce mailbox
+2. Send a test mailing to a non-existent address
+3. Check that bounce appears in the bounce mailbox
+4. Run `php artisan mailings:process-bounces --dry-run` to test parsing
+5. Verify the bounce is correctly detected and categorized
+
+### What to Expect
+
+- **1 Hard Bounce**: Recorded but no action taken
+- **2 Hard Bounces**: `email_verified_at` set to `null` for all profiles
+- **3 Hard Bounces**: Email suppressed from future mailings
+- **Email Sending**: Suppressed emails are automatically skipped during bulk sends
+
+## Troubleshooting
+
+- Check IMAP/POP3 credentials and server settings
+- Verify Return-Path is being set correctly on outgoing emails
+- Test bounce mailbox connection manually
+- Check Laravel logs for bounce processing errors
+- Use `--dry-run` flag to test without making changes