# Queue Workers Setup

## Overview

The application uses separate queue workers for different types of jobs:

- **High Priority Worker**: Handles critical emails (password resets, email verification)
- **Main Worker**: Handles standard emails and jobs (notifications, messages, etc.)
- **Mailing Workers**: Dedicated workers for bulk newsletter processing (3 workers for parallel processing)

## Queue Priority

### High Priority Worker
```bash
php artisan queue:work --queue=high --tries=3 --timeout=90
```

Dedicated worker for critical operations:
- `high` - Password resets, email verification
- Runs independently to ensure these emails are never delayed
- Fast timeout (90 seconds) for quick processing

### Main Worker
```bash
php artisan queue:work --queue=messages,default,emails,low --tries=3 --timeout=90
```

Processes queues in this order:
1. `messages` - Real-time messaging
2. `default` - General jobs
3. `emails` - Standard emails (notifications, contact forms, etc.)
4. `low` - Background tasks

**Note:** The `high` queue has been removed from this worker and now runs independently.

### Mailing Workers
```bash
php artisan queue:work --queue=mailing --tries=3 --timeout=600
```

Dedicated queue for bulk newsletters:
- `mailing` - Bulk newsletter batches
- Runs with 3 parallel workers for faster processing
- 10-minute timeout for large batches
- 3 retry attempts per batch

## Systemd Service Configuration

### High Priority Worker Service

**File:** `/etc/systemd/system/timebank-high-priority-worker.service`

```ini
[Unit]
Description=Timebank High Priority Queue Worker (Password Resets & Email Verification)
After=network.target

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/path/to/your/timebank/installation
ExecStart=/usr/bin/php artisan queue:work --queue=high --tries=3 --timeout=90
Restart=always
RestartSec=10

# Environment
Environment=PATH=/usr/bin:/bin
Environment=TIMEBANK_CONFIG=timebank_cc

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=timebank-high-priority

[Install]
WantedBy=multi-user.target
```

### Main Worker Service

**File:** `/etc/systemd/system/timebank-queue-worker.service`

```ini
[Unit]
Description=Timebank Main Queue Worker
After=network.target

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/path/to/your/timebank/installation
ExecStart=/usr/bin/php artisan queue:work --queue=messages,default,emails,low --tries=3 --timeout=90
Restart=always
RestartSec=10

# Environment
Environment=PATH=/usr/bin:/bin
Environment=TIMEBANK_CONFIG=timebank_cc

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=timebank-queue
    
[Install]
WantedBy=multi-user.target
```

### Mailing Worker Services

Create 3 separate service files for parallel processing:

**File 1:** `/etc/systemd/system/timebank-mailing-worker-1.service`
**File 2:** `/etc/systemd/system/timebank-mailing-worker-2.service`
**File 3:** `/etc/systemd/system/timebank-mailing-worker-3.service`

```ini
[Unit]
Description=Timebank Bulk Mailing Queue Worker 1
After=network.target

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/path/to/your/timebank/installation
ExecStart=/usr/bin/php artisan queue:work --queue=mailing --tries=3 --timeout=600
Restart=always
RestartSec=10

# Environment
Environment=PATH=/usr/bin:/bin
Environment=TIMEBANK_CONFIG=timebank_cc

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=timebank-mailing-1

[Install]
WantedBy=multi-user.target
```

> **Note:** Change `timebank-mailing-1` to `timebank-mailing-2` and `timebank-mailing-3` in the other service files.

## Installation Steps

### 1. Create Service Files

```bash
# Create all 5 service files
sudo nano /etc/systemd/system/timebank-high-priority-worker.service
sudo nano /etc/systemd/system/timebank-queue-worker.service
sudo nano /etc/systemd/system/timebank-mailing-worker-1.service
sudo nano /etc/systemd/system/timebank-mailing-worker-2.service
sudo nano /etc/systemd/system/timebank-mailing-worker-3.service
```

### 2. Update Paths

In each service file, replace:
- `/path/to/your/timebank/installation` with your actual installation path
- `www-data` with your web server user if different
- `TIMEBANK_CONFIG=timebank_cc` with your config name if different, i.e. timebank-default, uuro, etc

### 3. Reload Systemd

```bash
sudo systemctl daemon-reload
```

### 4. Enable Services (start on boot)
sign
```bash
# Enable high priority worker
sudo systemctl enable timebank-high-priority-worker

# Enable main worker
sudo systemctl enable timebank-queue-worker

# Enable mailing workers
sudo systemctl enable timebank-mailing-worker-1
sudo systemctl enable timebank-mailing-worker-2
sudo systemctl enable timebank-mailing-worker-3
```

### 5. Start Services

```bash
# Start high priority worker
sudo systemctl start timebank-high-priority-worker

# Start main worker
sudo systemctl start timebank-queue-worker

# Start mailing workers
sudo systemctl start timebank-mailing-worker-1
sudo systemctl start timebank-mailing-worker-2
sudo systemctl start timebank-mailing-worker-3
```

## Management Commands

### Check Status

```bash
# Check all workers
sudo systemctl status timebank-high-priority-worker
sudo systemctl status timebank-queue-worker
sudo systemctl status timebank-mailing-worker-1
sudo systemctl status timebank-mailing-worker-2
sudo systemctl status timebank-mailing-worker-3

# Or check all at once
sudo systemctl status 'timebank-*'
```

### Stop Workers

```bash
# Stop specific worker
sudo systemctl stop timebank-mailing-worker-1

# Stop all mailing workers
sudo systemctl stop timebank-mailing-worker-{1,2,3}

# Stop all workers
sudo systemctl stop 'timebank-*'
```

### Restart Workers

```bash
# Restart after code deployment
sudo systemctl restart timebank-high-priority-worker
sudo systemctl restart timebank-queue-worker
sudo systemctl restart timebank-mailing-worker-{1,2,3}
```

### View Logs

```bash
# View high priority worker logs
sudo journalctl -u timebank-high-priority-worker -f

# View main worker logs
sudo journalctl -u timebank-queue-worker -f

# View mailing worker logs
sudo journalctl -u timebank-mailing-worker-1 -f

# View all worker logs
sudo journalctl -u 'timebank-*' -f

# View last 100 lines
sudo journalctl -u timebank-queue-worker -n 100
```

## Performance

### With This Setup

- **Critical Emails**: Processed immediately by dedicated high priority worker
  - Password resets: Sent within seconds
  - Email verification: Sent within seconds
  - Never blocked by other jobs

- **Standard Emails**: Processed by main worker
  - Notifications, messages, contact forms
  - Fast processing without bulk mailing delays

- **Bulk Newsletters**: Processed by 3 parallel workers
  - ~36 emails/minute (with 5 sec delay)
  - ~180 emails/minute (with 1 sec delay)
  - 2,182 emails complete in ~60 minutes (5 sec delay) or ~12 minutes (1 sec delay)
  - Completely isolated from critical and standard emails

### Monitoring

```bash
# Watch queue in real-time
php artisan queue:monitor

# Check failed jobs
php artisan queue:failed

# Retry failed jobs
php artisan queue:retry all
```

## Deployment Integration

Add to your deployment script after code updates:

```bash
# Restart all queue workers after deployment
sudo systemctl restart timebank-high-priority-worker
sudo systemctl restart timebank-queue-worker
sudo systemctl restart timebank-mailing-worker-{1,2,3}
```

## Troubleshooting

### Workers Not Processing Jobs

1. Check if services are running:
   ```bash
   sudo systemctl status 'timebank-*'
   ```

2. Check Laravel logs:
   ```bash
   tail -f storage/logs/laravel.log
   ```

3. Check systemd logs:
   ```bash
   sudo journalctl -u timebank-mailing-worker-1 -n 50
   ```

### High Memory Usage

If workers consume too much memory, add memory limits to service files:

```ini
[Service]
# Restart worker after processing 1000 jobs
ExecStart=/usr/bin/php artisan queue:work --queue=mailing --tries=3 --timeout=600 --max-jobs=1000
```

### Stuck Jobs

```bash
# Clear stuck jobs
php artisan queue:flush

# Restart workers
sudo systemctl restart 'timebank-*'
```

## Configuration

Adjust these settings in `config/timebank_cc.php`:

```php
'mailing' => [
    'batch_size' => 10,              // Recipients per job
    'send_delay_seconds' => 5,       // Delay between emails
    'max_retries' => 3,              // Job retry attempts
    'retry_delay_minutes' => 15,     // Initial retry delay
],
```