timebank-cc-public/scripts/BACKUP_GUIDE.md

# Laravel Timebank Backup & Restore Guide

This guide provides comprehensive instructions for backing up and restoring your Laravel Timebank application data, including database and storage files.

## Table of Contents

1. [Overview](#overview)
2. [Quick Start](#quick-start)
3. [Installation & Setup](#installation--setup)
4. [Backup Scripts](#backup-scripts)
5. [Restore Scripts](#restore-scripts)
6. [Automation Setup](#automation-setup)
7. [Monitoring & Maintenance](#monitoring--maintenance)
8. [Troubleshooting](#troubleshooting)
9. [Best Practices](#best-practices)

## Overview

The backup system consists of several components:

- **Database Backup**: Compressed MySQL dumps with rotation
- **Storage Backup**: Incremental rsync-based file backups
- **Automated Scheduling**: Cron-based periodic backups
- **Retention Management**: Automatic cleanup of old backups
- **Verification**: Backup integrity checks
- **Email Notifications**: Success/failure notifications for all operations
- **Restore Tools**: Simple restoration procedures

### Backup Types

- **Daily**: 7 days retention, incremental storage backups
- **Weekly**: 4 weeks retention, full storage backups
- **Monthly**: 12 months retention, complete archives
- **Full**: On-demand complete backups

## Quick Start

### Perform a Manual Backup

```bash
# Complete backup (database + storage)
./scripts/backup-all.sh daily

# Database only
./scripts/backup-database.sh daily

# Storage only
./scripts/backup-storage.sh daily

# With verification and notifications
./scripts/backup-all.sh daily --verify --notify
```

### List Available Backups

```bash
# List database backups
./scripts/restore-database.sh --list-backups

# List storage backups
./scripts/restore-storage.sh --list-backups
```

### Restore from Latest Backup

```bash
# Restore database from latest backup
./scripts/restore-database.sh --latest

# Restore storage from latest backup
./scripts/restore-storage.sh --latest

# Merge storage (don't replace existing files)
./scripts/restore-storage.sh --latest --merge
```

### Configure Retention (Optional)

Customize how long backups are kept:

```bash
# View current settings
./scripts/cleanup-backups.sh --help

# Edit retention policy  
nano scripts/backup-retention.conf
```

**Quick settings:** Change DAILY_RETENTION, MONTHLY_RETENTION, or disk thresholds to suit your storage needs.

## Installation & Setup

### 1. Verify Prerequisites

Ensure the following are installed on your system:

```bash
# Required tools
which mysqldump mysql rsync tar gzip
```

### 2. Configure Environment

Your `.env` file must contain proper database credentials:

```env
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=timebank_cc
DB_USERNAME=your_username
DB_PASSWORD=your_password
```

### 3. Configure Retention Policy (Optional)

The backup system uses default retention settings, but you can customize them:

```bash
# View current retention settings
./scripts/cleanup-backups.sh --help

# Edit retention configuration (optional)
nano scripts/backup-retention.conf

# Common adjustments:
# - DAILY_RETENTION=14     # Keep daily backups for 2 weeks instead of 1
# - MONTHLY_RETENTION=730  # Keep monthly backups for 2 years instead of 1
# - DISK_WARNING_THRESHOLD=75  # Earlier disk space warnings
```

**Quick retention examples:**
- **More storage space**: Increase DAILY_RETENTION=14, MONTHLY_RETENTION=730
- **Less storage space**: Decrease DAILY_COUNT_LIMIT=3, MONTHLY_COUNT_LIMIT=6
- **Earlier disk warnings**: Set DISK_WARNING_THRESHOLD=75

### 4. Set Permissions

Ensure backup scripts are executable:

```bash
chmod +x scripts/*.sh
```

### 4. Test Backup

Run a test backup to verify everything works:

```bash
./scripts/backup-all.sh daily --verify
```

### 5. Check Backup Directory

Verify backups are created:

```bash
ls -la backups/
```

## Backup Scripts

### backup-database.sh

Creates compressed MySQL database backups.

**Usage:**
```bash
./scripts/backup-database.sh [daily|weekly|monthly]
```

**Features:**
- Compressed gzip output
- Single-transaction consistency
- Includes routines, triggers, events
- Automatic retention cleanup
- Backup verification
- Email notifications on success

**Example:**
```bash
# Daily database backup
./scripts/backup-database.sh daily

# Monthly database backup
./scripts/backup-database.sh monthly
```

### backup-storage.sh

Creates compressed archives of the storage directory.

**Usage:**
```bash
./scripts/backup-storage.sh [daily|weekly|monthly|full]
```

**Features:**
- Incremental backups using rsync
- Hard links for space efficiency
- Excludes cache and temporary files
- Full backup option available
- Compressed tar.gz output
- Email notifications on success

**Example:**
```bash
# Incremental storage backup
./scripts/backup-storage.sh daily

# Full storage backup
./scripts/backup-storage.sh full
```

### backup-all.sh

Master orchestration script for complete backups.

**Usage:**
```bash
./scripts/backup-all.sh [daily|weekly|monthly] [options]
```

**Options:**
- `--storage-only`: Backup only storage files
- `--database-only`: Backup only database
- `--verify`: Verify backups after creation
- `--notify`: Send notifications on completion

**Example:**
```bash
# Complete backup with verification
./scripts/backup-all.sh weekly --verify --notify

# Storage only backup
./scripts/backup-all.sh daily --storage-only
```

## Restore Scripts

### restore-database.sh

Restores database from backup files.

**Usage:**
```bash
./scripts/restore-database.sh [backup_file] [options]
```

**Options:**
- `--confirm`: Skip confirmation prompt
- `--list-backups`: List available backups
- `--latest`: Restore from latest backup

**Examples:**
```bash
# List available backups
./scripts/restore-database.sh --list-backups

# Restore latest backup
./scripts/restore-database.sh --latest

# Restore specific backup (full path)
./scripts/restore-database.sh backups/database/daily/timebank_daily_20240101.sql.gz

# Restore specific backup (filename only - auto-resolves path)
./scripts/restore-database.sh timebank_daily_20240101.sql.gz
```

**Safety Features:**
- Creates pre-restore backup
- Confirmation prompts
- Backup verification
- Detailed logging
- Email notifications on success
- Automatic path resolution

### restore-storage.sh

Restores storage files from backup archives.

**Usage:**
```bash
./scripts/restore-storage.sh [backup_file] [options]
```

**Options:**
- `--confirm`: Skip confirmation prompt
- `--list-backups`: List available backups
- `--latest`: Restore from latest backup
- `--merge`: Merge with existing files (don't replace)

**Examples:**
```bash
# Restore latest storage backup
./scripts/restore-storage.sh --latest

# Merge latest backup with existing files
./scripts/restore-storage.sh --latest --merge

# Restore specific backup (full path)
./scripts/restore-storage.sh backups/storage/weekly/weekly_20240101.tar.gz

# Restore specific backup (filename only - auto-resolves path)
./scripts/restore-storage.sh weekly_20240101.tar.gz
```

### restore-all.sh

Complete system restoration script for both database and storage.

**Usage:**
```bash
./scripts/restore-all.sh [options]
```

**Options:**
- `--latest`: Restore from latest backups (both database and storage)
- `--database-file FILE`: Specify database backup file
- `--storage-file FILE`: Specify storage backup file
- `--database-latest`: Use latest database backup only
- `--storage-latest`: Use latest storage backup only
- `--confirm`: Skip confirmation prompts
- `--list-backups`: List available backups

**Examples:**
```bash
# Restore both database and storage from latest backups
./scripts/restore-all.sh --latest

# Restore specific files
./scripts/restore-all.sh --database-file timebank_daily_20240101.sql.gz --storage-file daily_20240101.tar.gz

# Database only restore
./scripts/restore-all.sh --database-latest

# List all available backups
./scripts/restore-all.sh --list-backups
```

**Features:**
- Orchestrates complete system restoration
- Handles both database and storage restoration
- Automatic path resolution for backup files
- Pre-restore backups for safety
- Email notifications on success
- Post-restore Laravel optimization

## Automation Setup

### Cron Configuration

Copy and customize the cron configuration:

```bash
# Copy template
sudo cp scripts/cron-backup.conf /etc/cron.d/timebank-backup

# Edit paths and email addresses
sudo nano /etc/cron.d/timebank-backup

# Verify cron syntax
sudo cron -T
```

### Alternative: User Crontab

For non-root installations:

```bash
# Edit user crontab
crontab -e

# Add backup schedules (example)
0 2 * * * cd /path/to/timebank && ./scripts/backup-all.sh daily --verify
0 3 * * 0 cd /path/to/timebank && ./scripts/backup-all.sh weekly --verify
0 4 1 * * cd /path/to/timebank && ./scripts/backup-all.sh monthly --verify
```

### Email Notification Setup

All backup and restore scripts automatically send email notifications on successful completion.

#### Prerequisites

Install mail utilities:
```bash
# Ubuntu/Debian
sudo apt install mailutils postfix

# Configure postfix for "Local only" delivery during setup
```

#### Email Configuration

By default, notifications are sent to `$USER@localhost`. To customize:

```bash
# Set custom email address (optional)
export BACKUP_NOTIFY_EMAIL="admin@yourdomain.org"
```

#### Email Subjects and Content

**Backup Notifications:**
- Subject: "Timebank DB Backup Success"
- Subject: "Timebank Storage Backup Success"
- Content: Includes timestamp and backup details

**Restore Notifications:**
- Subject: "Timebank DB Restore Success"
- Subject: "Timebank Storage Restore Success"
- Subject: "Timebank Complete Restore Success"
- Content: Includes timestamp and restored file information

#### Checking Local Mail

View received notifications:
```bash
# Open mail client
mail

# List messages
mail -H

# Quick check for new messages
ls -la /var/mail/$USER
```

#### Webhook Notifications (Optional)

For additional notification methods:
```bash
# Webhook notifications
export BACKUP_WEBHOOK_URL="https://hooks.slack.com/your-webhook"
```

## Monitoring & Maintenance

### cleanup-backups.sh

Manages backup retention and disk space.

**Usage:**
```bash
./scripts/cleanup-backups.sh [options]
```

**Options:**
- `--dry-run`: Show what would be deleted
- `--force`: Force cleanup regardless of disk space
- `--verbose`: Show detailed information

**Features:**
- Automatic retention policy enforcement
- Disk space monitoring
- Empty directory cleanup
- Detailed reporting

**Example:**
```bash
# Dry run to see what would be cleaned up
./scripts/cleanup-backups.sh --dry-run --verbose

# Force cleanup
./scripts/cleanup-backups.sh --force
```

### Retention Policy Configuration

The cleanup script uses a configurable retention policy system that allows you to customize backup retention settings.

#### Configuration File

**Location**: `scripts/backup-retention.conf`

The configuration file controls:
- **Time-based retention**: How long to keep backups (in days)
- **Count-based retention**: How many recent backups to keep
- **Disk space thresholds**: When to trigger cleanup
- **Email notifications**: Enable/disable cleanup emails
- **Advanced settings**: Cleanup modes and verification options

#### Configuration Options

**Time-based Retention (days):**
```bash
DAILY_RETENTION=7          # Keep daily backups for 7 days
WEEKLY_RETENTION=28        # Keep weekly backups for 28 days  
MONTHLY_RETENTION=365      # Keep monthly backups for 365 days
PRE_RESTORE_RETENTION=30   # Keep pre-restore backups for 30 days
LOG_RETENTION=30           # Keep log files for 30 days
```

**Count-based Retention (number of files):**
```bash
DAILY_COUNT_LIMIT=7        # Keep 7 most recent daily backups
WEEKLY_COUNT_LIMIT=4       # Keep 4 most recent weekly backups
MONTHLY_COUNT_LIMIT=12     # Keep 12 most recent monthly backups
PRE_RESTORE_COUNT_LIMIT=5  # Keep 5 most recent pre-restore backups
SNAPSHOT_COUNT_LIMIT=3     # Keep 3 most recent storage snapshots
```

**Disk Space Management:**
```bash
DISK_WARNING_THRESHOLD=85  # Send warning at 85% disk usage
DISK_CRITICAL_THRESHOLD=95 # Force cleanup at 95% disk usage
```

**Notification Settings:**
```bash
EMAIL_NOTIFICATIONS_ENABLED=true  # Enable/disable email notifications
BACKUP_NOTIFY_EMAIL=admin@domain.org  # Custom email address (optional)
```

#### Customizing Retention Policies

**View Current Configuration:**
```bash
./scripts/cleanup-backups.sh --help
```

**Edit Configuration:**
```bash
# Edit the config file
nano scripts/backup-retention.conf

# Test changes with dry run
./scripts/cleanup-backups.sh --dry-run --verbose
```

#### Configuration Examples

**Conservative (longer retention):**
```bash
DAILY_RETENTION=14         # 2 weeks
WEEKLY_RETENTION=56        # 8 weeks  
MONTHLY_RETENTION=730      # 2 years
DAILY_COUNT_LIMIT=14       # More daily backups
MONTHLY_COUNT_LIMIT=24     # 2 years of monthly backups
```

**Aggressive (shorter retention):**
```bash
DAILY_RETENTION=3          # 3 days only
WEEKLY_RETENTION=14        # 2 weeks
MONTHLY_RETENTION=180      # 6 months
DAILY_COUNT_LIMIT=3        # Fewer daily backups
DISK_WARNING_THRESHOLD=75  # Earlier warning
```

**Space-constrained environment:**
```bash
DAILY_COUNT_LIMIT=3        # Keep only 3 daily backups
WEEKLY_COUNT_LIMIT=2       # Keep only 2 weekly backups
MONTHLY_COUNT_LIMIT=6      # Keep only 6 monthly backups
DISK_WARNING_THRESHOLD=70  # Early disk space warnings
DISK_CRITICAL_THRESHOLD=85 # More aggressive cleanup threshold
```

#### Validation and Safety

The configuration system includes:
- **Input validation**: Invalid values fall back to defaults
- **Range checking**: Values must be within reasonable ranges
- **Relationship validation**: Critical threshold must be higher than warning
- **Fallback system**: Uses defaults if config file is missing or corrupted

#### Advanced Settings

```bash
# Cleanup behavior
CLEANUP_MODE=both              # Options: age_only, count_only, both
CLEANUP_EMPTY_DIRS=true        # Remove empty directories after cleanup
VERIFY_BEFORE_DELETE=false     # Verify backup integrity before deletion

# Email control
EMAIL_NOTIFICATIONS_ENABLED=true  # Master switch for email notifications
```

### Health Monitoring

The backup system includes health checks:

```bash
# View health check results
cat backups/health_check.json

# Check backup logs
tail -f backups/backup.log

# View backup summary
./scripts/backup-all.sh daily --verify
```

### Log Management

Backup logs are stored in:
- `backups/backup.log` - Main backup log
- `backups/restore.log` - Restore operations log
- `/var/log/timebank-backup.log` - System cron log

Rotate logs using logrotate:

```bash
# Create logrotate configuration
sudo tee /etc/logrotate.d/timebank-backup <<EOF
/var/log/timebank-backup.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 644 root root
}
EOF
```

## Troubleshooting

### Common Issues

#### Permission Denied
```bash
# Fix script permissions
chmod +x scripts/*.sh

# Fix backup directory permissions
sudo chown -R $(whoami):$(whoami) backups/
```

#### MySQL Connection Issues
```bash
# Test database connection
mysql -h$DB_HOST -P$DB_PORT -u$DB_USERNAME -p$DB_PASSWORD -e "SHOW DATABASES;"

# Check .env file
grep "^DB_" .env
```

#### Disk Space Issues
```bash
# Check available space
df -h backups/

# Run cleanup
./scripts/cleanup-backups.sh --force

# Check large files
find backups/ -type f -size +100M -exec ls -lh {} \;
```

#### Backup Verification Failures
```bash
# Test gzip integrity
gzip -t backups/database/daily/*.sql.gz

# Test tar integrity
tar -tzf backups/storage/daily/*.tar.gz >/dev/null
```

### Recovery Procedures

#### Complete System Recovery

1. **Prepare clean environment:**
   ```bash
   # Install Laravel dependencies
   composer install --no-dev --optimize-autoloader
   npm install && npm run build
   ```

2. **Restore database:**
   ```bash
   ./scripts/restore-database.sh --latest --confirm
   ```

3. **Restore storage:**
   ```bash
   ./scripts/restore-storage.sh --latest --confirm
   ```

4. **Post-restore steps:**
   ```bash
   php artisan storage:link
   php artisan config:clear
   php artisan cache:clear
   php artisan migrate:status
   ```

#### Partial Recovery

**Database only:**
```bash
./scripts/restore-database.sh --latest
php artisan migrate:status
```

**Storage only:**
```bash
./scripts/restore-storage.sh --latest --merge
php artisan storage:link
```

## Best Practices

### Security

1. **Protect backup files:**
   ```bash
   chmod 600 backups/database/**/*.sql.gz
   chmod 600 backups/storage/**/*.tar.gz
   ```

2. **Use secure file transfer:**
   ```bash
   # Example: Upload to secure cloud storage
   rclone sync backups/ remote:timebank-backups/
   ```

3. **Encrypt sensitive backups:**
   ```bash
   # Example: GPG encryption
   gpg --symmetric --cipher-algo AES256 backup.sql.gz
   ```

### Performance

1. **Schedule during low-traffic periods**
2. **Monitor backup duration and adjust frequency**
3. **Use fast storage for backup directory**
4. **Consider incremental-only backups for large storage**

### Reliability

1. **Test restores regularly:**
   ```bash
   # Monthly restore test
   ./scripts/restore-database.sh --latest --confirm
   ```

2. **Monitor backup success:**
   ```bash
   # Check recent backup status
   find backups/ -name "*.gz" -mtime -1 -ls
   ```

3. **Verify backup integrity:**
   ```bash
   # Run verification on all recent backups
   ./scripts/backup-all.sh daily --verify
   ```

4. **Keep multiple backup locations:**
   - Local backups for quick recovery
   - Remote backups for disaster recovery
   - Cloud storage for long-term retention

### Documentation

1. **Keep this guide updated**
2. **Document any customizations**
3. **Maintain recovery contact information**
4. **Document environment-specific configurations**

---

## Support

For issues or questions regarding the backup system:

1. Check the troubleshooting section above
2. Review backup logs: `tail -f backups/backup.log`
3. Test with `--dry-run` options first
4. Ensure all prerequisites are installed

**Remember**: Always test your restore procedures in a non-production environment before relying on them in production!