timebank-cc-public/references/SSL_CERTIFICATE_RENEWAL.md

# SSL Certificate Renewal & Expiry Monitoring

This document covers Let's Encrypt certificate renewal on the production server,
the root cause of a past silent-expiry incident, the permanent fix, and the
in-app expiry monitor that guards against it happening again.

## Background: the 2026-06-27 silent expiry incident

The production `timebank.cc` certificate (covering `timebank.cc` and
`www.timebank.cc`) expired on 2026-06-26 without renewing. The other certs
(`dev.timebank.cc`, `ws.timebank.cc`) renewed fine.

**Root cause:** the `timebank.cc` certificate's renewal config used the
`standalone` authenticator:

```
# /etc/letsencrypt/renewal/timebank.cc.conf
authenticator = standalone
```

The `standalone` authenticator starts its own temporary web server on port 80
to answer the ACME HTTP-01 challenge. But Apache permanently owns port 80, so
every unattended renewal failed with:

```
Could not bind TCP port 80 because it is already in use by another process
```

Because the renewal ran via cron/systemd with output discarded (and the box's
local Postfix does not relay mail externally — see "Mail delivery" below), the
failure was completely silent until the certificate actually expired.

## The permanent fix: switch to the `apache` authenticator

The `apache` authenticator answers the ACME challenge *through the already-running
Apache* (it temporarily drops the challenge file into the webroot via the Apache
module). No port conflict, no stopping Apache, zero downtime.

```bash
# Reissue the cert using the apache authenticator (one-time switch).
# --force-renewal is required if the cert was recently renewed, otherwise
# certbot reports "not yet due" and does NOT rewrite the authenticator.
sudo certbot certonly --apache \
  --cert-name timebank.cc \
  -d timebank.cc -d www.timebank.cc \
  --non-interactive --force-renewal
```

Verify:

```bash
# Renewal config should now read: authenticator = apache
grep authenticator /etc/letsencrypt/renewal/timebank.cc.conf

# Dry-run the unattended path — must succeed with NO port-80 error.
sudo certbot renew --cert-name timebank.cc --dry-run
```

The `python3-certbot-apache` plugin must be installed (`certbot plugins` lists
`* apache`). It is installed on production.

### If a cert ever silently expires again

1. Run `sudo certbot certificates` to see expiry dates and which cert is affected.
2. Check `authenticator` in `/etc/letsencrypt/renewal/<name>.conf`.
3. If it is `standalone` on this Apache box, switch it to `apache` as above.

### Emergency manual renewal (if needed before the fix)

```bash
# Brief (~10s) Apache stop so standalone can bind port 80.
sudo certbot renew --cert-name timebank.cc \
  --pre-hook "systemctl stop apache2" \
  --post-hook "systemctl start apache2"
```

## Renewal timing

- `certbot` renews at **~30 days before expiry** (`renew_before_expiry = 30 days`).
- The systemd `certbot.timer` runs **twice daily**.
- There is also a separate cron in root's crontab as a second renewal attempt:
  `0 3 * * * /usr/bin/certbot renew --quiet && systemctl restart elasticsearch`
  (redundant with the systemd timer; kept, harmless now that the authenticator
  is fixed).

So a healthy cert is renewed ~30 days out and never reaches the monitor's warning
window. The monitor only fires if renewal has already been failing for ~10+ days.

## Expiry monitor (in-app)

A Laravel artisan command provides an early-warning safety net, independent of
certbot, so a renewal failure is caught well before the cert lapses.

- **Command:** `php artisan cert:check`
  (`app/Console/Commands/CheckCertExpiry.php`)
- **Config:** `cert_monitor.*` in `config/timebank_cc.php`, overridable via `.env`
  (`CERT_MONITOR_*` keys, documented in `.env.example`).
- **Schedule:** daily at 08:00 via the Laravel scheduler
  (`app/Console/Kernel.php`), running as `www-data`.

### Behaviour

- Reads every cert under `cert_monitor.live_path` (`/etc/letsencrypt/live`).
- Two tiers:
  - **NOTICE** when a cert has fewer than `notice_days` (20) left.
  - **URGENT** when fewer than `urgent_days` (10) left.
- Sends a single summary email to `cert_monitor.alert_email`
  (`admin@timebank.cc`), **synchronously via `Mail::raw`** (never queued, so the
  alert does not depend on a running queue worker).
- Mirrors every warning to the Laravel log as a fallback.
- Returns a non-zero exit code on URGENT.
- Stays silent (and exits 0) when all certs are outside the warning window.

### Testing overrides

```bash
# Always run from the app directory, as the web user:
cd /var/www/timebank_cc

# Dry run (reads certs, prints findings, sends no email):
sudo -u www-data php artisan cert:check --dry-run

# Force a NOTICE/URGENT to test email delivery (bypasses config thresholds):
sudo -u www-data php artisan cert:check --notice-days=999
sudo -u www-data php artisan cert:check --urgent-days=999
```

### Config keys

| Key (`cert_monitor.*`) | `.env` override | Default | Purpose |
|---|---|---|---|
| `enabled` | `CERT_MONITOR_ENABLED` | `true` | Enable/disable the monitor + schedule |
| `alert_email` | `CERT_MONITOR_EMAIL` | `admin@timebank.cc` | Recipient |
| `notice_days` | `CERT_MONITOR_NOTICE_DAYS` | `20` | First (NOTICE) threshold |
| `urgent_days` | `CERT_MONITOR_URGENT_DAYS` | `10` | Escalated (URGENT) threshold |
| `live_path` | `CERT_MONITOR_LIVE_PATH` | `/etc/letsencrypt/live` | Where to scan for certs |
| `run_at` | `CERT_MONITOR_RUN_AT` | `08:00` | Daily run time |
| `healthcheck_url` | `CERT_MONITOR_HEALTHCHECK_URL` | (unset) | healthchecks.io ping URL |

> **Production note:** the active `config/timebank_cc.php` on production has the
> non-`healthcheck_url` keys as **hardcoded literals** (added via an early
> `config:merge`), so those are NOT `.env`-overridable on prod. The values are
> correct, so this is cosmetic. `config:merge` only adds missing keys and will
> not rewrite existing ones — change these by editing the file directly if ever
> needed. Only `healthcheck_url` is wired to `env()`.

## Dead-man's-switch (healthchecks.io)

The email alert depends on the scheduler actually running. If the scheduler,
cron, or the whole server dies, no email is sent and silence looks like "all
healthy" — the exact silent-failure mode of the original incident.

A healthchecks.io dead-man's-switch inverts this: the scheduler pings a URL on
each successful run; if the ping stops arriving, healthchecks.io (external to the
server) notifies you.

- Wired in `app/Console/Kernel.php` via `->thenPing($url)`, attached only when
  `cert_monitor.healthcheck_url` is set.
- The URL is stored in production `.env` as `CERT_MONITOR_HEALTHCHECK_URL`.

### Setup

1. Create a free check at https://healthchecks.io.
2. Set **Period = 1 day**, **Grace = 1 day** (matches the daily 08:00 run, tolerant
   of one late run, alerts within ~2 days if the scheduler truly dies).
3. Add a notification method (e.g. email) under the check's integrations —
   otherwise it tracks status but never notifies.
4. Put the ping URL in production `.env`:
   ```
   CERT_MONITOR_HEALTHCHECK_URL=https://hc-ping.com/<uuid>
   ```
5. Clear config and verify the app reads it:
   ```bash
   cd /var/www/timebank_cc
   sudo -u www-data php artisan config:clear
   sudo -u www-data php artisan tinker --execute "var_export(timebank_config('cert_monitor.healthcheck_url'));"
   ```
6. Test the ping manually:
   ```bash
   curl -fsS "https://hc-ping.com/<uuid>" && echo " pinged"
   ```

## Certificate file access for `www-data`

The app runs as `www-data`, but `/etc/letsencrypt/live` and
`/etc/letsencrypt/archive` are `drwx------ root root`, so `www-data` cannot
traverse them to read certs. The individual cert files (`cert.pem`,
`fullchain.pem`) are already world-readable; only directory traversal is blocked.

**Fix** — grant the `ssl-cert` group read/traverse on those two directories only
(private keys remain `0600` root-only and are never exposed):

```bash
sudo usermod -aG ssl-cert www-data
sudo chgrp ssl-cert /etc/letsencrypt/live /etc/letsencrypt/archive
sudo chmod g+rx /etc/letsencrypt/live /etc/letsencrypt/archive
```

**Survive renewals** — certbot may recreate these dirs on renewal, so a deploy
hook re-applies the permissions after each renewal:

```bash
# /etc/letsencrypt/renewal-hooks/deploy/50-group-read.sh  (chmod +x)
#!/bin/bash
chgrp ssl-cert /etc/letsencrypt/live /etc/letsencrypt/archive 2>/dev/null
chmod g+rx /etc/letsencrypt/live /etc/letsencrypt/archive 2>/dev/null
```

Verify www-data can read certs but not keys:

```bash
sudo -u www-data php artisan cert:check --dry-run    # lists certs
sudo -u www-data cat /etc/letsencrypt/live/timebank.cc/privkey.pem  # should be denied
```

## Scheduler & user notes

- The Laravel scheduler must run as **`www-data`** (the web user), not root.
  Production's `www-data` crontab:
  ```
  * * * * * cd /var/www/timebank_cc && php artisan schedule:run >> /dev/null 2>&1
  ```
  Running it as root leaves root-owned files in `storage/` that later break the
  web user. (This was the case from ~2026-06-15 to 2026-06-27 and has been fixed;
  root-owned `storage/logs/*.log` were chowned back to www-data.)
- Always run manual `php artisan` commands from `/var/www/timebank_cc` (not `~`),
  or `artisan` won't be found.

## Mail delivery caveat

The production box's local Postfix delivers mail **locally only** — it does not
relay outbound. `mail`/sendmail-based alerts therefore fail silently (this is why
the original cron failures were never seen; they piled up in `/var/mail/root`).

The expiry monitor deliberately sends through the **Laravel app's configured SMTP**
(Greenhost: `smtp.greenhost.nl:465`), which is a known-good external delivery
path with correct SPF/DKIM, independent of local Postfix.

## Quick reference

```bash
# Status of all certs
sudo certbot certificates

# Verify unattended renewal works (no port-80 error)
sudo certbot renew --cert-name timebank.cc --dry-run

# Run the monitor manually
cd /var/www/timebank_cc && sudo -u www-data php artisan cert:check --dry-run

# Force an alert email (test)
cd /var/www/timebank_cc && sudo -u www-data php artisan cert:check --notice-days=999

# Confirm the monitor is scheduled
cd /var/www/timebank_cc && sudo -u www-data php artisan schedule:list | grep -i cert
```