timebank-cc-public/references/SSL_CERTIFICATE_RENEWAL.md

SSL Certificate Renewal & Expiry Monitoring

This document covers Let's Encrypt certificate renewal on the production server, the root cause of a past silent-expiry incident, the permanent fix, and the in-app expiry monitor that guards against it happening again.

Background: the 2026-06-27 silent expiry incident

The production timebank.cc certificate (covering timebank.cc and www.timebank.cc) expired on 2026-06-26 without renewing. The other certs (dev.timebank.cc, ws.timebank.cc) renewed fine.

Root cause: the timebank.cc certificate's renewal config used the standalone authenticator:

# /etc/letsencrypt/renewal/timebank.cc.conf
authenticator = standalone

The standalone authenticator starts its own temporary web server on port 80 to answer the ACME HTTP-01 challenge. But Apache permanently owns port 80, so every unattended renewal failed with:

Could not bind TCP port 80 because it is already in use by another process

Because the renewal ran via cron/systemd with output discarded (and the box's local Postfix does not relay mail externally — see "Mail delivery" below), the failure was completely silent until the certificate actually expired.

The permanent fix: switch to the apache authenticator

The apache authenticator answers the ACME challenge through the already-running Apache (it temporarily drops the challenge file into the webroot via the Apache module). No port conflict, no stopping Apache, zero downtime.

# Reissue the cert using the apache authenticator (one-time switch).
# --force-renewal is required if the cert was recently renewed, otherwise
# certbot reports "not yet due" and does NOT rewrite the authenticator.
sudo certbot certonly --apache \
  --cert-name timebank.cc \
  -d timebank.cc -d www.timebank.cc \
  --non-interactive --force-renewal

Verify:

# Renewal config should now read: authenticator = apache
grep authenticator /etc/letsencrypt/renewal/timebank.cc.conf

# Dry-run the unattended path — must succeed with NO port-80 error.
sudo certbot renew --cert-name timebank.cc --dry-run

The python3-certbot-apache plugin must be installed (certbot plugins lists * apache). It is installed on production.

If a cert ever silently expires again

1. Run sudo certbot certificates to see expiry dates and which cert is affected.
2. Check authenticator in /etc/letsencrypt/renewal/<name>.conf.
3. If it is standalone on this Apache box, switch it to apache as above.

Emergency manual renewal (if needed before the fix)

# Brief (~10s) Apache stop so standalone can bind port 80.
sudo certbot renew --cert-name timebank.cc \
  --pre-hook "systemctl stop apache2" \
  --post-hook "systemctl start apache2"

Renewal timing

• certbot renews at ~30 days before expiry (renew_before_expiry = 30 days).
• The systemd certbot.timer runs twice daily.
• There is also a separate cron in root's crontab as a second renewal attempt: 0 3 * * * /usr/bin/certbot renew --quiet && systemctl restart elasticsearch (redundant with the systemd timer; kept, harmless now that the authenticator is fixed).

So a healthy cert is renewed ~30 days out and never reaches the monitor's warning window. The monitor only fires if renewal has already been failing for ~10+ days.

Expiry monitor (in-app)

A Laravel artisan command provides an early-warning safety net, independent of certbot, so a renewal failure is caught well before the cert lapses.

• Command: php artisan cert:check (app/Console/Commands/CheckCertExpiry.php)
• Config: cert_monitor.* in config/timebank_cc.php, overridable via .env (CERT_MONITOR_* keys, documented in .env.example).
• Schedule: daily at 08:00 via the Laravel scheduler (app/Console/Kernel.php), running as www-data.

Behaviour

• Reads every cert under cert_monitor.live_path (/etc/letsencrypt/live).
• Two tiers:
  • NOTICE when a cert has fewer than notice_days (20) left.
  • URGENT when fewer than urgent_days (10) left.
• Sends a single summary email to cert_monitor.alert_email (admin@timebank.cc), synchronously via Mail::raw (never queued, so the alert does not depend on a running queue worker).
• Mirrors every warning to the Laravel log as a fallback.
• Returns a non-zero exit code on URGENT.
• Stays silent (and exits 0) when all certs are outside the warning window.

Testing overrides

# Always run from the app directory, as the web user:
cd /var/www/timebank_cc

# Dry run (reads certs, prints findings, sends no email):
sudo -u www-data php artisan cert:check --dry-run

# Force a NOTICE/URGENT to test email delivery (bypasses config thresholds):
sudo -u www-data php artisan cert:check --notice-days=999
sudo -u www-data php artisan cert:check --urgent-days=999

Config keys

Key (cert_monitor.*)	.env override	Default	Purpose
enabled	CERT_MONITOR_ENABLED	true	Enable/disable the monitor + schedule
alert_email	CERT_MONITOR_EMAIL	admin@timebank.cc	Recipient
notice_days	CERT_MONITOR_NOTICE_DAYS	20	First (NOTICE) threshold
urgent_days	CERT_MONITOR_URGENT_DAYS	10	Escalated (URGENT) threshold
live_path	CERT_MONITOR_LIVE_PATH	/etc/letsencrypt/live	Where to scan for certs
run_at	CERT_MONITOR_RUN_AT	08:00	Daily run time
healthcheck_url	CERT_MONITOR_HEALTHCHECK_URL	(unset)	healthchecks.io ping URL

Production note: the active config/timebank_cc.php on production has the non-healthcheck_url keys as hardcoded literals (added via an early config:merge), so those are NOT .env-overridable on prod. The values are correct, so this is cosmetic. config:merge only adds missing keys and will not rewrite existing ones — change these by editing the file directly if ever needed. Only healthcheck_url is wired to env().

Dead-man's-switch (healthchecks.io)

The email alert depends on the scheduler actually running. If the scheduler, cron, or the whole server dies, no email is sent and silence looks like "all healthy" — the exact silent-failure mode of the original incident.

A healthchecks.io dead-man's-switch inverts this: the scheduler pings a URL on each successful run; if the ping stops arriving, healthchecks.io (external to the server) notifies you.

• Wired in app/Console/Kernel.php via ->thenPing($url), attached only when cert_monitor.healthcheck_url is set.
• The URL is stored in production .env as CERT_MONITOR_HEALTHCHECK_URL.

Setup

1. Create a free check at https://healthchecks.io.
2. Set Period = 1 day, Grace = 1 day (matches the daily 08:00 run, tolerant of one late run, alerts within ~2 days if the scheduler truly dies).
3. Add a notification method (e.g. email) under the check's integrations — otherwise it tracks status but never notifies.
4. Put the ping URL in production .env:

   CERT_MONITOR_HEALTHCHECK_URL=https://hc-ping.com/<uuid>
   

5. Clear config and verify the app reads it:

   cd /var/www/timebank_cc
   sudo -u www-data php artisan config:clear
   sudo -u www-data php artisan tinker --execute "var_export(timebank_config('cert_monitor.healthcheck_url'));"
   

6. Test the ping manually:

   curl -fsS "https://hc-ping.com/<uuid>" && echo " pinged"
   

Certificate file access for www-data

The app runs as www-data, but /etc/letsencrypt/live and /etc/letsencrypt/archive are drwx------ root root, so www-data cannot traverse them to read certs. The individual cert files (cert.pem, fullchain.pem) are already world-readable; only directory traversal is blocked.

Fix — grant the ssl-cert group read/traverse on those two directories only (private keys remain 0600 root-only and are never exposed):

sudo usermod -aG ssl-cert www-data
sudo chgrp ssl-cert /etc/letsencrypt/live /etc/letsencrypt/archive
sudo chmod g+rx /etc/letsencrypt/live /etc/letsencrypt/archive

Survive renewals — certbot may recreate these dirs on renewal, so a deploy hook re-applies the permissions after each renewal:

# /etc/letsencrypt/renewal-hooks/deploy/50-group-read.sh  (chmod +x)
#!/bin/bash
chgrp ssl-cert /etc/letsencrypt/live /etc/letsencrypt/archive 2>/dev/null
chmod g+rx /etc/letsencrypt/live /etc/letsencrypt/archive 2>/dev/null

Verify www-data can read certs but not keys:

sudo -u www-data php artisan cert:check --dry-run    # lists certs
sudo -u www-data cat /etc/letsencrypt/live/timebank.cc/privkey.pem  # should be denied

Scheduler & user notes

• The Laravel scheduler must run as www-data (the web user), not root. Production's www-data crontab:

  * * * * * cd /var/www/timebank_cc && php artisan schedule:run >> /dev/null 2>&1
  

  Running it as root leaves root-owned files in storage/ that later break the web user. (This was the case from ~2026-06-15 to 2026-06-27 and has been fixed; root-owned storage/logs/*.log were chowned back to www-data.)
• Always run manual php artisan commands from /var/www/timebank_cc (not ~), or artisan won't be found.

Mail delivery caveat

The production box's local Postfix delivers mail locally only — it does not relay outbound. mail/sendmail-based alerts therefore fail silently (this is why the original cron failures were never seen; they piled up in /var/mail/root).

The expiry monitor deliberately sends through the Laravel app's configured SMTP (Greenhost: smtp.greenhost.nl:465), which is a known-good external delivery path with correct SPF/DKIM, independent of local Postfix.

Quick reference

# Status of all certs
sudo certbot certificates

# Verify unattended renewal works (no port-80 error)
sudo certbot renew --cert-name timebank.cc --dry-run

# Run the monitor manually
cd /var/www/timebank_cc && sudo -u www-data php artisan cert:check --dry-run

# Force an alert email (test)
cd /var/www/timebank_cc && sudo -u www-data php artisan cert:check --notice-days=999

# Confirm the monitor is scheduled
cd /var/www/timebank_cc && sudo -u www-data php artisan schedule:list | grep -i cert