Aggiunge monitoraggio ops: health check, log scanner, dashboard e notifiche.

Introduce incidenti con dedup, job Sidekiq ogni 3 min, scan log cron, /up/deep,
dashboard /admin/ops e push ntfy; include Sentry opzionale e fix test suite.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
2026-06-12 08:16:47 +02:00
parent ce8939fffb
commit 52cfffb83f
44 changed files with 1396 additions and 24 deletions

95
docs/OPS_MONITORING.md Normal file
View File

@@ -0,0 +1,95 @@
# Monitoraggio ops — Match Live TV
Sistema integrato per health check continui, analisi log, dashboard incidenti e notifiche push.
## Componenti
| Componente | Ruolo |
|------------|-------|
| `Ops::HealthMonitorJob` | Sidekiq — health check ogni ~3 min |
| `Ops::HealthChecks` | Disco, DB, Redis, MediaMTX, Garage, Sidekiq, HTTP pubblico |
| `Ops::LogScanner` | Pattern regex su log Docker (cron ogni 10 min) |
| `Ops::Incident` | DB — incidenti con deduplicazione |
| `Ops::Notifier` | Push ntfy + email opzionale |
| `/admin/ops` | Dashboard incidenti |
| `GET /up/deep` | Health check esteso (JSON) |
## Setup notifiche ntfy (telefono)
1. Installa l'app [ntfy](https://ntfy.sh) su Android/iOS
2. Scegli un topic segreto (es. `matchlivetv-ops-eminux-a8f3b2`)
3. Iscriviti al topic nell'app
4. Sul server, in `infra/.env`:
```bash
OPS_NTFY_URL=https://ntfy.sh/matchlivetv-ops-eminux-a8f3b2
OPS_NOTIFY_SEVERITIES=critical
OPS_NOTIFY_COOLDOWN_MINUTES=30
```
5. Test:
```bash
docker compose -f docker-compose.prod.yml exec -T rails bundle exec rails ops:test_notify
```
## Variabili ambiente
Vedi [`infra/.env.production.example`](../infra/.env.production.example).
| Variabile | Default | Descrizione |
|-----------|---------|-------------|
| `OPS_NTFY_URL` | — | URL POST ntfy |
| `OPS_NTFY_TOKEN` | — | Bearer token (topic protetto) |
| `OPS_ALERT_EMAIL` | — | Fallback email |
| `OPS_HEALTH_INTERVAL_SECS` | 180 | Intervallo health job |
| `OPS_HEALTH_TOKEN` | — | Token per `/up/deep` |
| `OPS_DISK_CRIT_PERCENT` | 90 | Soglia critical disco |
| `OPS_LOG_SUBSCRIBER` | false | Cattura 500 in-process |
## Cron produzione
```bash
cd /opt/matchlivetv/infra && bash scripts/install_production_cron.sh
```
| Schedule | Task |
|----------|------|
| ogni 3 min (Sidekiq) | `Ops::HealthMonitorJob` |
| `*/10 * * * *` | `scan_ops_logs.sh``ops:ingest_logs` |
## Uptime Kuma (consigliato)
Watchdog esterno se Rails è completamente giù:
- HTTP: `https://www.matchlivetv.it/up`
- HTTP deep: `https://www.matchlivetv.it/up/deep` (header `X-Ops-Token`)
- TCP: porta 1935 (RTMP)
- Notifica: stesso topic ntfy
## Workflow rilascio
1. `cd backend && bundle exec rspec`
2. Ciclo patch/test fino a 0 failure
3. `git commit``git push`
4. `bash scripts/deploy/sync_to_server.sh`
5. Rebuild: `docker compose -f docker-compose.prod.yml up -d --build rails sidekiq`
6. Migrate: `docker compose exec -T rails bundle exec rails db:migrate`
7. Cron: `bash scripts/install_production_cron.sh`
8. Smoke test:
- `curl -sf https://www.matchlivetv.it/up`
- `bundle exec rails ops:health_check` (nel container)
- Apri `/admin/ops`
## Sentry (opzionale)
Imposta `SENTRY_DSN` dopo `bundle install`. Gli errori creano anche incidenti ops.
## Troubleshooting
| Problema | Azione |
|----------|--------|
| Nessuna push | Verifica `OPS_NTFY_URL`, test con `ops:test_notify` |
| Troppi alert | Aumenta `OPS_NOTIFY_COOLDOWN_MINUTES`, usa Mute 24h in admin |
| Sidekiq stale | Verifica container `sidekiq` e log Sidekiq |
| Log scanner vuoto | Controlla `/opt/matchlivetv/log/cron-ops.log` |