Monitoring Metrics
The service exposes two monitoring surfaces:
| Surface | Command or endpoint | Use |
|---|---|---|
| Health | GET /health | Load balancer and uptime checks |
| Metrics | GET /metrics | Prometheus-compatible scrape endpoint |
| Live dashboard | just status live | Human operations dashboard |
Metrics Endpoint
GET /metrics returns Prometheus text format and does not require an API key. HTTP path labels use FastAPI route templates, with unmatched paths collapsed to __unmatched__ to avoid unbounded label cardinality. Celery workers write task duration and failure samples to Redis so the API metrics scrape can expose worker observations. It exports:
| Metric | Purpose |
|---|---|
infoconnect_http_requests_total | Request count by method, route-template path, and status |
infoconnect_http_errors_total | 4xx/5xx error count by method, route-template path, and status |
infoconnect_http_request_duration_seconds | API latency histogram for p95/p99 queries |
infoconnect_search_latency_seconds | Search latency histogram with rerank label |
infoconnect_task_duration_seconds | Celery task duration histogram |
infoconnect_indexing_failures_total | Indexing failure count by task |
infoconnect_queue_depth | Redis queue depth for default, preprocessing, and embedding queues |
infoconnect_component_health | API, Redis, and Qdrant health gauges |
infoconnect_resource_usage_percent | Host CPU, memory, and disk percent |
Dashboard Queries
Use these PromQL expressions in Grafana or another Prometheus dashboard:
sum(rate(infoconnect_http_requests_total[5m]))
sum(rate(infoconnect_http_errors_total[5m])) / sum(rate(infoconnect_http_requests_total[5m]))
histogram_quantile(0.95, sum(rate(infoconnect_http_request_duration_seconds_bucket[5m])) by (le, path))
histogram_quantile(0.99, sum(rate(infoconnect_http_request_duration_seconds_bucket[5m])) by (le, path))
histogram_quantile(0.95, sum(rate(infoconnect_search_latency_seconds_bucket[5m])) by (le, reranked))
sum(infoconnect_queue_depth) by (queue)
min(infoconnect_component_health) by (component)
max(infoconnect_resource_usage_percent) by (resource)
Operations Workflow
Use /metrics for automated scraping and just status live for hands-on triage. During UAT, capture both the Prometheus dashboard screenshot or query output and a just status docker --iterations 1 snapshot as go-live evidence.
Alert Rules
Load the go-live Prometheus rules from monitoring/prometheus-alerts.yml. These rules assume the API scrape job is named infoconnect-api; worker-down detection uses the task metrics and queue-depth gauges exported by the API /metrics endpoint rather than per-worker scrape targets. Keep the same labels in UAT and production so alert routing can separate environments by the Prometheus environment external label.
InfoConnectWorkerDown assumes the preprocessing and embedding queues are operated as distinct worker pools. In a single-pool deployment, successful work on one queue can hide a stalled queue because the task-throughput signal is shared. Treat single-pool sign-off as conditional unless you add per-queue or per-task-type throughput evidence to the release packet.
InfoConnectHighErrorRate is scoped to 5xx availability failures. It intentionally does not cover 4xx/auth-pressure detection; if credential probing, scraping, or rate-limit pressure is a go-live criterion, add a dedicated 401/403/429 alert or include audit-log evidence in the release packet.
| Alert | #94 condition | Threshold | First response |
|---|---|---|---|
InfoConnectAPIDown | API down | up{job="infoconnect-api"} == 0 for 2 minutes | Check GET /health, container status, and API logs. |
InfoConnectRedisDown | Redis down | infoconnect_component_health{component="redis"} == 0 for 2 minutes | Check Redis container health and broker connectivity. |
InfoConnectQdrantDown | Qdrant down | infoconnect_component_health{component="qdrant"} == 0 for 2 minutes | Check Qdrant container health and collection availability. |
InfoConnectWorkerDown | worker down | Queue backlog exists while task throughput is zero for 10 minutes, firing after 5 minutes | Check Celery worker containers, queue bindings, and recent task logs. |
InfoConnectHighErrorRate | high error rate | 5-minute HTTP 5xx ratio above 5% for 10 minutes | Inspect recent 5xx responses and route-specific logs. |
InfoConnectHighLatency | high latency | Route p95 latency above 2 seconds for 10 minutes | Check slow endpoints, Qdrant latency, rerank usage, and host pressure. |
InfoConnectIndexingFailureSpike | indexing failure spike | 3 or more indexing failures in 15 minutes | Inspect failed Celery task logs and recent uploaded documents. |
InfoConnectQueueBacklogGrowing | growing queue backlog | Any Redis queue above 100 jobs with a positive 15-minute slope | Check worker availability and embedding throughput before scaling. |
InfoConnectHighDiskUsage | high disk usage | Host disk usage above 85% for 15 minutes | Clear safe temp files or expand storage before Qdrant/Redis writes fail. |
InfoConnectHighMemoryUsage | high memory usage | Host memory usage above 85% for 15 minutes | Check API and worker memory; scale replicas before raising embedding concurrency. |
Alert Recipients And Escalation
Primary on-call: Alertmanager receiver infoconnect-ops-primary. Secondary escalation: Alertmanager receiver infoconnect-platform-secondary. Incident commander: Alertmanager receiver infoconnect-incident-command. Route critical alerts to infoconnect-ops-primary immediately; page infoconnect-platform-secondary if the alert remains firing for 15 minutes or if client isolation, indexing, Redis, or Qdrant availability is affected; add infoconnect-incident-command for unresolved critical alerts after 30 minutes or any client-impacting outage.
Before go-live, bind those receiver names to the active UAT or production rota in Alertmanager and paste the rota link into the release evidence packet. Confirm that the Alertmanager route for service=api, service=worker, service=redis, service=qdrant, and service=host reaches infoconnect-ops-primary.
Test-Fire Evidence
Test-fire every alert before UAT sign-off by lowering its threshold in a temporary Prometheus rule copy or by using Alertmanager's silence-safe test receiver. For each alert, capture the Prometheus ALERTS query showing alertstate="firing", the Alertmanager notification, and the operator acknowledgement. Store each evidence artifact with the release packet using this naming pattern: go-live-alert-<alert-name>-<environment>-<date>.md.
Record these checks for all alerts: InfoConnectAPIDown, InfoConnectRedisDown, InfoConnectQdrantDown, InfoConnectWorkerDown, InfoConnectHighErrorRate, InfoConnectHighLatency, InfoConnectIndexingFailureSpike, InfoConnectQueueBacklogGrowing, InfoConnectHighDiskUsage, and InfoConnectHighMemoryUsage.