Monitoring Metrics

The service exposes these health and metrics surfaces:

Surface	Command or endpoint	Use
Health	GET /health	Load balancer and uptime checks
Metrics	GET /metrics	Prometheus-compatible scrape endpoint
Live dashboard	just status live	Human operations dashboard

Metrics Endpoint

GET /metrics returns Prometheus text format and does not require an API key. HTTP path labels use FastAPI route templates, with unmatched paths collapsed to __unmatched__ to avoid unbounded label cardinality. Celery workers write task duration and failure samples to Redis so the API metrics scrape can expose worker observations. It exports:

Metric	Purpose
infoconnect_http_requests_total	Request count by method, route-template path, and status
infoconnect_http_errors_total	4xx/5xx error count by method, route-template path, and status
infoconnect_http_request_duration_seconds	API latency histogram for p95/p99 queries
infoconnect_search_latency_seconds	Search latency histogram with reranked label
infoconnect_task_duration_seconds	Celery task duration histogram
infoconnect_indexing_failures_total	Indexing failure count by task
infoconnect_queue_depth	Redis queue depth for default, preprocessing, and embedding
infoconnect_component_health	API, Redis, and Qdrant health gauges (1 healthy, 0 unhealthy)
infoconnect_resource_usage_percent	Host CPU, memory, and disk usage percent
infoconnect_metadata_discovery_readiness_clients	Clients by readiness state (ready, backfilling, blocked), with no client identity
infoconnect_publication_transitions_total	Publication transitions by bounded transition and outcome
infoconnect_publication_waitaof_failures_total	Publication durability barriers that did not confirm one local AOF fsync
infoconnect_publication_invariant_violations_total	Fail-closed publication invariant violations by bounded kind
infoconnect_publication_store_operations_total	Publication store operations by bounded store, operation, and outcome
infoconnect_publication_store_records	Current record count in each publication store
infoconnect_publication_authority_state	One-hot current publication authority state
infoconnect_publication_pending_acks	Captured publication worker acknowledgements still pending by kind
infoconnect_publication_runtime	Bounded publication runtime diagnostics, including serving_ready, publishing_ready, AOF health, fence age, and Redis headroom

A client that has not finished its Metadata Backfill is a client capability state, not a service fault. blocked and backfilling counts never make infoconnect_component_health or /health fail. Content search is unaffected; only metadata and both requests require a ready Client.

The readiness gauge is emitted as infoconnect_metadata_discovery_readiness_clients{state=...} for the bounded ready, backfilling, and blocked states. It carries aggregate counts only; it never includes a client label.

Health Endpoint Semantics

The Health API publication contract owns the response fields, component conditions, and HTTP status rules. Monitor serving availability separately from publishing readiness: a serving-safe fence pauses publication writes while keeping read traffic available.

The matching bounded metrics are:

• infoconnect_publication_runtime{metric="serving_ready"} is 1 only when this process can safely serve publication-backed reads.
• infoconnect_publication_runtime{metric="publishing_ready"} is 1 only when a new publication can enter the global fence immediately.

Keep infoconnect_publication_authority_state and fence_age_seconds on dashboards for diagnosis; do not use exact authority READY as a serving-availability proxy. Alert on dependency_health_changed transitions and dependency_timeout_cluster events rather than individual failed samples.

Dashboard Queries

Use these PromQL expressions in Grafana or another Prometheus dashboard:

sum(rate(infoconnect_http_requests_total[5m]))
sum(rate(infoconnect_http_errors_total[5m])) / sum(rate(infoconnect_http_requests_total[5m]))
histogram_quantile(0.95, sum(rate(infoconnect_http_request_duration_seconds_bucket[5m])) by (le, path))
histogram_quantile(0.99, sum(rate(infoconnect_http_request_duration_seconds_bucket[5m])) by (le, path))
histogram_quantile(0.95, sum(rate(infoconnect_search_latency_seconds_bucket[5m])) by (le, reranked))
sum(infoconnect_queue_depth) by (queue)
min(infoconnect_component_health) by (component)
max(infoconnect_resource_usage_percent) by (resource)

Operations Workflow

Use /metrics for automated scraping and just status live for hands-on triage. During UAT, capture both the Prometheus dashboard screenshot or query output and a just status docker --iterations 1 snapshot as go-live evidence.

Request-correlated logs

The existing API middleware accepts X-Request-ID, generates one when absent, and echoes it in the response. API logs and request-started Celery tasks carry that request_id. Celery publishes it in task headers, then binds it with task_id, task_name, the upload job_id, client_id, document_id, version_id, and generation where known. Children and retries inherit the same request ID. Worker output uses the same structlog configuration as the API: JSON with DEBUG=false, console output with DEBUG=true. The lookup below requires JSON output. Search runs in the API; it does not enqueue a Celery task.

Capture the response header or send an ID you can search for:

curl -i -X POST "$API_URL/search" \
  -H "X-API-Key: $API_KEY" \
  -H "X-Request-ID: incident-20261013-01" \
  -H "Content-Type: application/json" \
  -d '{"query":"policy retention","scope":"content"}'

On the Docker host, use the deployment's project name and env files. This is read-only:

REQUEST_ID=incident-20261013-01
docker compose --env-file .env --env-file .runtime/prod.env \
  -f docker-compose.yml -f docker-compose.prod.yml \
  logs --no-color --no-log-prefix --since 24h \
  api worker-preprocessing-1 worker-embedding-1 worker-embedding-2 2>&1 \
  | jq -R --arg id "$REQUEST_ID" 'fromjson? | select(.request_id == $id)'

For one container, use docker logs --since 24h <container> 2>&1 with the same jq filter. With tracing enabled, logs emitted within an active trace also have trace_id and span_id; paste trace_id into the collector's trace search. These IDs are log fields, not Prometheus labels. Existing metrics and alert rules are unchanged.

Failed background tasks

GET /metrics shows how many tasks failed per task name (infoconnect_indexing_failures_total), but not which task, document, or error. The Celery result backend stores each task's metadata by ID, and the celery Redis ACL user is denied SCAN/KEYS, so the celery-task-meta-* keyspace cannot be enumerated and direct store scans stay denied. The worker's task_failure signal therefore also appends one entry per failure to a bounded log at infoconnect:metrics:task_failure_log (app/utils/metrics.py), under the key pattern the production ACL already grants.

List recent failures with the operator lookup:

just failed-tasks                     # newest first, table
just failed-tasks --limit 50 --client <client-id>
just failed-tasks --json              # machine-readable

Each row carries the failing time, the task id (the API job_id for client jobs), the Celery task name, the Client, the document/version, and the error message. The log keeps the most recent 200 failures (TASK_FAILURE_LOG_MAX_ENTRIES) on the broker connection (DB 1) and is read with a bounded LRANGE — no scan, no ACL change. Entries are written by the worker that observes the failure; run the lookup on the deployment host (or set --redis-url). GET /status/{job_id} remains the per-job, Client-scoped view; a failure that predates the deploy will not appear in the log.

Rotation and retention

The production and Console overlays set Docker's json-file driver with max-size=100m and max-file=20, configurable through DOCKER_LOG_MAX_SIZE and DOCKER_LOG_MAX_FILE. This bounds logs to about 1.95 GiB per container. It covers 24 hours at up to about 23.7 KiB/s per container; size rotation is not a time guarantee. Measure each service's daily output, allow headroom for incident bursts, and raise the budget or ship off-host if it exceeds that rate. Seven services can consume about 13.7 GiB in total, apart from volumes and images.

Docker applies driver options when containers are created, not on restart. An approved deployment must recreate containers to apply the new policy. This change does not deploy or recreate anything in production. Do not delete log files under Docker's data directory; let Docker rotate them.

Search debug trace

Every /search request, for every API key, writes one bounded search_trace record while SEARCH_TRACE_ENABLED (the default) is on. The record goes only to the durable sink below: the ordinary application log keeps the redacted search_completed line above, with no query text and no scores. Set SEARCH_TRACE_ENABLED=false to roll back — no record is built or written anywhere and the behaviour matches the default path exactly.

The record carries the request request_id, the resolved client_id (never the API key), the scope and requested limit the search ran with, the original query, the corrected_query actually retrieved with after abbreviation expansion, known-typo correction and corpus fuzzy correction (query_rewritten reports whether they differ), the top results (results: the strongest content and metadata item per Document Version, with score, the page when the payload has one, and a text preview), and per-stage timings (stages: stage name, call count, total and max milliseconds). It is bounded in app/utils/search_trace.py: at most 10 results (results_limit reports that cap), 200 characters per query or preview, and 32 stage entries, so one record stays a few KiB.

Retention is 30 days in a durable sink. Compose mounts the API container at /var/log/infoconnect from the host directory SEARCH_TRACE_LOG_HOST_DIR, and the sink keeps <SEARCH_TRACE_LOG_PATH>/search-trace.jsonl there, rotated at UTC midnight and pruned to SEARCH_TRACE_RETENTION_DAYS (default 30) daily files. The host path is what makes the horizon survive container recreation and redeploys: Docker's json-file rotation alone dies with the container. Set SEARCH_TRACE_LOG_HOST_DIR to the host path that must keep the traces (the default is ./.runtime/search-trace); on a redeploy the API container is recreated and the sink keeps appending to the same host file. Size the host filesystem for search volume: one record is at most a few KiB and every search writes one.

Search trace permissions

The API container runs as uid/gid 999. Before starting services, just prod-up (including prod-up rebuild and prod-rebuild) creates the mounted /var/log/infoconnect directory, assigns ownership to 999:999, grants owner read/write/execute permission, and probes a write as that user. Compose resolves the host mount, including SEARCH_TRACE_LOG_HOST_DIR set through .env. Preparation or probe failure stops the recipe before service startup.

For direct Compose startup, create the host directory beforehand and give it that ownership (mkdir -p "$SEARCH_TRACE_LOG_HOST_DIR" && chown 999:999 "$SEARCH_TRACE_LOG_HOST_DIR"); Docker otherwise creates it root-owned on Linux. The API opens the sink file at startup and, if the directory is not writable, logs search_trace_sink_unavailable and writes no traces.

# Trigger a search with any configured key
curl -s -X POST "$API_URL/search" -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" -d '{"query":"attn jpm","scope":"both"}' >/dev/null

# Durable sink (host path, survives container recreation and redeploys)
jq -c 'select(.event == "search_trace")' "$SEARCH_TRACE_LOG_HOST_DIR"/search-trace.jsonl*
ls -1 "$SEARCH_TRACE_LOG_HOST_DIR"/search-trace.jsonl*   # one file per retained UTC day

Traces hold every customer's raw query text, so treat the sink as sensitive production data: only operators already cleared to read customer data may read it, keep it on the host (never copy it into tickets or shared drives), and rely on the 30-day prune instead of manual deletion.

scripts/qa_stack.py provisions the sink for isolated stacks: the API container mounts the stack's own host directory .qa/<project>/search-trace, so a live QA check reads the traces from disk while the app log stays redacted.

Optional off-host search

No shipping service is enabled or deployed here. An operator can separately deploy Vector or Grafana Alloy on the host, decode the Docker JSON envelope and the inner structlog JSON, and send records over authenticated TLS to Loki or a hosted log store. Preserve timestamp, service/container, request_id, trace_id and task identity. Use service/environment as indexed labels; keep request/trace/task/document IDs as structured fields, not high-cardinality Loki labels. For example, after JSON decoding:

{service="infoconnect", environment="production"} | json | request_id="incident-20261013-01"

Choose off-host retention of at least 24 hours, persistent shipping checkpoints, bounded disk buffering, redaction rules, access controls and an alert for dropped logs. Existing DEBUG logs can contain search text, so review fields before shipping. Docker socket access grants host-level power; prefer read-only log-file ingestion with the minimum host permissions. Shipping and collector deployment need separate approval.

Optional OpenTelemetry tracing

Tracing is off by default. The tracing extra in pyproject.toml owns the pinned SDK, OTLP HTTP/protobuf exporter and HTTPX/Redis instrumentation dependencies; uv.lock owns their resolved versions. No auto-instrumentation agent, gRPC exporter, OpenTelemetry log exporter or metrics exporter is installed. Default builds omit the extra entirely.

Disabled startup never imports or initializes OpenTelemetry and creates no exporter, instrumentation threads or collector traffic. It omits service wrappers and the trace log processor. Install the optional extra before enabling; setting the flag without installing it fails startup. See the configuration reference for environment variables and defaults.

Spans cover query parsing/spelling, dense and lexical query preparation, Metadata dense preparation, lane fusion, rerank, publication begin/candidate/final checks, Qdrant calls, HTTPX and synchronous/asynchronous Redis commands. A hybrid Qdrant request runs dense, sparse prefetch and RRF on the server in one call; its server-side sub-times cannot be separated by client spans. Embedding and rerank are local FastEmbed inference, not remote HTTP services; embedding.dense, embedding.sparse and rerank.predict measure that work. Stages that the request does not use produce no synthetic spans.

API requests accept W3C traceparent/tracestate. Celery task headers propagate those fields from the API and between child tasks. Each task creates a consumer span. Prefork workers initialize their exporters after fork; shutdown flushes the batch queue. Trace export keeps timing, topology and bounded method/status/operation attributes, not request bodies, HTTP URLs, Redis statements, exception messages or stack traces. Audit logs still use their existing sanitizers; other application logs are not filtered by the trace exporter and need the shipping review described above.

Local collector example

Use a separate Compose project and non-conflicting API/Redis/Qdrant ports for local testing. Do not run these commands against a shared or production stack. A standalone local Jaeger instance can receive OTLP HTTP and expose its UI:

docker run --rm --name ifc-local-traces \
  -p 127.0.0.1:16687:16686 -p 127.0.0.1:14318:4318 \
  jaegertracing/jaeger:2.6.0

For local Python API and worker processes:

uv sync --frozen --extra dev --extra tracing
export TRACING_ENABLED=true
export OTEL_SERVICE_NAME=infoconnect-local
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:14318/v1/traces
export TRACING_SAMPLE_RATIO=1.0

Start the API and both worker pools with these variables and the isolated stack's URLs. The flag is a startup setting: restart API and workers after changing it. Browser clients may send traceparent/tracestate through CORS and read the exposed X-Request-ID. Send a search and an upload, then open http://127.0.0.1:16687 and look up their logged trace IDs. The upload's trace continues in Celery after the HTTP response.

For containerized API/workers, build with TRACING_EXTRA=1 and set TRACING_ENABLED=true. The collector endpoint must be reachable inside every enabled container: localhost there means that container, not the host. Put a separately managed collector on the isolated network, or configure a host-gateway route in a local override. Rebuild and recreate only that isolated project. Empty TRACING_EXTRA and TRACING_ENABLED=false restore the default runtime.

Production collector requirements

A later approved deployment must provide an OTLP HTTP/protobuf receiver reachable from all API and worker replicas, TLS and authentication, restricted network ingress, persistent trace storage and an explicit retention/access policy. Configure sampling for the expected volume, collector memory limits, bounded queues, durable buffering, and export/drop/error monitoring. API and worker service names must be searchable together. Mount any custom CA and set the optional exporter headers through secret management. Exporter timeout is 5 seconds and export runs on a bounded batch queue; a collector outage must not change publication or search correctness, but can lose trace evidence. No collector, shipping agent, production flag change or deployment is part of this change.

Alert Rules

Load the go-live Prometheus rules from monitoring/prometheus-alerts.yml. These rules assume the API scrape job is named infoconnect-api; worker-down detection uses the task metrics and queue-depth gauges exported by the API /metrics endpoint rather than per-worker scrape targets. Keep the same labels in UAT and production so alert routing can separate environments by the Prometheus environment external label.

InfoConnectWorkerDown assumes the preprocessing and embedding queues are operated as distinct worker pools. In a single-pool deployment, successful work on one queue can hide a stalled queue because the task-throughput signal is shared. Treat single-pool sign-off as conditional unless you add per-queue or per-task-type throughput evidence to the release packet.

InfoConnectHighErrorRate is scoped to 5xx availability failures. It intentionally does not cover 4xx/auth-pressure detection; if credential probing, scraping, or rate-limit pressure is a go-live criterion, add a dedicated 401/403/429 alert or include audit-log evidence in the release packet.

Alert	Condition	Threshold	First response
InfoConnectAPIDown	API down	up{job="infoconnect-api"} == 0 for 2 minutes	Check GET /health, container status, and API logs.
InfoConnectRedisDown	Redis down	infoconnect_component_health{component="redis"} == 0 for 2 minutes	Check Redis container health and broker connectivity.
InfoConnectQdrantDown	Qdrant down	infoconnect_component_health{component="qdrant"} == 0 for 2 minutes	Check Qdrant container health and collection availability.
InfoConnectWorkerDown	worker down	Queue backlog exists while task throughput is zero for 10 minutes, firing after 5 minutes	Check Celery worker containers, queue bindings, and recent task logs.
InfoConnectHighErrorRate	high error rate	5-minute HTTP 5xx ratio above 5% for 10 minutes	Inspect recent 5xx responses and route-specific logs.
InfoConnectHighLatency	high latency	Route p95 latency above 2 seconds for 10 minutes	Check slow endpoints, Qdrant latency, rerank usage, and host pressure.
InfoConnectIndexingFailureSpike	indexing failure spike	3 or more indexing failures in 15 minutes	Inspect failed Celery task logs and recent uploaded documents.
InfoConnectQueueBacklogGrowing	growing queue backlog	Any Redis queue above 100 jobs with a positive 15-minute slope	Check worker availability and embedding throughput before scaling.
InfoConnectHighDiskUsage	high disk usage	Host disk usage above 85% for 15 minutes	Clear safe temp files or expand storage before Qdrant/Redis writes fail.
InfoConnectHighMemoryUsage	high memory usage	Host memory usage above 85% for 15 minutes	Check API and worker memory; scale replicas before raising embedding concurrency.
InfoConnectAPIContainerMemoryWarning	API container memory high	cAdvisor usage above 85% of its limit for 5 minutes	Check cgroup current, anon, swap, and pressure for each API replica.
InfoConnectAPIContainerMemoryCritical	API container memory critical	cAdvisor usage above 95% of its limit for 1 minute	Inspect API cgroup and process memory; do not raise the limit as a substitute for bounding work.
InfoConnectAPIContainerSwapGrowing	API container swap growing	Any positive 5-minute swap delta (immediate)	Compare swap, anon, reclaim, and PSI; preserve cgroup evidence.
InfoConnectAPIContainerOOMKill	API container OOM kill	Any cAdvisor OOM event (immediate)	Preserve kernel and cgroup OOM evidence; check for a container restart.
InfoConnectAPIContainerRestart	API container restart	Any Docker container start event in 5 minutes, including initial starts (immediate)	Confirm whether the restart was an approved deployment; correlate Docker and kernel logs.
InfoConnectMemoryPressureWarning	API container memory pressure warning	memory_pressure_warning with outcome=warning in Loki (immediate)	Inspect container-scoped memory and swap before the limit is reached.
InfoConnectPublicationAuthorityUnavailable	publication serving unavailable	infoconnect_publication_runtime{metric="serving_ready"} < 1 for 30 seconds	Read serving is unsafe; keep search fail-closed and follow the publication recovery runbook.
InfoConnectPublicationPublishingDegraded	publication publishing degraded	publishing_ready == 0 and ignoring(metric) serving_ready == 1 for 1 minute	Reads remain available; inspect fence overlap, owners, and pending acknowledgements before restarting anything.
InfoConnectPublicationAOFUnhealthy	publication AOF unhealthy	infoconnect_publication_runtime{metric="aof_healthy"} < 1 for 30 seconds	Check Redis AOF and publication persistence configuration.
InfoConnectPublicationLowRedisHeadroom	publication Redis headroom low	infoconnect_publication_runtime{metric="redis_headroom_bytes"} < 134217728 for 5 minutes	Stop publishers and restore Redis capacity before authority writes fail.
InfoConnectPublicationDurabilityFailure	publication durability barrier failed	Any increase in infoconnect_publication_waitaof_failures_total (immediate)	Keep affected clients fenced and follow lost-result recovery.
InfoConnectPublicationInvariantViolation	publication invariant violation	Any increase in infoconnect_publication_invariant_violations_total (immediate)	Keep the authority fenced; inspect invariant diagnostics.
InfoConnectPublicationFenceStuck	publication fence stuck	infoconnect_publication_runtime{metric="fence_age_seconds"} > 30 for 1 minute	Check leases and worker fence/convergence acknowledgements.
InfoConnectPublicationWorkerACKMissing	publication worker acknowledgement missing	infoconnect_publication_pending_acks > 0 for 30 seconds	Check captured workers; activation requires every ACK.
InfoConnectPublicationJournalGrowth	publication journal growth	infoconnect_publication_runtime{metric="journal_entries"} > 100000 for 15 minutes	Apply the documented backup-aware retention procedure.

Container Memory Alerts

The Prometheus rules in monitoring/prometheus-alerts.yml use cAdvisor job infoconnect-cadvisor and the Docker Compose label container_label_com_docker_compose_service="api". Scrape every API replica and retain that label. The rules compare container_memory_usage_bytes with container_spec_memory_limit_bytes; swap growth and OOM events use container_memory_swap and container_oom_events_total. The alert file defines rules only; it does not configure or deploy cAdvisor.

The warning fires above 85% of the container limit for 5 minutes. Critical fires above 95% for 1 minute or when swap grows over 5 minutes. OOM alerts fire without a pending period; restart detection uses the Docker event stream below. For diagnosis, capture cgroup current/max, memory.stat anon, swap current/max, memory.events max/oom/oom_kill, /proc/1/status RSS, and memory.pressure PSI at 15–30 second intervals from an external collector.

monitoring/loki-alerts.yml routes the structured memory_pressure_warning event with outcome=warning to Alertmanager as service=api, severity=warning. Load it in Loki Ruler and keep the existing service=api Alertmanager route to infoconnect-ops-primary. This rule requires the existing log shipping and Loki Ruler path; this repository change does not deploy a collector, load production rules, or test a notification.

InfoConnectAPIContainerRestart lives in monitoring/loki-alerts.yml. A later approved deployment must continuously ship Docker Engine events from every Docker host to Loki, with stream labels job="infoconnect-docker-events", compose_service from Actor.Attributes["com.docker.compose.service"], instance for the host, and container from Actor.ID. Preserve the JSON event body and its event timestamp. The source is Docker's event stream, for example docker events --filter type=container --format '{{json .}}'. Container stdout shipping alone does not include these events. The shipper needs Docker Engine read access; persist events and monitor collection gaps, because Docker only retains its last 256 events for replay.

The rule fires without a pending period on Type="container", Action="start". This detects automatic restarts and manual stop/start sequences even when the container ID and creation time stay unchanged. Initial starts and deployment recreates also alert; correlate them with the approved change window. cAdvisor's container_start_time_seconds is Docker's creation time and cannot detect a same-container restart. Detection latency includes shipping and Ruler evaluation; the event remains eligible for 5 minutes. No Docker event collector is installed or enabled by this repository change. Until this prerequisite and the Loki Ruler rule are configured, restart alerting is unavailable.

Container OOM And Restart Diagnosis

docker inspect cannot show OOM history: State.ExitCode and State.OOMKilled reset on every container start, so a running container that was OOM-killed an hour ago reports exit=0 oom=false. The durable evidence lives in the kernel log (Memory cgroup out of memory: Killed process ...) and in dockerd's restarting container ... exitCode=137 journal lines.

Run the host-side check on the Docker host to summarize both surfaces:

uv run python scripts/check_container_oom.py --since "24 hours ago"
uv run python scripts/check_container_oom.py --syslog /var/log/syslog --json

The script exits non-zero when it finds a kernel OOM kill or an exitCode=137 restart in the window, so it can also run from cron or an external health checker as a poor-man's OOM alert. Treat exitCode=137 as an unexpected-kill signal and correlate it with the kernel OOM lines — a forced SIGKILL produces 137 too. exitCode=3 restarts are uvicorn startup failures (dependency not ready), a different failure class.

When a container OOMs, compare the victim's anon-rss against its Compose memory limit (PROD_API_MEMORY, PROD_QDRANT_MEMORY, ...) from .runtime/prod.env, then regenerate the plan with just prod-runtime-plan rather than hand-editing the generated file.

Alert Recipients And Escalation

Primary on-call: Alertmanager receiver infoconnect-ops-primary. Secondary escalation: Alertmanager receiver infoconnect-platform-secondary. Incident commander: Alertmanager receiver infoconnect-incident-command. Route critical alerts to infoconnect-ops-primary immediately; page infoconnect-platform-secondary if the alert remains firing for 15 minutes or if client isolation, indexing, Redis, Qdrant, or publication availability is affected; add infoconnect-incident-command for unresolved critical alerts after 30 minutes or any client-impacting outage.

Before go-live, bind those receiver names to the active UAT or production rota in Alertmanager and paste the rota link into the release evidence packet. Confirm that service=api routes warning and critical alerts to infoconnect-ops-primary, and that service=worker, service=redis, service=qdrant, service=publication, and service=host reach the same primary receiver.

Test-Fire Evidence

Test-fire every Prometheus and Loki alert before UAT sign-off by lowering its threshold in a temporary rule copy or by using Alertmanager's silence-safe test receiver. Capture the firing rule state, Alertmanager notification, and operator acknowledgement. Store each evidence artifact with the release packet using this naming pattern: go-live-alert-<alert-name>-<environment>-<date>.md.

Record these checks for all alerts: InfoConnectAPIDown, InfoConnectRedisDown, InfoConnectQdrantDown, InfoConnectWorkerDown, InfoConnectHighErrorRate, InfoConnectHighLatency, InfoConnectIndexingFailureSpike, InfoConnectQueueBacklogGrowing, InfoConnectHighDiskUsage, InfoConnectHighMemoryUsage, InfoConnectAPIContainerMemoryWarning, InfoConnectAPIContainerMemoryCritical, InfoConnectAPIContainerSwapGrowing, InfoConnectAPIContainerOOMKill, InfoConnectAPIContainerRestart, InfoConnectMemoryPressureWarning, InfoConnectPublicationAuthorityUnavailable, InfoConnectPublicationPublishingDegraded, InfoConnectPublicationAOFUnhealthy, InfoConnectPublicationLowRedisHeadroom, InfoConnectPublicationDurabilityFailure, InfoConnectPublicationInvariantViolation, InfoConnectPublicationFenceStuck, InfoConnectPublicationWorkerACKMissing, and InfoConnectPublicationJournalGrowth.