Sentinel-HFT — AI Agents (Workstreams 4 & 5)

Status: v1.1.0 — both phases shipped, REST + UI surfaces wired. Owner: Borja Tarazona Last updated: 2026-04-22

This document is the operator-level reference for the two AI agents in Sentinel-HFT:

• Workstream 4 — Phase 1 agent (offline RCA). A deterministic nightly job that turns the day's trace + audit + compliance artifacts into a Markdown root-cause digest, archives it, and exposes it through a REST + UI surface.
• Workstream 5 — Phase 2 agent (online triage). A streaming consumer that runs three windowed detectors (latency z-score, reject CUSUM, fill-quality SPRT), enriches each firing with an LLM one-paragraph suggestion, and persists every alert to a sidecar BLAKE2b-chained log.

Both agents share three invariants:

1. Human-in-the-loop. Neither agent re-arms or disarms the risk gate, neither closes a control loop into the hardware. The RCA agent writes Markdown; the triage agent writes alerts. Operators read both.
2. Deterministic by default. Every prompt is hashed (SHA-256) and archived next to the output. The default backend is the templated, network-free path so pytest and CI runs are reproducible.
3. Tamper-evident. Triage alerts are persisted in a sidecar file with the same BLAKE2b chain discipline as the on-chain risk-audit log. The RCA digests are archived with the SHA-256 of their input prompt so the regulator can re-derive the digest from the JSON sidecar.

Workstream 4 — Nightly RCA digest

Pipeline

out/hl/**                              (drill artifacts: JSON + audit + DORA)
   │
   ▼
sentinel_hft.ai.rca_features.build_features_from_root()
   │   • per-drill throughput, reject histogram, p50/p99/mean stage latency
   │   • audit-chain integrity (record count, head hash, chain_ok)
   │   • compliance rollups (OTR, self-trade, fat-finger, CAT, MAR)
   │   • detector pass: anomaly list (kind / drill / stage / value / baseline / z)
   │   • provenance block (file paths + sha256 + size)
   ▼
sentinel_hft.ai.rca_nightly.run_nightly()
   │   • prompt template (NIGHTLY_PROMPT) + feature dict
   │   • prompt_sha256 = hash of the rendered prompt
   │   • backend dispatch:
   │       - "anthropic"  → Claude Haiku 4.5, temp 0
   │       - "template"   → deterministic, network-free
   │       - "auto"       → anthropic if ANTHROPIC_API_KEY set, else template
   ▼
out/digests/YYYY-MM-DD.md       ← human-readable Markdown
out/digests/YYYY-MM-DD.json     ← machine-readable sidecar (features + prompt hash + backend + model)

Schemas

Feature schema — sentinel-hft/rca-features/1. Top level keys: window, drills, aggregate, anomalies, provenance. See the docstring of sentinel_hft/ai/rca_features.py for the per-drill block shape.

Digest schema — sentinel-hft/rca-digest/1. JSON sidecar fields: schema, date, markdown, backend, model, prompt_sha256, generated_at, features.

Anomaly thresholds (frozen)

The detector pass in rca_features.py is intentionally non-LLM-driven: thresholds are explicit constants so a digest is auditable bit-for-bit.

The LLM only explains candidate root causes for items already on the anomaly list; it does not invent new anomalies, nor set thresholds.

Backend selection

The default in production is --backend auto:

• Anthropic API — claude-haiku-4-5, temperature 0. Used when ANTHROPIC_API_KEY is set and the anthropic package imports cleanly. Falls through silently to template on transient API errors so the nightly job never blocks ops on a network event.
• Template — pure-Python rendering of the same anomaly list into the same ## Headline / ## Anomalies / ## Candidate root causes / ## Recommended actions / ## Chain integrity section layout. This is the path used by pytest and any air-gapped deployment.

CLI

# Run today's digest (default backend = auto)
sentinel-hft ai rca-nightly \
    --artifacts out/hl \
    --digest-dir out/digests

# Force the deterministic path (no API key required)
sentinel-hft ai rca-nightly --backend template

# Re-run a historical date
sentinel-hft ai rca-nightly --date 2026-04-22

# List archived digests, newest first
sentinel-hft ai rca-list --digest-dir out/digests

REST surface (/api/ai/rca/*)

Mounted by sentinel_hft.server.app. Stateless, read-mostly.

Pydantic note. DigestSummary.schema would shadow pydantic.BaseModel.schema, so the wire-name is preserved via Field(alias="schema") + response_model_by_alias=True. The TypeScript client mirrors this in sentinel-web/lib/sentinel-types.ts.

UI surface (/sentinel/rca)

Two-column dashboard:

• Left rail — list of archived digests (most recent first), with the date, backend tag, and anomaly count per row.
• Right pane — selected digest. Shows backend, model, prompt SHA-256 (short), schema; an aggregate strip with the first nine top-level feature keys; the Markdown body; and the anomaly table coloured by severity.

The Run digest form on the same page calls POST /api/ai/rca/run with date + backend, then re-fetches the list and auto-selects the new entry. Useful for both the demo flow and for catching up after a missed nightly.

Scheduled-task config

# Production cron (UTC)
schedule: "30 4 * * *"
command: |
  sentinel-hft ai rca-nightly \
      --artifacts /var/sentinel/out/hl \
      --digest-dir /var/sentinel/out/digests \
      --backend auto \
      --no-print
env:
  ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}

The --no-print flag suppresses Markdown to stdout — the cron's job is to write the archive, not flood the operator's mail. Operators read the archive through the dashboard.

Workstream 5 — Online triage

Pipeline

trace events (PCIe DMA ring in prod / Unix pipe in sim)
   │   each event = TriageEvent { timestamp_ns, kind, stage, value, passed, meta }
   ▼
sentinel_hft.ai.triage_detectors.DetectorEnsemble.observe(event)
   │   ┌──────────────────────────────┐
   │   │ LatencyZScoreDetector        │  per-stage Welford μ/σ; fires at z ≥ z_threshold
   │   │ RejectRateCUSUMDetector      │  two-sided CUSUM on reject stream; fires on drift up
   │   │ FillQualitySPRTDetector      │  Wald SPRT on fill-at-expected-price; fires on log-LR breach
   │   └──────────────────────────────┘
   ▼
sentinel_hft.ai.triage_stream.TriageAgent
   │   • runbook lookup: (detector, severity) → docs/runbooks/{page}.md
   │   • optional LLM enrichment (Claude Haiku 4.5, temp 0, best-effort)
   │   • pager hook (pluggable Callable)
   ▼
sentinel_hft.audit.alert_log.AlertChain.append()
   │   • BLAKE2b-256, low-128-bit prev pointer (same discipline as on-chain risk audit)
   │   • sidecar file: out/triage/alerts.alog
   ▼
verify_chain() / read_alerts() — used by REST + UI

Detectors (mirror live_bot/circuit_breaker.py from the Volat project)

All three are pure stdlib (no numpy). Each owns its own rolling window. Calibration knobs live alongside each class.

Sidecar alert log

Format spec in the docstring of sentinel_hft/audit/alert_log.py.

• Magic — b"SALT" (Sentinel ALerT)
• Version — 1
• Per-record fields — framing, seq_no, timestamp_ns, severity (info / warn / alert), detector, stage, detail, score (Q32 fixed point), window_n, flags, prev_hash_lo (16 bytes), variable-length detector / stage / detail bytes, full_hash_lo (16 bytes).
• Hash chain — for record i: h_i = BLAKE2b-256(payload) with payload excluding prev_hash_lo and full_hash_lo. Then record_{i+1}.prev_hash_lo == low128(h_i) and record_i.full_hash_lo == low128(h_i).

A tampered, deleted, or reordered alert fails one of those checks and verify_chain() returns chain_ok=False with the exact bad_index and bad_reason.

The sidecar is intentionally separate from the on-chain risk-audit log: triage alerts originate in software, are observational, and have a different schema. Extending the 96-byte RTL record would force a hardware respin — a sidecar with the same hash discipline gives the combined verifier a clean two-pass.

LLM enrichment

When ANTHROPIC_API_KEY is set and the anthropic package imports cleanly, each firing gets a one-paragraph plain-English suggestion appended to the alert detail. When unavailable, a deterministic template string is used. Network failure never blocks the alert from being persisted — the alert lands in the chain regardless; only the enrichment text degrades to the template fallback.

Runbook lookup

triage_stream.RUNBOOK_PAGES maps (detector) → page fragment. Operators see one click away from the right runbook entry.

Evaluation harness

sentinel_hft.ai.triage_eval.run_evaluation() replays a scripted, fully-labelled event stream through TriageAgent and scores precision / recall / F1 against the ground-truth anomaly windows.

• True positive — alert raised inside (or within hit_window_ns after) an injected anomaly window for the same detector family.
• False positive — alert outside any anomaly window.
• False negative — anomaly window that never produced an alert from the matching detector.

Quality bar (current default scenario):

Pure-Python, deterministic seed, runs in <1 s.

CLI

# Run the scripted scenario, print precision/recall/F1
sentinel-hft ai triage-eval

# Persist the full report (events + per-alert detail) to disk
sentinel-hft ai triage-eval -o out/triage/eval_report.json

REST surface (/api/ai/triage/*)

The GET /api/ai/triage/alerts endpoint runs the chain verifier on every call and returns both the pass/fail verdict and the most-recent N decoded records. If the file does not exist yet (no alerts have ever fired), the response is a clean empty chain — not a 404.

UI surface (/sentinel/triage)

Top row:

• Chain integrity card — head hash, record count, bad-index (if any), table-limit input, refresh button.
• Eval action card — run button + 9-stat grid for events, labelled anomalies, alerts fired, TP / FP / FN, precision, recall, F1.

Below: the most-recent alerts table (most-recent first) with seq, ISO timestamp (from ns), severity badge, detector, stage, score, window_n, detail, hash low-128 (8 chars + tooltip).

When an evaluation has been run, an extra drill-down section appears with the anomaly-windows table (family / stage / matched) and the eval-alerts table (detector / severity / score / matched=TP/FP).

Cross-agent invariants

Determinism

Both agents use temperature 0 on the LLM call when one happens, and both default to a non-LLM path that produces bit-identical output for identical input. The RCA prompt is hashed (SHA-256) and stored next to the output. The triage scripted scenario is seeded.

Replayability

• Every digest's JSON sidecar carries the full feature dict that produced it. Re-running run_nightly on the same artifact tree yields the same Markdown (modulo the LLM backend's own determinism).
• Every alert in the sidecar log is independently verifiable: the combined verify_chain walks the file and re-derives every hash.
• The eval harness is deterministic — the same scripted scenario runs the same way on every machine.

Audit posture

What the agents intentionally do not do

• They never re-arm or disarm the risk gate.
• They never modify a config file or push a parameter change.
• They never write to the on-chain risk-audit log.
• They never block trades on detection — the risk gate already does that, deterministically, in hardware.

The agents compress the operator's attention funnel and produce audit-grade evidence of "what looked off and when". Acting on the evidence is a human decision.

Code map

sentinel_hft/ai/
├── rca_features.py        # WS4 — feature pipeline (deterministic detector pass)
├── rca_nightly.py         # WS4 — prompt + backends + archive writer
├── triage_detectors.py    # WS5 — Welford / CUSUM / SPRT
├── triage_stream.py       # WS5 — TriageAgent + runbook lookup + LLM enrichment
├── triage_eval.py         # WS5 — scripted scenario + precision/recall scorer
└── attribution_explainer.py  # shared LLM helper (used by both)

sentinel_hft/audit/
└── alert_log.py           # WS5 — sidecar BLAKE2b-chained log

sentinel_hft/server/
└── ai_api.py              # FastAPI router mounted at /api/ai/*

sentinel-web/
├── lib/sentinel-api.ts    # typed client: getRcaList / getRcaDetail / runRcaDigest /
│                          #               getTriageAlerts / runTriageEval
├── lib/sentinel-types.ts  # DigestSummary / DigestDetail / AlertChainView / TriageEvalResponse
└── app/sentinel/
    ├── rca/page.tsx       # WS4 dashboard
    └── triage/page.tsx    # WS5 dashboard

Tests

• tests/test_ai_api.py — FastAPI router tests for /api/ai/* (8/8).
• tests/test_rca_nightly.py — end-to-end pipeline + backend selection
  • idempotence tests (12/12).
• tests/test_triage_detectors.py — per-detector calibration.
• tests/test_triage_stream.py — TriageAgent loop + runbook lookup + alert-log persistence.
• tests/test_triage_eval.py — quality bar (recall=1.0, precision≥0.70, F1≥0.80) on the default scripted scenario.
• tests/test_alert_log.py — sidecar format + chain verifier + tamper-and-detect.

All tests are network-free by default (template backend, no API calls) and run in well under one second per file.

References

• docs/ROADMAP.md §Workstream 4 / Workstream 5 — original spec.
• docs/SENTINEL_CORE_AUDIT.md — six-axis rubric used to audit the core RTL; the audit-chain primitives the alert sidecar mirrors are closed under finding B-S0-1.
• docs/COMPLIANCE.md — regulation × control crosswalk consumed by the aggregate block of the RCA feature dict.
• live_bot/circuit_breaker.py (Volat project) — original three-detector design that the WS5 detectors mirror.