SENTINEL — Trust & Safety Triage Agent

Context: A marketplace platform needed automated trust-and-safety triage that routes fraud, refund abuse, ATO, and content-violation reports to policy-cited enforcement actions — without drifting on borderline cases.

Problem: Open LLMs hallucinate citations and under-escalate high-risk edge cases when unguarded. A naive RAG + LLM pipeline would produce plausible-sounding but unverifiable enforcement decisions.

What I built: A 7-node LangGraph state machine (guard_input → classify → policy_retrieve → decide → route → guard_output → human_review) wiring Chroma policy-RAG with structural citation enforcement, Instructor/Pydantic structured outputs at temp=0, LLM Guard / Llama Guard layered rails, and SqliteSaver checkpointing for human-in-the-loop interrupts on all high-risk actions.

Impact: 90.0% routing accuracy on the PASS gate run · 0 ungrounded citations across 75 live cases · 0.699 mean Ragas faithfulness. Eval also surfaced a reproducible under-escalation bias on 4 borderline cases — an actionable model-improvement signal.

Tradeoff I made: Two-layer grounding rather than one: structural enforcement guarantees zero hallucinated citations at code level; Ragas tracks rationale fidelity separately and honestly at 0.70 — not claimed as 1.0.

guard_input — prompt-injection and PII scan (LLM Guard / Llama Guard with offline builtin fallback).

classify — Llama 3.1 via Instructor + Pydantic (Mode.JSON, 3-retry auto-repair) → Classification(category).

policy_retrieve — Chroma top-k=4, category-filtered policy chunks from 20 section-ID-keyed policy sections.

decide — Llama 3.1 + Instructor → Decision(action, rationale, citations[]) at temp=0.

route — risk-tier assignment: approve → auto-approve queue · restrict/ban → enforcement queue · escalate → human-review queue.

guard_output — structural citation check (_enforce_grounding): any citation outside the retrieved set fails the decision before it leaves the graph.

human_review — LangGraph interrupt() on all ban/escalate actions; SqliteSaver checkpointing so paused cases survive restarts; FastAPI approve/resume endpoints drive the review queue.