LLM inference control plane: Redis Streams queue and backpressure
Context: Async inference requests needed admission control, observable queue depth, and predictable failure modes under burst load.
Problem: Unbounded enqueue would saturate GPU workers; API callers had no signal when the system was at capacity.
What I built: FastAPI control plane with Redis Streams job queue, sliding-window rate limiting (100 req / 60 s per API key), HTTP 503 backpressure at 256 pending jobs, and Prometheus metrics scraped every 15 s across a 5-service Docker Compose stack.
Impact: Queue saturation returns explicit 503 responses instead of silent degradation; 8 Prometheus metric families track latency, cache hits, rate limits, and queue depth.
Tradeoff I made: Accepted rejected requests at saturation in exchange for predictable worker behavior and debuggable queue signals.
GCP
Kubernetes
Docker
Redis
REST APIs