Contact me View CV

Case Study · LIFT

LIFT — LLM Inference with Fast Throughput

High-throughput LLM serving with Redis admission control and fixed eval suites that block promotion when pass rate regresses.

Quality gates before promotion — serving changes only ship after the fixed eval suite passes in CI.

Problem
Queue blowups and p95 spikes under bursty multi-tenant inference load.
Role
Served models with vLLM, cached prompts in Redis, micro-batched queue admission, and gated releases on fixed eval suites.
Stack
FastAPI, vLLM, Redis, Prometheus, Grafana
Impact
~5 ms cache hits vs 8,273 ms P50 / 8,423 ms P95 uncached (T4, Locust) · ~60% hit rate · 503 @ 256 queued
Links
GitHub · Live (internal endpoint)

Serving tuned for predictable latency under burst load, with eval pipelines that gate promotion—not slide decks about generic MLOps maturity.

Operational guarantees: pass/fail suites per scenario, CI blocks on regression, and production stays on the last known-good build until evals recover.

Context: Locust load tests on NVIDIA T4 (Mistral-7B-Instruct-AWQ); metrics are end-to-end request latency unless noted.

Eval-gated serving

LIFT — LLM Inference with Fast Throughput

Context: A multi-tenant copilot endpoint handled mixed prompt lengths and bursty request concurrency across teams.

Problem: Peak traffic produced queue blowups, p95 latency spikes, and poor GPU utilization due to uneven request scheduling.

What I built: Async FastAPI on vLLM with Redis admission control, plus a fixed task-suite eval pipeline wired into CI so prompt/model changes cannot promote when pass rate drops.

Impact: ~5 ms end-to-end on cache hits vs 8,273 ms P50 / 8,423 ms P95 uncached GPU inference on T4 (Locust); ~60% hit rate; HTTP 503 when 256 jobs are pending.

Tradeoff I made: Slower release cadence when suites fail—accepted in exchange for never silently shipping a worse model to all tenants.

See “Model drifts” in the architecture sandbox →

8,273 ms

P50 GPU inference (T4, uncached)

P95 / P99
8,423 ms / 8,434 ms
Model
Mistral-7B-Instruct-AWQ (4-bit)
Sample
Locust load test on NVIDIA T4

~5 ms

cache-hit latency (end-to-end)

Uncached P50
8,273 ms
Uncached P95
8,423 ms
Hit rate
~60% (Locust workload)

Architecture diagram

Inference and evals architecture diagram

Client requests -> FastAPI gateway -> vLLM worker pool -> Redis queue backpressure -> Runtime signals

Before & After

Before

Concurrent bursts produced queue buildup, unstable token throughput, and inconsistent latency for users.

p95 volatility during peaks

After

Backpressure and batching controls stabilized runtime behavior and improved throughput efficiency under contention.

~5 ms cache hits · 8,273 ms P50 uncached · 503 @ 256 queued

Execution Footprint

Core stack and operating patterns used to deliver the outcomes above.

Tech stack

FastAPI vLLM Redis Prometheus Grafana

Techniques / models / operations

Micro-Batching Backpressure Queues Load Testing Eval Gates

Explore

More case studies

Use the controls below to browse related work.

← Back to portfolio