Contact me View CV

Case Study · BENCH

BENCH — Benchmarking Engine for Neural Code Helpers

agent-coding-bench — measurement-first coding-agent evaluation with EvalPlus pass@k and per-task JSONL traces.

Problem
Coding-agent quality was measured ad hoc—no reproducible EvalPlus pass@k pipeline or per-task traces.
Role
Configured YAML codegen runs, scored HumanEval+ with EvalPlus, and logged per-task latency and cost in JSONL traces.
Stack
Python, EvalPlus, YAML configs, JSONL traces
Impact
164 tasks · EvalPlus 1,412 ms wall (parallel=2) · 3,445→1,412 ms · codegen 662–6,614 ms
Links
GitHub · Live (internal pilot)

Measurement-first evaluation for coding agents: execution-based scoring, reproducible artifacts, and per-task latency and cost traces—not ad-hoc script runs.

Operational guarantees: EvalPlus-compatible samples.jsonl, per-run manifests, and traces.jsonl with base_status / plus_status per task.

Context: HumanEval+ mini (164 tasks) on WSL/Linux with EvalPlus execution; latest run uses openai_humaneval_mini.yaml and gpt-4o-mini (eval_parallel=2).

Agent coding benchmark

agent-coding-bench: measurement-first coding-agent evaluation

Context: Comparing LLM codegen backends required execution-based scoring—not manual spot checks on generated functions.

Problem: Ad-hoc script runs produced inconsistent pass rates, no per-task latency or cost traces, and no EvalPlus-compatible artifact trail.

What I built: YAML-configured CLI that writes EvalPlus-compatible samples.jsonl, runs HumanEval+ execution scoring, emits traces.jsonl with per-task latency and token cost, and stores run manifests for reproducible comparison.

Impact: Full HumanEval+ mini split (164 tasks) runs through EvalPlus in 1,412 ms wall time (eval_parallel=2) vs 3,445 ms on an earlier run; per-task codegen latency 662–6,614 ms (mean ~1.7 s) in stored traces.jsonl.

Tradeoff I made: Single-function HumanEval+ scope over multi-file SWE-agent loops—in exchange for reproducible pass@k measurement and interview-defensible artifacts.

164

HumanEval+ mini tasks per run

Dataset
HumanEval+ mini split
Config
openai_humaneval_mini.yaml
Sample
164 traces.jsonl lines per run

1.4s

EvalPlus suite wall time

Early run
3,445 ms eval wall
Latest
1,412 ms (parallel=2)
Codegen
662–6,614 ms per task (mean ~1.7 s)
Sample
eval_summary.json · traces.jsonl

Architecture diagram

Agent benchmark pipeline: YAML config through EvalPlus to traces

YAML config -> codegen (LLM) -> samples.jsonl -> EvalPlus execution -> traces.jsonl + manifest

Before & After

Before

Ad-hoc script runs produced inconsistent pass rates and no comparable latency or cost traces across backends.

No samples.jsonl · no eval_summary.json

After

YAML-configured CLI writes samples.jsonl, runs EvalPlus, and logs per-task latency and cost in traces.

164 tasks · 3,445→1,412 ms eval wall · 662–6,614 ms codegen

Execution Footprint

Core stack and operating patterns used to deliver the outcomes above.

Tech stack

Python EvalPlus OpenAI API YAML configs JSONL traces

Techniques / models / operations

Execution-based scoring pass@k measurement Run manifests Per-task cost + latency

Explore

More case studies

Use the controls below to browse related work.

← Back to portfolio