Project · Inference Serving
Inference Bench
A benchmarking API for LLM inference serving. It simulates real AI-app workloads against a small model on a single affordable GPU, lets you toggle TensorRT-LLM serving configurations from an API, runs load tests, and persists the results so you can compare configurations over time.
Why it exists
Inference serving has a lot of knobs — quantization, batch size, speculative decoding, KV-cache reuse, chunked prefill — and the usual advice is “it depends.” This project makes the trade-offs concrete and measurable.
Pick a workload, run a baseline at 100 concurrent users, flip one knob, and read the delta. Scale to 500 users to find the breaking point. Even on a small model and modest hardware, the effect of each configuration is real — and here you can watch it move.
Three real-world profiles
Chat, coding, and embeddings each model a different product — different models, request shapes, and the metrics that actually matter for them.
Flip one knob, read the delta
Every run is saved with its exact serving config, so a comparison is always apples-to-apples: same workload, one setting changed.
Find the breaking point
Run a baseline at 100 users, then scale to 500 and watch latency climb and the error rate spike as the server saturates.
Works with no GPU
If the model server is unreachable, the engine returns realistic, config-sensitive results flagged as simulated — so the whole thing is explorable right now.
How it works
The browser talks to the API; everything else — prompt synthesis, the load generator, the model engine, and persistence — runs server-side in a single container, with Azure Blob Storage as the database’s durable home across restarts.
- Source
- Where requests enter
- Process
- Deterministic logic
- Model
- Where the LLM runs
- Memory
- Persistent results
- Output
- Final benchmark result
What you can tune
Each run accepts a serving config. These are the toggles — and why each one matters for inference performance.
- Quantizationfp16 / fp8 / int4
- Precision of the weights. Decode is memory-bound, so lower precision decodes faster and shrinks the model — int4 is what fits an 8B model on a 16 GB T4.
- Batch size1 – 256
- How many requests share one forward pass. Continuous batching raises throughput, but a bigger batch nudges per-request latency up — the core trade-off.
- Speculative decodingdraft · EAGLE · n-gram
- Propose several tokens ahead and verify them in one pass. Choose the method — a small draft model (pick which one), a trained EAGLE head, or n-gram prompt-lookup — and emit multiple tokens per step with the same output, lower latency.
- Prefix cachingGPU fraction + CPU offload
- Reuse the KV cache for shared prompt prefixes (like a long system prompt). Size it as a fraction of GPU memory, and add a host-memory offload tier so even more prefixes stay cached instead of being evicted.
- Target GPUT4 → A100 → H100 → B200
- Pick the GPU the run executes against. It scales decode (bandwidth) and prefill (FP16 compute), and drives the sizing estimate: how many of that GPU it takes to fit the model.
- Chunked prefillinterleave prefill & decode
- Slice a long prompt into chunks so prefill stops monopolising the GPU and stalling in-flight decodes — smoother tail latency under mixed load.
- Max sequence lengthcontext budget
- The prompt + generation length the engine reserves KV-cache space for. Longer windows cost more VRAM per sequence, lowering how many fit in a batch.
The three profiles
Each profile simulates a real product. They stress the server in completely different ways, so each one watches different metrics.
Chat
Llama 3.1 8B · int4
Medium prompts, streamed replies, variable concurrency
Watches: Time-to-first-token & tail latency
Coding
Qwen2.5-Coder 7B · int4
Long prompts, long outputs, lower concurrency
Watches: Throughput (tokens/sec) & total generation time
Embeddings
BGE-small-en-v1.5
Short inputs, very high concurrency, batch-friendly
Watches: Requests/sec & P99 latency
See it move
Open the dashboard, pick a profile, flip some toggles, and run a load test against the live API.
Try it now