API Request Latency
Inspect request timing for calls captured by the proxy.
Availability
Partially implemented in stable. Read the limits below before relying on this workflow.
Reviewed against stable v0.5.1. Rust candidate additions are labeled separately. Release status.
The pain this solves
- Anecdotal responsiveness is difficult to compare across captured HTTP calls.
- First response byte is not always the same as first model token.
What ships with it
- Proxy HTTP handling records first response byte and total request duration.
- Captured call records can include provider and model fields.
- Direct, unsupported, and tunneled WebSocket traffic is not guaranteed to have the same measurement.
Questions this answers
- How to measure LLM API latency in terminal?
- Time to first token tracking for AI coding agents
- Why is my AI coding agent slow?
- LLM API response time monitoring tool
How it works
For HTTP responses handled by the proxy, Chau7 records time to the first response byte and total request duration through response completion. The first byte is not necessarily the first generated token.
Recorded calls can include provider and model metadata and appear in analytics summaries. This metric depends on the client using a supported proxy route.
WebSocket traffic is tunneled rather than decoded as an HTTP response stream, and direct or unsupported client requests are outside this measurement path. No universal provider comparison or all-agent coverage is claimed.
Why it matters
Request timing gives a concrete view of how long a supported proxied call takes to begin returning data and to finish. It should be read as proxy-observed HTTP timing, not as a universal model TTFT or display latency benchmark.
In the Rust/GPUI candidate
The unreleased Rust/GPUI candidate at commit 350d851b457e exposes local input-dispatch, PTY queue, and output-to-CPU-paint timings through diagnostics. These are local app/terminal timings, separate from provider request latency; CPU paint does not measure physical display scanout.
Candidate features and migration plan. Download links continue to serve the latest stable release.
Frequently asked questions
Does Chau7 measure time-to-first-token?
The stable proxy records first response byte and total HTTP request duration. First byte is not necessarily the first generated model token.
Are all agent calls included?
Only calls that use a supported route through the local proxy are measured; direct and unsupported traffic is not guaranteed to appear.
Does the WSS tunnel provide the same timing detail?
Do not assume so. The WSS path tunnels frames, while the described response timing applies to HTTP responses handled by the proxy.