Blog post · November 20, 2026 · aviznetworks.com
Aviz Networks details per-application inference metering from network traffic
- About Aviz Networks
- Also names It
- Network
- Nvidia
- Read the original
Summary
ASN AI Correlator passively meters OpenAI-compatible inference traffic, producing one record per call with requester, model, tokens, latency and completion. GPU operators need per-application data because inference engines report only aggregates, limiting billing, SLA enforcement and capacity planning.
Measurement Approach
The correlator observes a copy of mirrored traffic and does not sit in the request path. It rebuilds each connection between application and inference server, decrypting traffic where needed, and reads both the request and the response. Requests reveal the model, conversation size, requested output length, streaming mode and agentic tool use.
Responses reveal token consumption, delivery timing and how each call ended, whether completed, truncated, failed or abandoned. Token counts come from the server's usage report when present and are estimated from observed traffic otherwise, with estimated records marked as such. Timing metrics, including time to first token, end-to-end latency, inter-token latency and output throughput, are computed from packet timestamps.
Visibility Beyond the Inference Engine
Engines expose internal state such as queue depth, batches and GPU memory but lack caller identity and disconnection data. Network observation adds per-call attribution, latency as seen on the network including proxies and queueing, and independently observed token counts. It also detects abandoned requests, where clients disconnect before completion while engines may keep generating.
Additional signals include truncated answers that hit output limits, applications that routinely reserve far more output capacity than they use, and load context recorded at arrival time. The approach works with any engine using the OpenAI-compatible interface, including endpoints not registered with the operator.
Encrypted Traffic and Accuracy Reporting
Production inference is increasingly served over HTTPS. Instead of a terminating proxy, a lightweight agent on the server shares per-session keys with ASN over a secured channel, and ASN decrypts its own mirrored copy. Server certificates remain on the server, the live connection is not modified, and traffic passes unobserved when keys are unavailable.
Each record carries a capture quality verdict of complete, structurally incomplete, or affected by wire loss. Operators can limit billing and SLA reports to fully observed calls, and track the share of incomplete records as a health metric for the monitoring path.
Deployment Options
ASN can run on the NVIDIA BlueField-3 DPU fronting the GPU server, where it sees all traffic before it reaches applications and uses no host CPU. The vendor states that tenants cannot bypass or alter this vantage point. For fleets without DPUs, ASN runs as a container on the GPU server, tapping traffic at the NIC on cores not used by application workloads.
Records are identical under both models. The vendor reports that the container option is running today on a multi-application vLLM host, feeding per-application dashboards. Records stream into existing analytics tools such as Kafka, Elasticsearch and Kibana.
Operational Impact
Per-application token metering supports usage-based chargeback in place of flat-rate capacity pricing. Independently measured responsiveness allows SLA disputes to be resolved from data, and abandoned generation, over-reserved budgets and truncated answers inform right-sizing decisions. The network-derived count is one that neither operator nor tenant controls.
The vendor states that prompt text is not exported unless the operator enables it, and credentials are never exported in clear. Operators choose which data categories leave the appliance. The vendor also cites NVIDIA benchmarking guidance and OpenTelemetry GenAI conventions as the basis for its latency and token metrics.
Inference engines expose their internal state, while network observation captures per-application calls, token use, latency and completion status. This Blog Summary is a fact-based summary of the vendor blog, and the approach is relevant to enterprise decision-makers operating shared GPU infrastructure.
Our summary. The original keeps its own byline.