NVIDIA AIPerf Launches as a New LLM Benchmarking Tool for Inference at Scale

NVIDIA AIPerf Replaces GenAI-Perf With Faster, Scalable Testing for AI Inference Workloads

NVIDIA is addressing that problem with AIPerf, a new open-source benchmarking tool designed specifically for generative AI inference.

NVIDIA introduced AIPerf on September 18, 2026, describing it as the designated successor to GenAI-Perf and a ground-up rewrite rather than a minor update.

The tool is designed to generate realistic workloads, measure latency and throughput, collect GPU telemetry, replay production-style traces and run distributed benchmarks across multiple nodes.

That makes AIPerf relevant not only for developers testing an LLM on a single server, but also for teams building large-scale inference infrastructure.

What Is NVIDIA AIPerf?

AIPerf is a command-line benchmarking tool for measuring the performance of generative AI models served by inference systems.

Its purpose is to create controlled workloads and report how an inference system behaves under those workloads.

The project provides metrics including:

Time to First Token (TTFT)
Inter-Token Latency (ITL)
Request latency
Output token throughput

It also reports percentile distributions, including p50, p90, p95 and p99, helping engineers understand both typical and worst-case performance.

The project is publicly available through NVIDIA's AIPerf repository, while NVIDIA's documentation currently lists AIPerf v0.12.0 as the latest stable version.

Why NVIDIA Rebuilt GenAI-Perf

AIPerf exists because benchmarking AI inference at high concurrency creates a problem that is easy to overlook.

The benchmark client itself can become the bottleneck.

NVIDIA says many traditional benchmark systems rely on a single-process design that can become constrained under high request rates or concurrency. AIPerf instead uses a multiprocess architecture, separating load generation and result processing while coordinating the components through ZMQ.

The goal is straightforward:

The server should be the thing being measured, not the benchmarker's ability to generate traffic.

This distinction becomes increasingly important as AI inference systems handle more simultaneous users and longer agentic workflows.

AIPerf Supports 15+ Endpoint Types

AIPerf is not limited to one type of LLM request.

NVIDIA says the tool supports 15 or more endpoint types, including chat, responses, ranking, embeddings, image generation and other workloads.

The current documentation shows dedicated support for workloads such as:

OpenAI-compatible chat APIs
Responses APIs
Embedding models
NIM ranking endpoints
Hugging Face ranking endpoints
Cohere ranking endpoints
Multimodal embedding workloads
Image generation
Video generation
Other custom endpoint configurations

This wider endpoint coverage means developers can use one benchmarking framework across different parts of a generative AI stack rather than maintaining separate benchmark tools for every model type.

AIPerf Can Simulate More Realistic Traffic

One of the most useful parts of the new tool is its ability to control how requests arrive.

A simple benchmark might send requests at a perfectly constant rate.

Real production traffic rarely behaves that way.

Users appear at different times, workloads arrive in bursts and prompt lengths vary significantly.

AIPerf supports arrival patterns such as constant, Poisson and gamma distributions, along with configurable burstiness and gradual ramping of concurrency or request rate.

This allows engineers to test an inference server under conditions that are closer to real-world usage.

Constant Traffic

A constant pattern can be useful for establishing a clean baseline.

For example, a benchmark can maintain a fixed request rate to determine how an inference system behaves under stable load.

Poisson Traffic

Poisson arrivals introduce random variation between requests.

Instead of receiving exactly the same number of requests every second, the server experiences natural gaps and bursts.

NVIDIA demonstrates this approach in its AIPerf example by combining request-rate controls with variable input and output lengths.

Gamma and More Flexible Workloads

AIPerf also supports other arrival distributions and configurable workload characteristics.

This gives engineers more control over how closely a benchmark should resemble an expected production environment.

Developers Can Replay Production-Style Workloads

Synthetic workloads are useful, but production systems often behave differently from laboratory benchmarks.

AIPerf addresses that by supporting external datasets and trace replay.

NVIDIA says the tool supports datasets such as ShareGPT and trace formats associated with Mooncake, Baseten and WEKA AgentX.

The documentation also lists support for raw payload replay, multi-turn conversations and additional benchmark datasets.

That creates a path from simple testing to workload reproduction.

An engineering team can begin with synthetic prompts, then move toward traffic patterns based on the types of interactions its infrastructure actually handles.

Four Core Metrics Matter Most

AIPerf provides detailed performance information, but four measurements are especially important.

Time to First Token

TTFT, or Time to First Token, measures how long a system takes to begin producing a response after receiving a request.

For interactive AI applications, this is one of the most visible latency signals.

A model may eventually generate thousands of tokens per second, but users still experience the initial delay before the first token appears.

Inter-Token Latency

ITL, or Inter-Token Latency, measures the time between generated tokens.

A system can have acceptable initial latency but still feel slow if tokens arrive unevenly.

ITL therefore provides a closer look at the decoding stage.

Request Latency

Request latency measures the full time required to complete a request.

It combines the different stages of the inference process into one end-to-end measurement.

Output Token Throughput

Output token throughput measures how quickly the system generates tokens across concurrent requests.

This metric is especially relevant for capacity planning because it helps engineers understand how much work their infrastructure can handle.

Percentiles Reveal Problems Average Numbers Hide

AIPerf does not only report averages.

It provides percentile breakdowns, including p50, p90, p95 and p99, as well as minimum, maximum and standard-deviation values.

This matters because an average can hide poor experiences for a smaller group of requests.

Imagine an inference service where the average TTFT looks healthy, but the p99 is dramatically higher.

That could mean most users get fast responses while a small but important portion experience severe delays.

For production AI systems, those long-tail cases can matter considerably.

GPU Telemetry Is Built Into the Benchmark

Another useful feature is the ability to correlate AI performance with hardware activity.

When DCGM or pynvml is available, AIPerf can collect GPU power draw, utilization and memory consumption as part of the benchmark output.

This removes the need to completely separate performance measurement from hardware monitoring.

For example, if latency suddenly increases while GPU memory pressure also rises, engineers have additional evidence when diagnosing the cause.

That can be useful when optimizing inference configurations or investigating capacity limits.

AIPerf Supports Distributed Benchmarking

Modern AI inference is increasingly distributed.

Large deployments may use multiple servers, GPUs and Kubernetes clusters rather than a single machine.

AIPerf is designed for these environments too.

NVIDIA says the tool can run multi-node Kubernetes deployments, replay production traces, perform KV-cache-related workflows and run sweeps across different concurrency levels.

The documentation also describes distributed testing as one of the areas where AIPerf expands on GenAI-Perf.

That makes the project relevant to teams operating inference infrastructure beyond a single GPU server.

AIPerf Can Benchmark More Than One Model Configuration

Benchmarking is most useful when engineers can compare different configurations.

AIPerf supports model selection and custom endpoints, while its documentation includes tools for parameter sweeps, adaptive search and multi-run statistics.

This can allow teams to compare:

Different models
Different concurrency levels
Different request rates
Different prompt lengths
Different inference servers
Different hardware configurations

Rather than asking only whether a model works, developers can examine how much performance changes as workload conditions change.

AIPerf Is a Command-Line Developer Tool

AIPerf is built for engineers rather than casual AI users.

Installation is available through standard Python tooling, including uv and virtual environments, according to NVIDIA's official walkthrough.

The project also provides a terminal-based live dashboard during benchmark runs.

The interface displays live progress, performance statistics and execution logs, while completed runs can be exported into CSV and JSON formats for additional analysis.

This makes the results easier to integrate into engineering workflows and repeat later.

How AIPerf Compares With GenAI-Perf

AIPerf is not simply another name for GenAI-Perf.

NVIDIA describes it as a ground-up rewrite and the designated successor to GenAI-Perf.

The new architecture changes how workload generation and result processing operate.

The documentation also shows expanded endpoint support, distributed testing, server-metrics collection, multi-turn benchmarking and additional workload capabilities compared with the earlier tool.

For teams already using GenAI-Perf, NVIDIA provides a migration guide to explain the differences between the two command-line tools.

Why LLM Benchmarking Is Becoming More Important

AI models are becoming more capable, but model quality is only one side of an AI deployment.

An application can use an excellent model and still deliver a poor experience if:

Time to first token is too high
Throughput collapses under concurrency
GPU memory becomes a bottleneck
Tail latency becomes unpredictable
Infrastructure cannot handle production traffic

This is especially relevant for agentic AI systems.

An AI agent may make several model requests during one workflow, potentially combining planning, tool selection, retrieval and execution.

A small latency increase at each stage can add up across the complete workflow.

That makes reliable inference benchmarking increasingly important.

Who Should Use NVIDIA AIPerf?

AIPerf is mainly intended for teams responsible for building or operating AI inference systems.

It can be useful for:

AI Infrastructure Engineers

Teams can evaluate how inference servers behave under different traffic conditions.

Model Developers

Model teams can compare inference performance across model variants and configurations.

Cloud Providers

Infrastructure providers can test capacity, concurrency and latency before deploying AI services at scale.

Enterprise AI Teams

Organizations running internal LLM applications can estimate whether their infrastructure will handle expected workloads.

AI Agent Developers

Teams developing agentic applications can benchmark the model-serving layer that supports multi-step workflows.

What Developers Should Know

AIPerf is not a model optimizer by itself.

It does not automatically make an LLM faster.

Instead, it provides measurement infrastructure that helps engineers understand where performance problems exist.

The value comes from reproducible testing.

A benchmark can be run with a defined request rate, input distribution, output distribution and random seed. That makes it possible to repeat the same experiment and compare results after an infrastructure or model change.

That reproducibility is particularly useful when teams are optimizing AI systems over time.

NVIDIA AIPerf arrives at a time when AI inference is becoming as important as model development itself.

The new tool replaces GenAI-Perf with a redesigned architecture focused on high-concurrency benchmarking, broader workload support and realistic traffic simulation.

Its support for 15-plus endpoint types, trace replay, configurable traffic patterns, distributed testing and GPU telemetry makes it more than a basic response-time checker.

The most useful part may be its focus on understanding performance under conditions that resemble production rather than relying only on idealized single-user tests.

As AI applications move toward larger workloads and more complex agents, developers need to know not just whether a model can answer a request, but how the entire inference system behaves when real traffic arrives.

AIPerf is built around that question.