Nvidia has put more detail behind the Rubin GPU, the accelerator half of its coming Vera Rubin platform, and the pitch is aimed squarely at inference systems that must crank out generated tokens quickly and cheaply. The company says Vera Rubin systems are due later this year, with deliveries set to begin in the fall.
The full Vera Rubin NVL72 rack pairs 36 Vera CPUs with 72 Rubin GPUs. Nvidia’s new disclosures focus on the GPU and on the plumbing around it: how data reaches the tensor cores, how matrix work is scheduled, how attention math avoids becoming the slow bit, and how GPUs talk to one another across NVLink without drowning in coordination traffic.
Rubin packages two compute dies together using Nvidia’s High Bandwidth Interface. Nvidia says the GPU has 224 streaming multiprocessors, 896 tensor cores, and 288GB of HBM4 delivering 22TB/s of memory bandwidth. For inference, the headline number is 50 sparse PFLOPS using Nvidia’s NVFP4 format, though the chip also supports several other formats.
- NVFP4 inference: 50 PFLOPS with sparsity, according to Nvidia.
- NVFP4 training: 35 PFLOPS.
- FP8 and FP6 training: 17.5 PFLOPS.
- FP16 and BF16: 4 PFLOPS.
- FP32: 130 TFLOPS.
- FP64: 33 TFLOPS.
Rubin changes how mixture-of-experts data is described
Nvidia says Rubin updates the Tensor Memory Accelerator, the hardware engine that handles address calculations and loads array data into a GPU’s shared local memory. That matters for mixture-of-experts models, where a router selects only some expert subnetworks for each token instead of activating every parameter.
Those expert weights can be spread across GPUs to fit within available HBM. On Blackwell, Nvidia says the TMA maintained separate descriptors in memory for the location of each expert. As expert counts grow, that metadata work eats more compute.
Rubin’s TMA can maintain and update a unified mixture-of-experts descriptor inside the TMA instruction at runtime, according to Nvidia. The company says that cuts descriptor bookkeeping and leaves more GPU cycles for inference work, which is the expensive part customers are actually renting or buying.
Tensor cores do more work per loop
Nvidia also says Rubin doubles tensor core throughput along the K dimension, the shared inner dimension in matrix multiplication. In plain terms, that dimension affects how many passes the tensor core must make over the inputs before producing the result.
In Nvidia’s example, a matrix operation that takes four loop iterations on Blackwell takes two on Rubin. The company says the change helps kernels limited by throughput, memory, or latency, and applies to both context processing and token decode phases during inference.
Softmax gets faster for long-context attention
Rubin also targets softmax, a core operation in transformer attention. Nvidia says some advanced models now support context lengths up to one million tokens, which makes attention performance a major constraint during inference.
Softmax relies on exponential math handled by the streaming multiprocessor’s Special Function Unit. Nvidia previously increased that path in Blackwell Ultra, which doubled FP32 and BF16/FP16 exponential throughput compared with first-generation Blackwell GB200. Rubin keeps the 2x FP32 gain over Blackwell and raises BF16/FP16 exponential throughput to 4x Blackwell, according to Nvidia.
Kernel scheduling and NVLink coordination get less clumsy
Rubin adds finer-grained dependency handling between producer and consumer kernels. Nvidia’s example is activation generation in an LLM, where one kernel writes data that the next kernel needs. On Blackwell, a later kernel may wait for a broader set of thread-block work to finish. On Rubin, Nvidia says the consumer kernel can start on individual thread blocks once each block’s producer output is ready, reducing kernel-to-kernel delay and improving tensor core occupancy.
At rack scale, Nvidia is also changing NVLink communication. CUDA kernels can initiate GPU-to-GPU communication through Nvidia’s Collective Communications Library, so synchronization overhead becomes part of the performance budget. On Blackwell, Nvidia says a transfer may involve stores, a memory barrier, and an atomic flag. Rubin introduces counted writes, replacing the barrier and atomic operations with a single write counter update on the receiving GPU.
Nvidia says the result is less network traffic, lower latency, and better compute utilization across the rack. Those are company claims until independent systems are shipping and measured, but the design target is clear: keep Rubin’s tensor cores fed while agentic AI workloads spray model weights, key-value cache data, and synchronization messages across a 72-GPU box.
This story draws on original reporting from Tom's Hardware.
