NVIDIA CUDA: The Engine Behind Modern AI

What CUDA actually is

CUDA (Compute Unified Device Architecture) is NVIDIA's parallel computing platform, first released in 2006. It combines a hardware architecture, a programming model, and a suite of libraries that together let developers run general-purpose code on a GPU rather than a CPU. Before CUDA, using a GPU for anything other than graphics meant shoehorning your problem into graphics-specific APIs, which was neither pleasant nor particularly tractable. CUDA changed that by giving developers a direct path to the thousands of cores on a GPU through code that looks, at least superficially, like C.

The reason CUDA became the default platform for AI research and production is straightforward: NVIDIA built it, supported it well, and got there first. PyTorch, TensorFlow, JAX, and virtually every major ML framework depend on CUDA. When you call loss.backward() in PyTorch, the gradient computation runs on a CUDA kernel. The framework makes this invisible, but the dependency is total.

The programming model

The central concept in CUDA is the kernel: a function that runs on the GPU and executes simultaneously across many threads. Those threads are organised into blocks, and blocks into a grid. Each thread knows its own position within its block and the grid, and uses that position to determine which piece of data it is responsible for. A matrix multiplication might assign one thread per output element, with thousands of threads computing their assigned element at the same time.

The GPU executes threads in groups of 32 called warps, and all threads in a warp execute the same instruction at the same time. This is the Single Instruction, Multiple Threads model (SIMT), and it is worth understanding what it actually implies. SIMT differs from classical SIMD (Single Instruction, Multiple Data) in that each thread has its own program counter and register state, so threads within a warp can technically branch independently. The cost is that when threads in the same warp diverge (one thread takes an if branch that others do not), the warp executes both paths serially, masking off threads not taking each path. This is called warp divergence, and it is one of the primary ways a naively written kernel loses most of its theoretical throughput.

Tiling and compute shaders

Tiling is the single most important optimisation pattern in GPU computing, and it is what allows cuBLAS and cuDNN to achieve performance close to the hardware's theoretical peak. The idea is to partition a large matrix into smaller sub-matrices (tiles) that fit in shared memory. A block of threads cooperates to load one tile of the input into shared memory, performs the computation for that tile, then moves to the next. Shared memory sits on-chip and is roughly 30 times faster than global memory, so keeping the arithmetic there rather than returning to global memory for every element access makes a substantial difference. An H100 has 228KB of shared memory per streaming multiprocessor, which sounds modest, but it is enough to hold the working set for a tile-sized chunk and cycle through the full computation without thrashing global memory.

Compute shaders are the cross-vendor alternative, rooted in the graphics world and still very much in use. Where CUDA targets NVIDIA hardware specifically, compute shaders run on any GPU through the standard graphics APIs: OpenGL, Vulkan, DirectX, Metal, and (in the browser) WebGPU. The trade-off is portability against capability. CUDA exposes NVIDIA hardware features (warp-level primitives, tensor cores, specific memory types) that the graphics APIs simply cannot reach. For research and training, CUDA is the realistic choice. For browser-based inference or cross-vendor deployment, compute shaders and WebGPU are the practical alternative.

Memory is the bottleneck

The GPU memory hierarchy has three tiers. Registers are per-thread and the fastest. Shared memory is on-chip, shared between threads in the same block, and roughly 30 times faster than global memory. Global memory is the full pool on the card (16GB to 80GB on current hardware) and relatively slow to access. Writing a performant CUDA kernel means staging data through shared memory to reduce global memory traffic, keeping registers occupied, and ensuring threads in the same warp access contiguous memory addresses, a pattern called coalesced access.

For AI workloads, the practical implication is that moving data between CPU and GPU is expensive, and you want to keep it on the GPU as long as possible. A training loop that copies tensors to CPU between batches for logging or evaluation stalls the GPU while it waits. The correct pattern is to keep computation on-device and transfer only the minimum back to host: scalar loss values, checkpoint weights.

The library stack

Most engineers using AI frameworks never write raw CUDA kernels. They call into NVIDIA's libraries, which are optimised to a degree that would take a team years to replicate. The core of the compute stack is cuBLAS for linear algebra and cuDNN for deep learning primitives (convolution, batch normalisation, activation functions, attention, pooling). These two handle the arithmetic of nearly all neural network training and inference. NCCL (NVIDIA Collective Communications Library) handles multi-GPU and multi-node communication: all-reduce, broadcast, gather, scatter. It is what makes distributed training across a cluster scale as well as it does.

What is less often discussed is how far the stack extends beyond those names. CUTLASS provides lower-level template building blocks for custom high-performance matrix operations in C++, for cases where the standard libraries do not match the shape of your problem. Triton, developed by OpenAI, sits at a more approachable level and lets you write GPU kernels in Python that compile down to PTX (NVIDIA's low-level instruction set), without writing CUDA C directly. PyTorch uses Triton for its custom fused kernels, and it has made writing performant custom operators substantially more accessible than it was.

Deeper in the stack: cuSPARSE handles sparse matrix operations, which matter as pruned models become more common for inference. cuFFT provides GPU-accelerated Fast Fourier Transforms. cuRAND generates random numbers on-device, avoiding the CPU round-trip that would otherwise stall stochastic operations during training. TensorRT is NVIDIA's inference optimisation platform: it takes a trained model, applies graph optimisations and kernel fusions, calibrates for quantisation, and produces a deployment-optimised engine that typically runs 2 to 5 times faster than the original. For production inference on NVIDIA hardware, TensorRT is where most serious deployments end up. Nsight (specifically Nsight Systems for system-level timelines and Nsight Compute for kernel-level analysis) is the profiling suite you reach for when a kernel performs worse than expected and you need to understand exactly why.

When you need to engage with it directly

If you are using PyTorch or TensorFlow for standard training and inference, you are already using CUDA. The frameworks handle kernel dispatch, memory management and type casting. You need to engage with CUDA directly when writing custom operators (a novel attention mechanism, a fused kernel to replace a sequence of slower operations, a quantisation scheme the standard libraries do not yet support), or when debugging performance: understanding whether your training job is memory-bandwidth bound rather than compute-bound, or identifying unintentional host-device transfers mid-loop, requires knowing what is happening underneath the abstraction.

NVIDIA's position here is not easy to displace. ROCm, AMD's open-source alternative, has improved substantially in recent years, but the weight of tooling, documentation, pre-built libraries and accumulated community knowledge still tilts heavily towards CUDA. For teams building serious AI systems today, CUDA is less a technology choice than a given. Understanding it at the level of the programming model and memory hierarchy makes you a better user of the frameworks that sit on top of it.