Graph-based deep learning on AMD GPUs: building hipDNN’s normalization engine

Modern AI frameworks think in computation graphs, so the GPU libraries underneath them need to understand graphs too. AMD asked us to help accelerate hipDNN, its new graph-based deep-learning library for ROCm. We contributed a HIP-native engine for normalization operations and cut the time its test suite spends on the CPU by moving reference implementations to the GPU.

About the customer

AMD makes the GPUs that compete with NVIDIA’s, and builds ROCm, the open GPU computing platform behind much of today’s HPC and AI software. Our collaboration goes back a decade to when we helped AMD create rocRAND, their GPU random-number library. We also brought SPIR-V support to four of their core libraries. This project took that collaboration into the heart of their AI software stack.

The challenge

Frameworks such as PyTorch, TensorFlow and JAX get much of their performance from GPU libraries beneath them. On NVIDIA GPUs, that library is cuDNN. Its graph-based design lets frameworks describe multiple operations together, so the library can fuse them and pick the best execution plan.

AMD’s existing library, MIOpen, exposes individual operations through a C API. Frameworks that increasingly reason in graphs cannot use that integration model efficiently, so AMD is building hipDNN, a graph-based library modeled after cuDNN. But a new library is only as good as the operations it supports: hipDNN did not yet cover the normalization and related operations that AI models rely on, which are BatchNorm, LayerNorm, RMSNorm and Resampling. AMD asked us to help close that gap and, at the same time, tackle the other thing that makes or breaks a new library: the speed of its test suite.

Our approach

We worked on two fronts at once. Both served the same goal: making hipDNN a mature foundation that frameworks like PyTorch and TensorFlow can build on.

Native support for operations that matter. Normalization appears in every transformer layer, so a library that lacks it forces frameworks onto slower paths. For AMD, a gap here would translate directly into weaker performance for the LLMs and vision-language models that their customers run. We had recently ported these exact kernels within AMD’s MIOpen library: proven code, proven team, low risk. The new engine compiles kernels at runtime, so every AMD GPU generation is supported without new implementations, which means less maintenance and no library gaps when a new chip ships.

Testing that developers actually run. HipDNN verifies every operation against a reference implementation, and those references previously ran on the CPU. This is slow for large tensors, and slow tests cost more than CI minutes: they delay every pull request and stretch release cycles. We moved both the references and the test-data generation to the GPU.

A technical blog post with the full details accompanies this case study and will be linked here once it’s published.

The result

HipDNN now has a HIP-native engine covering BatchNorm, LayerNorm, RMSNorm and Resampling. Other engines can pick up these operations through the same graph-based interface.

The test suite got faster too. As an example, the BatchNorm forward-inference tests GPU references got the mean runtime cut from 177.4 to 152.3 seconds. That is a 1.16x improvement, measured on an AMD Instinct MI210 with ROCm 10.1. The full test suite was sped up by 1.28x to 1.75x across devices.

Our work helped accelerate ROCm’s transition to a graph-based C++ deep-learning library. HipDNN is a more practical foundation for framework integration on AMD GPUs, and a stronger base for the whole ecosystem to build on.

Key takeaway

A modern library interface is only half the story. The other half is what runs underneath: native backend engines that compile graphs efficiently and test infrastructure fast enough that developers actually run it. By delivering both, we helped AMD grow hipDNN from a promising design into a practical foundation for the ROCm ecosystem.

Did you enjoy reading about how we helped AMD build hipDNN’s normalization engine and GPU-accelerated test suite? You can read about more projects we worked on by checking out our other case studies.

Leave a Comment

Your email address will not be published. Required fields are marked *