We ported molecular dynamics software (AMBER, GROMACS, OpenMM) from CUDA to AMD’s HIP framework for LUMI and FRONTIER supercomputers. All software achieved expected performance or better on AMD MI250X GPUs. We share some details about what we did for AMBER and OpenMM.
Introduction
For years, Nvidia dominated the supercomputing market. However, AMD gained significant traction with its MI100 and MI250 GPUs. By 2021, AMD GPUs began powering half of the world’s supercomputers, with major wins in 2022. Two of the most prominent supercomputers requiring AMD GPU support were:
- LUMI (Finland, EU)
- FRONTIER (Tennessee, USA)
From 2020 to early 2023, we ported and optimized the following molecular dynamics and HPC software for AMD MI250X:
- AMBER
- OpenMM
- GROMACS
- NAMD
- VkFFT
- OpenFoam
- PETSc
- PENNANT
While AMD often receives credit for the porting work, our team played a key role in ensuring compatibility and performance.
About the Customers
We collaborated closely with AMD, who managed communications with the supercomputing centers.
LUMI
- Location: Finland
- Ranking: #3 in the Top500 (2022–2023)
- Hardware:
- HPE Cray EX235a
- AMD Optimized 3rd Gen EPYC 64C 2GHz
- AMD Instinct MI250X
- Slingshot-11 interconnect
- Project Lead: EuroHPC/CSC
FRONTIER
- Location: Tennessee, USA
- Ranking: #1 in the Top500 (2022–2024)
- Hardware:
- HPE Cray EX235a
- AMD Optimized 3rd Gen EPYC 64C 2GHz
- AMD Instinct MI250X
- Slingshot-11 interconnect
- Project Lead: DOE/SC/Oak Ridge National Laboratory
Other supercomputers like HPC6, HPC7, and El Capitan also adopted AMD GPUs, marking a significant milestone in AMD’s re-entry into the supercomputing market.
The Challenge
AMD needed HPC software support for the MI250X to secure deals with supercomputing centers. Under tight deadlines, they requested our assistance, particularly for molecular dynamics software. Given our expertise with porting GROMACS to OpenCL, we began with this type of software.
Our Approach
This was a large-scale project, and while we can’t share every detail, we’ve selected two key highlights:
AMBER
AMBER’s only GPU implementation was for CUDA. A straightforward HIPification (converting CUDA to HIP for AMD GPUs) wasn’t possible due to a critical difference:
- CUDA assumes a warp size of 32.
- AMD GPUs (GCN/CDNA) require a warp size of 64.
Parts of the code, such as non-bonded forces calculation and neighbor list construction, were hardcoded for CUDA’s warp size. We deciphered these problematic kernels, so we could generalize or recreate them for AMD GPUs without hardcoding assumptions.
OpenMM
AMD developers initially performed the HIPification, but we assisted with performance optimization. Again, the warp size difference (64 vs. 32) posed challenges. While less hardcoding existed compared to AMBER, we still made significant changes to improve performance by enabling a warp to process 2 tiles at a time.
Results:
- All correctness tests passed for CDNA and RDNA GPUs on Linux and Windows.
- Some optimizations even improved performance on competing platforms (e.g. PME calculations). You’re welcome.
- The HIP platform was initially a separate plugin but was later integrated into OpenMM 8.2 as a standard feature.
The Outcome
All molecular dynamics software achieved expected performance levels on the MI250X, leading to final acceptance by LUMI and FRONTIER.
As we went through all the code, but had limited time, we worked mostly on low-hanging fruit. More optimizations are possible in all mentioned software.
Key Takeaway
Direct porting CUDA software to AMD GPUs is often straightforward, using the available tools. However:
- Complex software requires special attention and more work, as results can be wrong.
- Performance optimization is not free, and demands additional engineering.
Did you enjoy reading about how we ported large-scale molecular dynamics software to supercomputers powered by modern AMD GPUs? You can read about more projects we worked on by checking out our other case studies.