Porting GROMACS from CUDA to OpenCL: giving researchers a choice of hardware

GROMACS is one of the most used molecular dynamics packages in the world, running simulations of proteins, polymers and other soft matter on supercomputers and GPU clusters. In 2014 it could only use NVIDIA GPUs, because all its GPU code was written in CUDA. Research groups that had invested in other hardware could not use it at full speed. In this case study we describe how we ported GROMACS from CUDA to OpenCL in several months and what that taught us about large CUDA ports.

About the software

GROMACS does soft matter simulations on molecular scale

GROMACS is an open source molecular dynamics simulation kit, used by research groups and companies worldwide. It simulates molecules over time: proteins folding, polymers stretching, nanotubes forming. These simulations are extremely compute-hungry, so GROMACS runs its heaviest computation on GPUs.

In 2014, all of that GPU code was written in CUDA, which only runs on NVIDIA hardware. For an open source project with a world-wide user base, that was a real limitation: the software had a single hardware supplier, whether its users liked it or not.

The challenge

Port a large, actively developed CUDA codebase to OpenCL, without breaking existing users, and bring it to a state where the GROMACS project could take over maintenance. Our team was small, and the deadline was set by an external event: we wanted to show a working demo at SC14 in New Orleans, five months away from the moment of decision.

Three things made this project hard:

  • CUDA is not just a language, it is an ecosystem. The code did not only use CUDA syntax, it used NVIDIA tooling, libraries and assumptions about the hardware. A port has to replace all of that, not just translate syntax.
  • The project moved while we ported. GROMACS was under active development, so we had to keep our port in sync with an upstream that did not stand still.
  • Correctness before speed. Molecular dynamics produces numbers that researchers publish. Wrong results are worse than slow results, so every ported kernel needed verification against the CUDA implementation. We needed to not only port the code, but also get expertise in the subject of molecular dynamics, and understand all related design choices of GROMACS.

Our approach

We did this project pro bono and open source, on purpose. Every consultancy project before it was under NDA and could not be shown to anyone. A public port of a well-known scientific package would demonstrate exactly what we could do, with full transparency, to the research community we wanted to work with. We also wanted to show how powerful OpenCL can be, and what we could do with it. Below we describe how we organized the work.

Working with the GROMACS team from day one

We did not port in secret and hope for acceptance. From the start we worked with the GROMACS developers at KTH Royal Institute of Technology. Szilárd Páll, Mark Abraham and Berk Hess answered all our questions, reviewed our direction and helped us understand the design decisions behind the CUDA code. AMD supported the project by helping us tackle driver-level obstacles.

This mattered strategically. A port that the upstream project does not want becomes a fork, and forks die. A port built together has a path into the official release.

A phased plan: correctness first, speed second

We split the port into phases. Phase 1 was to get all failing kernels working correctly on both NVIDIA and AMD hardware. Only after correctness would we move to phase 2: optimizing for speed on specific devices and adding features such as double precision support. This kept the project reviewable: every step was small enough to verify against the CUDA reference implementation.

Open source as a recruiting and training tool

The port was our first product and our first large open source effort. We used it to gain experience with distributed, open development. We also invited external developers to contribute, with the explicit promise that strong contributors could end up with a job proposal. One volunteer, Dimitrios Karkoulis, contributed from the very start in his free time. Several of our own engineers, including team lead Anca Hamuraru and Teemu Virolainen, grew into their roles on this project.

The result

The OpenCL backend got merged into the main GROMACS release and was supported for years. It was eventually deprecated and removed once the GROMACS project chose SYCL and HIP as its portable backends. It shows the port served users for years, and that the ecosystem moved on. The code is available under GROMACS’s own open source license, so the community could use it, inspect it and build on it for free.

Key takeaway

A large CUDA port is not a translation exercise, it is a collaboration with the upstream project. By porting GROMACS to OpenCL together with its developers, in the open, we gave thousands of research groups a free choice of GPU vendor, and built the expertise that we now bring to every CUDA port we do.

At Stream HPC we work with all big GPU languages, which makes it possible to write correct and performant software cross APIs and hardware.

Did you enjoy reading about how we ported such complex software to OpenCL? You can read about more projects we worked on by checking out our other case studies.