It’s one of the age-old problems of software development: You’ve started a new project, set up fancy CI infrastructure, and everything works well. But as time goes on, you add more and more features to the project. Compile-times on your local checkout of the projects are mostly fine as you are used to incremental compilation, and full recompilations are only occasionally required. But your CI pipelines compile the project from scratch every time, and they start to take while. And then a long time. And then a really long time.
So too was the problem for rocPRIM, a ROCm library that provides GPU-accelerated parallel primitives, written in AMD’s HIP programming language. While rocPRIM itself is written as a header-only library due to the many templated primitives it exposes, tests and benchmarks create a large number of different instantiations of parallel primitives that blow up the compile times. Let’s look at some ways to improve the compile times of such HIP projects.
Gathering Baselines
As any GPU programmer knows, the first step in the optimization process is gathering baselines. HIP is a C++ dialect, and so rocPRIM the build process is defined using CMake. We will measure the length of the build process simply by using the time command built into Bash.
The set of GPUs that should be supported by HIP projects is typically supplied ahead of time by passing it to CMake, and the same is true for rocPRIM. Additionally, for CI for GPU-projects in general, we’d like to split building and testing so that we don’t need to attach a GPU to our build servers. We’ll build the library, benchmarks, and tests all at once for all the different GPU architectures that we wish to support so that we don’t need to recompile the (host) code every time. For the purposes of this blog post we’ll choose a hand full of common architectures, ranging from RDNA2 though 4 and from CDNA1 through 3.
The easiest way to improve build times is by using a faster computer and so we’ll choose to compile our baseline on a server with an AMD Epyc 9575F processor with 64 cores and 128 threads, with Ubuntu 24.04 and ROCm 7.2.4 installed. Furthermore, we configure CMake to use the Ninja build tool, which automatically distributes the compilation tasks across cpu.threads + 2processes.
$ # Configure a rocPRIM build
$ # rocPRIM is checked out at 086911ee.
$ mkdir build && cd build
$ cmake ~/rocm-libraries/projects/rocprim -DCMAKE_CXX_COMPILER=amdclang++ \
-GNinja -DBUILD_TEST=ON -DBUILD_BENCHMARK=ON -DCMAKE_BUILD_TYPE=Release \
-DAMDGPU_TARGETS="gfx1030;gfx1100;gfx1151;gfx1201;gfx908;gfx90a;gfx942"
$ time ninja
real 25m39.363s
user 633m3.332s
sys 6m49.851s
$Profiling CMake Builds
25 minutes is not actually that bad if you’re the patient type, but lets see if this can be improved. The next step is to gather some more detailed information. By default, Ninja produces a log file (.ninja_log) that contains detailed information about which file was processed, when, and how long it took. While the file itself is not very good at giving a good overview of the build process, it can be post-processed with a tool like ninjatracing to convert it into a format that can be opened with Perfetto.
This already highlights one big problem: Our builds suffer from a very strong tail distribution. Even though we were compiling with 130 threads, most objects finished compiling in only a few minutes, while the total build time is governed by a few objects that have very long compile times!
One simple way to improve the build performance in the context of CI pipelines is actually to compile with fewer cores: Assuming that the objects with long compile times get processed first, the remaining cores can then distribute the objects with shorter compile times, and ideally finish around the same time. This way, the rest of the machine is available for example for running multiple build processes in parallel.
Background: The HIP Compilation Process
HIP executables are actually multiple binaries in a rain coat: When compiling HIP source code for multiple GPUs, the compiler actually compiles the code for the host and each GPU separately, before putting them together in a multi-architecture binary. This can be verified by running the clang compiler driver in verbose mode, which tells us the entire compilation process of a HIP executable. Note that some output is omitted for brevity.
$ amdclang++ -otest test.hip -v --offload-arch=gfx1201 --offload-arch=gfx942
AMD clang version 22.0.0git ([...])
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm-7.2.0/lib/llvm/bin
Configuration file: /opt/rocm-7.2.0/lib/llvm/bin/clang++.cfg
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/13
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/13
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Candidate multilib: x32;@mx32
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda-13.2, version
Found HIP installation: /opt/rocm-7.2.0/lib/llvm/bin/../../.., version 7.2.26015
"/opt/rocm-7.2.0/lib/llvm/bin/clang-22" -cc1 [...] -fcuda-is-device [...] -target-cpu gfx1201 [...] -o /tmp/test-gfx1201-c41951.o test.hip [...]
"/opt/rocm-7.2.0/lib/llvm/bin/lld" [...] -o /tmp/test-gfx1201-053ec4.out /tmp/test-gfx1201-c41951.o
"/opt/rocm-7.2.0/lib/llvm/bin/clang-22" -cc1 [...] -fcuda-is-device [...] -target-cpu gfx942 [...] -o /tmp/test-gfx942-7f9b8a.o [...] test.hip
"/opt/rocm-7.2.0/lib/llvm/bin/lld" [...] -o /tmp/test-gfx942-b85234.out /tmp/test-gfx942-7f9b8a.o [...]
"/opt/rocm-7.2.0/lib/llvm/bin/clang-offload-bundler" [...] -input=/tmp/test-gfx1201-053ec4.out -input=/tmp/test-gfx942-b85234.out -output=/tmp/test-fc9973.hipfb [...]
"/opt/rocm-7.2.0/lib/llvm/bin/clang-22" -cc1 [...] -target-cpu x86_64 [...] -fcuda-include-gpubinary /tmp/test-fc9973.hipfb [...] -o /tmp/test-5e7755.o [...] test.hip
"/opt/rocm-7.2.0/lib/llvm/bin/ld.lld" [...] -o test [...] /tmp/test-5e7755.o [...] -lamdhip64 -lstdc++To summarize, the compilation process basically looks like this:
- Compile (
-cc1, the internal Clang C++ compiler) the device code (-fcuda-is-device) fromtest.hipfor RDNA4 (-target-cpu gfx1201). Note that while clang passes a cuda-specific flag, Clang knows how to deal with this flag for HIP as well. - Link the device code object into an ELF file using lld.
- Repeat the previous two steps for CDNA3 (
-target-cpu gfx942). - Pack the ELF files for the different GPU architectures into an offload bundle (
/tmp/test-fc9973.hipfb– “hip fat binary”). This is an archive format that bundles the ELF files for all architectures. More information about this file format is available on the Clang Offload Bundler documentation page. - Compile
test.hipfor the host (-target-cpu x86_64) and embed the offload bundle (-fcuda-include-gpubinary /tmp/test/fc9973.hipfb). - Finally, link the object file produced in the previous step into a proper executable using lld.
Here we can also easily tell the the difference between a compiler driver, amdclang++, and the C++ compiler: The compiler driver is responsible for invoking the different tools required to build a functioning executable, while the actual C++ compiler (cc1) is responsible for compiling C++ source files into object files. Crucially, when building a multi-architecture HIP executable with Clang, it splits out the compilation process over separate compilations for the host and the different device architectures.
Parallelizing Builds
As any GPU programmer knows, the best solution to optimize something is obviously to parallelize it. Fortunately, as we’ve seen in the previous section, compiling HIP objects requires a mostly independent compilation step for each architecture that we wish to support. Those steps can be split out into separate processes and run concurrently. By chopping the total work into smaller pieces, the files that take a long time to compile can be distributed to cores that would otherwise be idling.
An easy way to do this is using the --offload-jobs=<N> option, a relatively recent addition to Clang. As the name suggests, this attempts to split up the compilation across <N> separate tasks that can be executed simultaneously. There are still some caveats, though.
First, this requires using the “new” offload compiler driver. As of writing, the latest version of ROCm is 7.2.4, and it does not have the new offload driver is not enabled by default. It can be manually enabled for now by passing --offload-new-driver to the compiler. The new offload driver has been made the default in unstable versions of ROCm, though, and so passing this flag manually will no longer be required in a ROCm version soon to come. Using the new offload driver changes the compilation process slightly: In Clang, compilation is split out between a frontend pass, which compiles C++ into LLVM intermediate representation (IR), and a backend pass, which compiles LLVM IR into AMD GPU machine code. These passes are normally run in the same process, but --offload-new-driver splits this up into separate processes. In fact, all files related to the backend compilation are now passed to a new tool, the clang-linker-wrapper, and compiled into AMDGPU machine code. It is precisely this tool that can benefit from --offload-jobs.
That brings us to the second caveat: Only the backend compilation can be parallelized using the new offload driver. That is unfortunately a limitation in the software architecture of the Clang compiler driver, or so I’m told, and part of the reason for moving into the new flow is to at least facilitate parallelization of the backend compiler. Fortunately, the backend makes up a significant part of the total compile time, and so this is still likely to help a bit.
A final problem with --offload-jobs=<N> is that a constant value for <N> conflicts with our build system. If we let Ninja use the default cpu.threads + 2 threads to compile the project, and use <N> threads for HIP compilation passes, then there will be way more active processes than that there are CPU cores in the machine. That leads to trashing, which makes the overall compilation less efficient.
This can be resolved using another recent addition to Clang, GNU Make Jobserver integration for the new compiler driver. The GNU Make Jobserver lets processes coordinate distribution over the machine, even if one of those processes wants to launch several additional sub-processes. For this to work properly, Ninja also needs to be aware of the jobserver. This was a longstanding requested feature in Ninja, and has recently finally been added and released in Ninja 1.13. Ubuntu 24.04 only includes Ninja 1.12, though, so we have to get it elsewhere. Additionally, this requires an actual GNU Make Jobserver. The Ninja source also includes a minimal implementation that works fine for our purposes.
Let see it in action. First, set up a newer version of Ninja. Since we’ll need the source for the actual job server anyway, lets just compile it from scratch.
# Compile a more recent version of Ninja (or get it via your favorite package manager)
$ git clone https://github.com/ninja-build/ninja -b v1.13.0
$ mkdir ninja/build && cd ninja/build
$ cmake .. -GNinja -DBUILD_TESTING=OFF -DCMAKE_INSTALL_PREFIX=install -DCMAKE_BUILD_TYPE=Release
$ ninja install
$ export PATH=$(PWD)/install/bin:$PATH
$ ninja --version
1.13.0Next, reconfigure rocPRIM so that CMake picks up on the new version of Ninja, and rebuild using the minimal jobserver.
$ cmake ~/rocm-libraries/projects/rocprim -DCMAKE_CXX_COMPILER=amdclang++ \
-GNinja -DBUILD_TEST=ON -DBUILD_BENCHMARK=ON -DCMAKE_BUILD_TYPE=Release \
-DAMDGPU_TARGETS="gfx1030;gfx1100;gfx1151;gfx1201;gfx908;gfx90a;gfx942" \
-DCMAKE_CXX_FLAGS="--offload-new-driver --offload-jobs=jobserver"
$ time ~/ninja/misc/jobserver_pool.py ninja
ninja: Jobserver mode detected: -j128 --jobserver-auth=fifo:/home/[...]/build/jobserver_pool
real 14m19.541s
user 655m57.911s
sys 9m58.834s
$A 45% reduction in compile time is not bad! Lets see how much further we can take it.
Further Parallelizing Builds
If we investigate the ninjatracing output of the build with the jobserver enabled, we can see that the distribution of compile times for the individual objects is somewhat flatter when compared to the original version, but there is still room for improvement.
Fortunately there is still some parallelization left to exploit: The frontend of the compilation. As mentioned before, Clang’s compiler driver cannot currently parallelize the compilation of HIP-source to LLVM IR, even though the compilation for each device architecture is independent of eachother. We can get around that by writing our own compiler driver which separates the compilation into completely different processes.
This is easier than it sounds: We just have to pre-process the compiler commandline, split it out into separate calls that each have a single --offload-arch= set and only compile the device code using --offload-device-only. After running those in parallel, the offload bundle can be manually constructed using clang-offload-bundler. Next, the host code is compiled using --offload-host-only. The offload bundle containing the GPU code is included in the host compilation by passing -fcuda-include-gpubinaryto cc1 using -Xclang.
Some other details apply: We have to help Clang relate device and host compilations that belong to the same compilation units by passing a “CUID”, a random hex string of 8 characters, to each compilation. Also, by using --offload-host-only the compiler puts each individual GPU binary into an offload bundle for us already, but we just want to have the raw ELF file when putting the combined offload bundle together. The compiler can be instructed to not create offload bundles for us by passing --no-gpu-bundle-output. Finally, we have to pass on the commands related to compression of the offload bundle, --offload-compress and --offload-compression-level. The compiler normally does this for us, but since we invoke the offload bundler manually, we have to do that now.
Preprocessing the compiler commandline with CMake based build scripts can easily be done by passing CMAKE_CXX_COMPILER_LAUNCHER to CMake. Originally intended for tools like ccache, the command passed to this option is prepended to the compiler command line, and is supposed to invoke the compiler on behalf of Ninja. It can then modify the command line as described above and invoke sub-compilations using for example python’s subprocess API.
By integrating this script with the jobserver, the compilation tasks can again be cleanly distributed across the cores. One caveat here is that the compilation must make progress even if no new slots can be acquired from the job server, as there is always one implicit slot allocated by the parent process.
The source code for this script, phc.py (for Parallel HIP Compiler), can be found here: https://github.com/StreamHPC/phc.
Lets see it in action. Note that because the entire device compilation is now parallelized, without outputting a separate IR object, we don’t have to use the new offload compiler driver anymore.
$ cmake ~/rocm-libraries/projects/rocprim -DCMAKE_CXX_COMPILER=amdclang++ \
-GNinja -DBUILD_TEST=ON -DBUILD_BENCHMARK=ON -DCMAKE_BUILD_TYPE=Release \
-DAMDGPU_TARGETS="gfx1030;gfx1100;gfx1151;gfx1201;gfx908;gfx90a;gfx942" \
-DCMAKE_CXX_COMPILER_LAUNCHER=$HOME/phc/phc.py
$ time ~/ninja/misc/jobserver_pool.py ninja
ninja: Jobserver mode detected: -j128 --jobserver-auth=fifo:/home/[...]/build/jobserver_pool
real 9m36.538s
user 768m4.752s
sys 8m9.907s
$Another 33% reduction in compile time, for a 63% reduction in total!
Caching
Another great way to improve build speed is by object file caching. This method stores object files into a cache after building them, indexed by a key. The key is typically a hash derived from source files, include files, compiler binaries, compiler command line, and environment variables. When an object file is being compiled, the cache is checked first. If there is a hit, the already compiled object file is returned immediately. Otherwise, the compilation is executed for real, and the result is stored in the cache before returning. Caching is in general very effective when both changes are small and where incremental builds aren’t available, often the case with CI pipelines for example.
Sccache is a compiler caching tool that works pretty much out of the box for many ROCm projects, including rocPRIM. One of its primary aspect is the shared caching behavior (sccache stands for shared c cache after all), which lets users or CI pipelines easily share the cache via for example S3, Memcached, or Redis. There are still some caveats to keep in mind though.
First, in order to get a cache hit, the source and build directories must match. That means that if you are sharing a cache with your coworkers, everybody should use the same directory for the source and build. Second, sccache starts a background service when first launched, but by default it shuts down after a timeout of 10 minutes. As we’ve seen before, the rocPRIM object build times are very skewed, and this can actually cause sccache’s background server to shut down during the build. Set SCCACHE_IDLE_TIMEOUT=0 to disable sccache from doing that. Third, Clang’s --offload-compress flag does not work well with sccache at the time of writing (issue), and so we have to disable it for rocPRIM by passing -DBUILD_OFFLOAD_COMPRESS=OFF to CMake when configuring the project.
Ubuntu 24.04 only includes a very old version of sccache that does not yet include HIP support. Since then there were also some bugs that we’ve fixed, so its best to just get the latest version from github. Sccache integration is enabled via the same CMAKE_<lang>_COMPILER_LAUNCHER CMake options as the script used in the previous section, so it’s relatively straightforward to set up.
$ # Download sccache and extract from GitHub's release artifacts.
$ SCCACHE_VERSION=0.16.0
$ curl -L \
https://github.com/mozilla/sccache/releases/download/v$SCCACHE_VERSION/sccache-v$SCCACHE_VERSION-x86_64-unknown-linux-musl.tar.gz \
| tar -zxf - --wildcards --strip-components 1 '*/sccache'
$ export SCCACHE_IDLE_TIMEOUT=0
$ cmake ~/rocm-libraries/projects/rocprim -DCMAKE_CXX_COMPILER=amdclang++ \
-GNinja -DBUILD_TEST=ON -DBUILD_BENCHMARK=ON -DCMAKE_BUILD_TYPE=Release \
-DAMDGPU_TARGETS="gfx1030;gfx1100;gfx1151;gfx1201;gfx908;gfx90a;gfx942" \
-DCMAKE_CXX_COMPILER_LAUNCHER=$PWD/sccache \
-DBUILD_OFFLOAD_COMPRESS=OFF
$ ninja
real 26m49.064s
user 0m5.489s
sys 0m19.882s
$ # Now try again with a warmed up cache
$ # Simulate applying a change to the library
$ echo 'struct Test{};' >> ~/rocm-libraries/projects/rocprim/rocprim/include/rocprim/warp/warp_sort.hpp
$ ninja clean
$ time ninja
real 5m40.425s
user 0m6.292s
sys 0m19.491s
$ # Simulate a fresh pipeline with no changes to code
$ ninja clean
$ time ninja
real 0m4.470s
user 0m6.203s
sys 0m20.362sOf course, with a header only library that has many tests and benchmarks, changes will often cascade into many objects that have to be rebuilt. On the other hand, changes that only affect single benchmarks, tests, or documentation, are much faster to build, on the order of seconds. Your mileage may vary.
Parallelization + Caching
The final caveat concerns caching and parallelizing at the same time. CMake only allows specifying a single CMAKE_<lang>_COMPILER_LAUNCHER, so we can’t use these these tools out of the box. Resolving this is unfortunately not as simple as writing a script that executes sccache ~/phc.py $@, as sccache attempts to invoke the compiler by itself to figure out include paths. When invoking sccache ~/phc.py amdclang++ ..., sccache assumes that ~/phc.py is the compiler instead of amdclang++, and attempts to invoke ~/phc.py -E -v -. The easiest way to work around this issue is by writing a wrapper script that interacts with phc.py to rewrite the command line to fool Sccache. This wrapper script is then used as the CMake compiler launcher.
It works as follows: In our wrapper script, we rewrite the command line from amdclang++ ... to sccache phc.py .... The real compiler is passed from the wrapper script to phc.py via an environment variable, and phc.py prepends it to the command line if this environment variable is set. Note that we’ll also add the real compiler as well as phc.py to the Sccache cache key by setting SCCACHE_EXTRAFILES to include these files, so that any cached objects will be properly invalidated if we change the compiler binary or phc.py script.
#!/usr/bin/env bash
# Pass the actual compiler to PHC via an environment variable
export PHC_COMPILER=$(which $1)
# Make sure that sccache includes the real compiler in the cache hash.
# Also add the scripts that we've written for good measure.
export SCCACHE_EXTRAFILES="$PHC_COMPILER:$HOME/phc.py"
# We no longer need the real compiler now that we're passing it
# explicitly, so discard it.
shift
# Invoke sccache now.
# Assumes that sccache is on the PATH!
exec sccache ~/phc.py $@Next, we’ll add in the compiler into the compiler command line in phc.py if its passed explicitly:
# Extract arguments. Note: Skip the script name.
cmd = sys.argv[1:]
# Add in the compiler if we've passed it explicitly
if compiler := os.environ.get('PHC_COMPILER'):
cmd.insert(0, compiler)We can repeat the previous test now, but with both parallelizing and caching enabled. Note that this time, we don’t have to start our own jobserver using Ninja’s jobserver_pool.py: Compilation tasks are redistributed by sccache, which starts its own background service by default. This service also acts as a GNU Make Jobserver, which integrates with phc.py basically automatically.
$ export SCCACHE_IDLE_TIMEOUT=0
$ cmake ~/rocm-libraries/projects/rocprim -DCMAKE_CXX_COMPILER=amdclang++ \
-GNinja -DBUILD_TEST=ON -DBUILD_BENCHMARK=ON -DCMAKE_BUILD_TYPE=Release \
-DAMDGPU_TARGETS="gfx1030;gfx1100;gfx1151;gfx1201;gfx908;gfx90a;gfx942" \
-DCMAKE_CXX_COMPILER_LAUNCHER=$HOME/phc/phc_sccache.sh \
-DBUILD_OFFLOAD_COMPRESS=OFF
$ time ninja
real 10m18.873s
user 0m7.964s
sys 0m16.262s
$ # Now try again with a warmed up cache
$ # Simulate applying a change to the library
$ echo 'struct Test1{};' >> ~/rocm-libraries/projects/rocprim/rocprim/include/rocprim/warp/warp_sort.hpp
$ ninja clean
$ time ninja
real 1m17.099s
user 0m6.429s
sys 0m21.714s
$ # Simulate a fresh pipeline with no changes to code
$ ninja clean
$ time ninja
real 0m4.554s
user 0m6.634s
sys 0m19.950sConclusion
In this article we’ve discussed various ways to improve HIP compilation throughput for clean builds specifically. These are effective mainly in situations where the compilation times of different object files in a project are very skewed. In this situation, the total compilation process has a long tail where few cores are processing slow-to-compile objects for a long time, while the rest of the machine is idling. By splitting these tasks up into smaller bits, better utilization can be achieved, leading to a better overall throughput.
By using the latest features of Clang and Ninja, we can easily parallelize the backend compilation process, where the compiler converts from LLVM intermediate representation to AMD GPU machine code. This is relatively easy to set up, and already leads to a reduction of about 45% in wall clock compile time for rocPRIM.
This reduction can be improved to about 63% less than the original compile time by also splitting out the front end of the compilation, where the compiler converts HIP source into LLVM intermediate representation. Setting this up is more involved because it requires emulating the Clang compiler driver and interaction with the GNU Make Jobserver to manually split out the compilation process into separate processes.
Finally, compilation caching can also be used to speed up the compilation, by storing object files in a cache and retrieving them if the relevant sources, includes, and compiler command-line arguments haven’t changed. With the right setup, caching can even be integrated with parallelization from earlier to provide very fast clean-build compilation times.
