Written by Máté Ferenc Nagy-Egri and Gergely Mészáros

Disclaimer: if you’ve stumbled across this page in search of fixing up the ROCm SDK’s CMake HIP language support on Windows and care only about the fix, please skip to the end of this post to download the patches. If you wish to learn some things about ROCm and CMake, join us for a ride.

Finally, ROCm on Windows

The recent release of the AMD’s ROCm SDK on Windows brings a long awaited rejuvenation of developer tooling for offload APIs. Undoubtedly it’s most anticipated feature is a HIP-capable compiler. The runtime component amdhip64.dll has been shipping with AMD Software: Adrenalin Edition for multiple years now, and with some trickery one could consume the HIP host-side API by taking the API headers from GitHub (or a Linux ROCm install) and creating an export lib from the driver DLL. Feeding device code compiled offline and given to HIP’s Module API  was attainable, yet cumbersome. Anticipation is driven by the single-source compilation model of HIP borrowed from CUDA. That is finally available* now!

[*]: That is, if you are using Visual Studio and MSBuild, or legacy HIP compilation atop CMake CXX language support.

We cannot fathom how an ecosystem that has existed on Linux for 7 years and has relied primarily on CMake as its build system could ship with unfinished CMake support. Starting from 3.21 CMake has native support for the HIP language. We understand that the Windows release wanted to make Visual Studio support a priority (driven by the HIP VS Extension), but showing the back seat for the myriad of existing projects on Linux is a missed opportunity at best. StreamHPC being the authors of multiple larger libraries: rocRAND, rocPRIM, rocThrust and hipCUB, we took great care while developing rocm-examples that our and others’ existing workflows do not break.

Using HIP in CMake is summarized well on the Using CMake page at AMD’s doc site, so we wouldn’t repeat all that’s there.

The main point is that CMake only ships half of the HIP language support, the part defining the HIP-specific variables, properties etc. How that translates to toolchain invocations are in separate scripts shipping with the toolchains themselves. This second part is what’s half-baked in the initial Windows release of ROCm (documented under the ROCm on Windows section).

Join us for the fix in CMake and the thought process behind it to enable developing HIP on Windows using CMake HIP language support.

Step-by-Step

Prerequisites

We will be riding the waves of CMake on the command-line. For this, minimally install the following SW:

• Git for Windows
  • if you’re a developer of any kind, you likely have this installed already
• Visual Studio 2022 Build Tools
  • Tick “Desktop development with C++” workload.
  • From Individual components select
    • MSVC v143 – VS 2022 C++ x64/x86 build tools (Latest)
    • Windows 10/11 (depending on your host system) SDK (any version)
• ROCm Windows SDK
• CMake 3.27
  • Latest at the time of writing
• Ninja

Let the games begin!

We clone the rocm-examples repo and try to configure a build using defaults for everything.

PS C:\Users\User\rocm-examples> cmake -S . -B build
-- Building for: Visual Studio 17 2022
...
CMake Error at C:/Program Files/Kitware/CMake/3.27.1/share/cmake-3.27/Modules/CMakeDetermineHIPCompiler.cmake:10 (message):
  HIP language not currently supported by "Visual Studio 17 2022" generator

Looking at the relevant part of CMakeDetemineHIPCompiler.cmake:10, we can see that this is a hardcoded error inside CMake. The default generator of CMake on Windows is the latest Visual Studio found on the system. HIP support in MSBuild (VS’s build system) is driven by the “AMD HIP Toolchain” extension. There are two blockers here:

• The extension currently only applies to IDE installations of VS, it will not apply to Build Tools-only installations.
• CMake needs to be taught how to use that extension.

If the inability to use the Visual Studio generators bothers you, consider upvoting this and this issue on AMD’s and Kitware’s trackers.

Moving on to using Ninja (non Multi-Config for brevity, however we advise defaulting to Ninja Multi-Config in most cases):

PS C:\Users\User\rocm-examples> cmake -G Ninja -S . -B build
-- The CXX compiler identification is MSVC 19.36.32537.0
...
-- The HIP compiler identification is unknown
CMake Error at C:/Program Files/Kitware/CMake/3.27.1/share/cmake-3.27/Modules/CMakeDetermineHIPCompiler.cmake:104 (message):   	 
  Failed to find ROCm root directory.

“HIP compiler identification is unknown”, “Failed to find ROCm root directory”… so after installing the ROCm SDK, CMake won’t find it in its default location? Well, skimming through CMakeDetermineHIPCompiler.cmake, we will soon find that right after the previous hardcoded error, if CMAKE_HIP_COMPILER is not set, CMake will enter a guessing game, of where it may be, potentially guided by the HIPCXX environment variable. Unless a compiler is found, the last desperate attempt is to consult the CMAKE_HIP_COMPILER_ROCM_ROOT variable, which it will use as a guide to finding the SDK side script driving the language support.

Environment variables by nature diffuse, leak into every context and are observable at every level of a stack. This is why we love using them at times, when we wish to push some information to the bottom of a call stack or a toolchain without the use of global variables or extra function arguments. The danger of env vars is exactly the same: they leak into contexts we may not have intended. They change behaviour at a global scale. If we know the purpose and there’s a convenient way to restrict the effect to some specific context, we should prefer that method.

We chose the most up-front method and set CMAKE_HIP_COMPILER on the command-line.

PS C:\Users\User\rocm-examples> cmake -G Ninja -D CMAKE_HIP_COMPILER="${env:HIP_PATH}bin\clang++.exe" -S . -B build
-- The HIP compiler identification is Clang 17.0.0 with GNU-like command-line
CMake Error at C:/Program Files/Kitware/CMake/3.27.1/share/cmake-3.27/Modules/CMakeDetermineHIPCompiler.cmake:143 (message):
  The ROCm root directory:

   C:/Program Files/AMD/ROCm/5.5

  does not contain the HIP runtime CMake package, expected at one of:

   C:/Program Files/AMD/ROCm/5.5/lib/cmake/hip-lang/hip-lang-config.cmake
   C:/Program Files/AMD/ROCm/5.5/lib64/cmake/hip-lang/hip-lang-config.cmake

Notice how we used the HIP_LANG env var in the value of CMAKE_HIP_COMPILER created by the installer (note it has a trailing backslash). This way should we update our toolchain, we’ll always get the latest compiler. This time around CMake argues that it can’t find hip-lang-config.cmake. This is the file that maps the CMake properties into the toolchain invocations. Unfortunately AMD managed to omit it from the SDK, even though we have the folder and other files in that folder.

PS C:\Users\User\rocm-examples> Get-ChildItem ${env:HIP_PATH}lib\cmake\hip-lang


	Directory: C:\Program Files\AMD\ROCm\5.5\lib\cmake\hip-lang


Mode             	LastWriteTime     	Length Name
----             	-------------     	------ ----
-a----     	7/21/2023  11:17 AM        	922 hip-lang-targets-release.cmake
-a----     	7/21/2023  11:17 AM       	3577 hip-lang-targets.cmake

This file usually comes from hipamd (AMD’s implementation of the HIP interface) component. If we don’t want to manually configure it, we can just take the version from a Linux installation and tweak it slightly.

Got hip-lang-config.cmake and hip-lang-targets.cmake. Next try:

PS C:\Users\User\rocm-examples> cmake -G Ninja -D CMAKE_HIP_COMPILER="${env:HIP_PATH}bin\clang++.exe" -S . -B build        	 
-- The HIP compiler identification is Clang 17.0.0 with GNU-like command-line
CMake Error at build/CMakeFiles/3.27.1/CMakeHIPCompiler.cmake:1 (set):
  Syntax error in cmake code at

	C:/Users/User/rocm-examples/build/CMakeFiles/3.27.1/CMakeHIPCompiler.cmake:1

  when parsing string

	C:\Program Files\AMD\ROCm\5.5\bin\clang++.exe

  Invalid character escape '\P'.

CMake hasn’t a very sophisticated scripting language, its type system is fairly weak, so it’s easy to get into trouble on Windows with backslashes in paths being interpreted as escape characters. To fix this issue, specify what type of cache variable we want to define on the command line using -D VAR_NAME:VAR_TYPE=VAR_VALUE. Next try:

PS C:\Users\User\rocm-examples> cmake -G Ninja -D CMAKE_HIP_COMPILER:FILEPATH="${env:HIP_PATH}bin\clang++.exe" -S . -B build
-- The HIP compiler identification is Clang 17.0.0 with GNU-like command-line
CMake Error at C:/Program Files/Kitware/CMake/3.27.1/share/cmake-3.27/Modules/Platform/Windows-Clang.cmake:170 (message):
  The current configuration mixes Clang and MSVC or some other CL compatible
  compiler tool.  This is not supported.  Use either clang or MSVC as both C,
  C++ and/or HIP compilers.

This error is a quality of life improvement introduced in CMake 3.25, although CMake HIP language support only needs 3.21. The reason behind it is that CMake decides globally whether to use MSVC-like or GNU-like command-line syntax globally. Mixing-matching the two will result in compiler or linker errors. If a project uses both CXX and HIP language support, they must use the same style command-line.

So we specify the C++ compiler to be ROCmCC as well and clean configure (toolchain selection is cached and one cannot change the compiler after it has been found):

PS C:\Users\User\rocm-examples> cmake -G Ninja -D CMAKE_CXX_COMPILER:FILEPATH="${env:HIP_PATH}bin\clang++.exe" -D CMAKE_HIP_COMPILER:FILEPATH="${env:HIP_PATH}bin\clang++.exe" -S . -B build
-- The CXX compiler identification is Clang 17.0.0 with GNU-like command-line
...
-- The HIP compiler identification is Clang 17.0.0 with GNU-like command-line
CMake Error at C:/Program Files/Kitware/CMake/3.27.1/share/cmake-3.27/Modules/CMakeFindDependencyMacro.cmake:76 (find_package):
  By not providing "FindAMDDeviceLibs.cmake" in CMAKE_MODULE_PATH this
  project has asked CMake to find a package configuration file provided by
  "AMDDeviceLibs", but CMake did not find one.

  Could not find a package configuration file provided by "AMDDeviceLibs"
  with any of the following names:

	AMDDeviceLibsConfig.cmake
	amddevicelibs-config.cmake

  Add the installation prefix of "AMDDeviceLibs" to CMAKE_PREFIX_PATH or set
  "AMDDeviceLibs_DIR" to a directory containing one of the above files.  If
  "AMDDeviceLibs" provides a separate development package or SDK, be sure it
  has been installed.

Some of the package config files are missing from the installation, as requested from hip-lang-config.cmake:87 (find_dependency). For the uninitiated: find_dependency is essentially find_package, but meant to be used inside invocations of find_package. This file indeed is missing, we borrow it from Linux as well.

Our next attempt gives the exact same error, and that is because of one of the most subtle package search quirks of CMake. If you consult the Config Mode Search Procedure of find_package, you’ll find:

Even though AMD’s APT repo doesn’t add the install to the PATH, testing is often done in rocm-terminal where it is on the PATH. /bin is taken off the end of PATH entries, hence /opt/rocm/ is now on the find_package search prefix. The Windows installer (correctly) doesn’t add the installed bin folder to the PATH, so hip-lang-config.cmake needs HINTS added to the find_dependency invocations to help it find its own components. The CXX piggyback scripts, hip-config.cmake ever so casually sidesteps this issue by not exposing the problematic components on Windows as a “hotfix”. The undeployed hip-lang-config.cmake have not yet received this treatment.

After we’ve added the HINTS, we are within arm’s reach of our goal. Next we get:

PS C:\Users\User\rocm-examples> cmake -G Ninja -D CMAKE_CXX_COMPILER:FILEPATH="${env:HIP_PATH}bin\clang++.exe" -D CMAKE_HIP_COMPILER:FILEPATH="${env:HIP_PATH}bin\clang++.exe" -S . -B build
-- The HIP compiler identification is Clang 17.0.0 with GNU-like command-line
CMake Error at C:/Program Files/AMD/ROCm/5.5/lib/cmake/amd_comgr/amd_comgr-targets.cmake:75 (message):
  The imported target "amd_comgr" references the file

 	"C:/Program Files/AMD/ROCm/5.5/lib/libamd_comgr.so.2.5.50500"

  but this file does not exist.  Possible reasons include:

  * The file was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and contained

 	"C:/Program Files/AMD/ROCm/5.5/lib/cmake/amd_comgr/amd_comgr-targets.cmake"

  but not all the files it references.

Of course, if we borrow files from Linux, they’ll refer to Linux shared objects. Patching up amd_comgr-config.cmake we also borrowed from Linux we’ll get:

PS C:\Users\User\rocm-examples> cmake -G Ninja -D CMAKE_CXX_COMPILER:FILEPATH="${env:HIP_PATH}bin\clang++.exe" -D CMAKE_HIP_COMPILER:FILEPATH="${env:HIP_PATH}bin\clang++.exe" -S . -B build
-- The HIP compiler identification is Clang 17.0.0 with GNU-like command-line
CMake Error at C:/Program Files/AMD/ROCm/5.5/lib/cmake/hip-lang/hip-lang-targets.cmake:82 (message):
  The imported target "hip-lang::amdhip64" references the file

 	"C:/constructicon/builds/gfx/two/23.10/drivers/compute/install/native/Release/x64/hip/lib/amdhip64.lib"

  but this file does not exist.  Possible reasons include:

  * The file was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and contained

 	"C:/Program Files/AMD/ROCm/5.5/lib/cmake/hip-lang/hip-lang-targets.cmake"

  but not all the files it references.

Oooh, “C:/construction/builds/gfx/two/23.10/… ” looks very much like a hardcoded build server path. Of course, nobody hardcoded this, it’s CMake generating this file. A seasoned CMake developer at this point inspects all the scar tissue covering their body, their mind’s eye replays a few episodes of scripting nightmares long past, and from the depths of recollection emerges a passage from the install() command’s DESTINATION.

Following our scar tissue induced lead, we zoom in on the installation of hip-lang-targets, and while the targets are installed in proper GNUInstallDirs locations CONFIG_LANG_PACKAGE_INSTALL_DIR is suspiciously non-standard. Tracing down its source value through a layer of indirection its value is indeed prefixed with CMAKE_INSTALL_PREFIX, making it an absolute path, thus rendering the package non-relocatable. This happens to work™ on Linux, where packages on build servers are likely targeting /opt/rocm, just like the actual DEB package install paths. The hardcoded package path happens to coincide between the build server and the user’s machine.

Fixing that up to be a path relative to the config file itself, we get the following:

PS C:\Users\User\rocm-examples> cmake -G Ninja -D CMAKE_CXX_COMPILER:FILEPATH="${env:HIP_PATH}bin\clang++.exe" -D CMAKE_HIP_COMPILER:FILEPATH="${env:HIP_PATH}bin\clang++.exe" -S . -B build
-- The HIP compiler identification is Clang 17.0.0 with GNU-like command-line
CMake Error at C:/Program Files/AMD/ROCm/5.5/lib/cmake/hip-lang/hip-lang-targets.cmake:88 (message):
  The imported target "hip-lang::amdhip64" references the file

 	"C:/Program Files/AMD/ROCm/5.5/bin/amdhip64.dll"

  but this file does not exist.  Possible reasons include:

  * The file was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and contained

 	"C:/Program Files/AMD/ROCm/5.5/lib/cmake/hip-lang/hip-lang-targets.cmake"

  but not all the files it references.

The amdhip64.dll doesn’t ship with the SDK but with Radeon Software. It’s part of the driver, hence it’s found in C:\Windows\System32.

(Note that recording the DLL as an IMPORTED_LOCATION property on hip-lang::amdhip64 one introduces a requirement that the build machine has to have a Radeon driver install. In most cases this shouldn’t be an issue, but omitting it will render CMake oblivious of its existence. A proper fix from AMD will have to consider how to expose the DLL.)

Fixing that up, configuration finally passes. We can now build our code using ninja:

PS C:\Users\User\rocm-examples> cmake --build --target hip_saxpy
[1/2] Building HIP object HIP-Basic/saxpy/CMakeFiles/hip_saxpy.dir/main.hip.obj
[2/2] Linking HIP executable HIP-Basic\saxpy\hip_saxpy.exe

If the samples would argue about some device binary missing from the executable, that just means that the wrong device binaries were embedded into the executable. The target property HIP_ARCHITECTURES controls which binary flavours are embedded into the target. Typically one would set it using the variable CMAKE_HIP_ARCHITECTURES, suggested to be explicitly set by the docs, otherwise the implementation chooses some defaults. CMakeDetermineHIPCompiler.cmake is trying to invoke rocm_agent_enumerator, an extremely lightweight utility just listing ROCm devices and parses the gfxXYZ entries from the response. This utility is missing from Windows for good reasons.

The runtime behind ROCm is ROCclr (ROCm Common Language Runtime), which on Linux is implemented by ROCr, turning API calls to amdkfd** (AMD’s kernel-mode driver) calls. ROCclr exists on Windows too, but instead of amdkfd, it’s implemented by PAL, the Platform Abstraction Layer. As a consequence, it is clearly visible that none of its existing search methods are going to work on Windows, so even if it were present, rocm_agent_enumeratir wouldn’t function. The rocminfo utility’s place is taken by hipInfo.exe.

To find what device binary formats we need to embed into our HIP executables, we have to query the devices in our system. eg.:

PS C:\Users\User\rocm-examples> & "$env:HIP_PATH\bin\hipInfo.exe" | Select-String gfx

gcnArchName:                  	gfx1032
gcnArchName:                  	gfx1035

So by specifying these graphics IP names as the default value for the HIP_ARCHITECTURES target property, all our compiled code should function correctly.

PS C:\Users\User\rocm-examples> cmake -G Ninja -S . -B build -DCMAKE_CXX_COMPILER:FILEPATH="${env:HIP_PATH}bin\clang++.exe" -DCMAKE_HIP_COMPILER:FILEPATH="${env:HIP_PATH}bin\clang++.exe" -DCMAKE_HIP_ARCHITECTURES=”gfx1032;gfx1035”

Running the sample codes should look something like this:

PS C:\Users\User\rocm-examples\build> .\HIP-Basic\saxpy\hip_saxpy.exe
Calculating y[i] = a * x[i] + y[i] over 1000000 elements.
First 10 elements of the results: [ 3, 5, 7, 9, 11, 13, 15, 17, 19, 21 ]

Do note that the default build configuration is Debug for CMake. Performance should improve by also adding -D CMAKE_BUILD_TYPE=Release to the configuration command-line.

[**] Fun fact: the name comes from the earliest incantations of HSA, Heterogeneous System Architecture. This standard was coined shortly after AMD acquired ATi to obtain their vast range of graphics IP. The first APUs were born and the marketing name from AMD at the time was “The future is Fusion” and the SW stack implementing these integration features were called FSA, Fusion System Architecture. Because other vendors, primarily ARM, was interested in this design, it was spun off into a separate consortium, but naturally ARM disliked the fact that AMD brand naming is engraved into the now standard technology. Thus, FSA was renamed to HSA. Much of AMD’s implementation however already used Fusion as the naming of SW components and some of those remain to this day. The driver name amdkfd expands to AMD Kernel Fusion Driver, but really it was set out to implement ultimately what’s called HSA today.

Conclusion

Capturing the above in a series of patch files, one may patch the AMD HIP SDK for Windows to enable CMake HIP language support. Download the ZIP file with the packages from here, extract it anywhere on your disk, and from the folder containing the patches using Git on a command-line with Administrator Privileges patch your installation:

Get-ChildItem .\00*.patch | ForEach-Object { git apply --unsafe-paths "--directory=${env:HIP_PATH}." --ignore-whitespace --whitespace=nowarn $_.FullName }