https://github.com/yxsamliu updated https://github.com/llvm/llvm-project/pull/200208
>From 44e5fecc304190a881022be8c5cb16b67ae22b5f Mon Sep 17 00:00:00 2001 From: "Yaxun (Sam) Liu" <[email protected]> Date: Thu, 28 May 2026 11:11:01 -0400 Subject: [PATCH] [docs][HIP] Document offload PGO workflow Add a section to HIPSupport.rst describing IR-level profile-guided optimization for HIP device code. -fprofile-generate instruments both host and device; the runtime writes a host .profraw and one set of device .profraw files per --offload-arch= value, with the standard LLVM_PROFILE_FILE substitutions applying to both. Host and each per-architecture device profile are merged independently with llvm-profdata, and the use-phase build feeds them back via -Xarch_host -fprofile-use= and -Xarch_<gpu-arch> -fprofile-use= (with -Xarch_device as a single-arch shorthand). Also add a CUDA/HIP Language Changes entry in ReleaseNotes.rst. --- clang/docs/HIPSupport.rst | 98 +++++++++++++++++++++++++++++++++++++ clang/docs/ReleaseNotes.rst | 7 +++ 2 files changed, 105 insertions(+) diff --git a/clang/docs/HIPSupport.rst b/clang/docs/HIPSupport.rst index 82070a4042679..d9e10310e7e64 100644 --- a/clang/docs/HIPSupport.rst +++ b/clang/docs/HIPSupport.rst @@ -951,6 +951,104 @@ Open Questions / Future Developments 4. Offload support might be extended to cases where the ``parallel_policy`` is used for some or all targets. +Profile-Guided Optimization for Device Code +=========================================== + +Clang supports IR-level profile-guided optimization (PGO) for HIP device +code on AMD GPUs. ``-fprofile-generate`` instruments both host and +device code; running the instrumented binary writes separate host and +device raw profiles, which are merged independently and consumed by a +second build that passes the appropriate profile to each side. + +Prerequisites +------------- + +The toolchain must be built with the AMDGPU profile runtime enabled, +which requires building ``compiler-rt`` for the ``amdgcn-amd-amdhsa`` +target via the runtimes build. A minimal CMake configuration is: + +.. code-block:: console + + $ cmake <llvm-project>/llvm \ + -DLLVM_ENABLE_PROJECTS='clang;lld' \ + -DLLVM_ENABLE_RUNTIMES=compiler-rt \ + -DLLVM_RUNTIME_TARGETS='default;amdgcn-amd-amdhsa' \ + -DRUNTIMES_amdgcn-amd-amdhsa_CACHE_FILES=<llvm-project>/compiler-rt/cmake/caches/AMDGPU.cmake \ + -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES='compiler-rt;libc' \ + -DRUNTIMES_amdgcn-amd-amdhsa_RUNTIMES_USE_LIBC=llvm-libc + +``COMPILER_RT_BUILD_PROFILE_ROCM`` (on by default for non-Apple hosts) +controls building the host-side ROCm/HIP device profile collection +runtime; leave it enabled. ``RUNTIMES_USE_LIBC=llvm-libc`` is required +so the amdgcn profile compile picks up LLVM-libc's ``-isystem`` / +``-nostdlibinc`` headers. + +Generate phase +-------------- + +The driver forwards ``-fprofile-generate`` to the device compiler and +links the device profile runtime into the embedded device image. + +.. code-block:: console + + $ clang++ -x hip demo.hip \ + --offload-arch=gfx1100 --offload-arch=gfx1101 \ + -fprofile-generate=pgo_data \ + -o demo.instr + + $ ./demo.instr + +When the instrumented binary exits, the runtime writes raw profile +files into ``pgo_data/``. Host profiles use the standard LLVM profile +filename; device profiles use the same filename with the GPU +architecture name prepended to the basename, so each +``--offload-arch=`` value produces its own set of device files. The +usual ``LLVM_PROFILE_FILE`` substitutions (``%p`` for process ID, +``%m`` for binary signature, etc.) apply to both, so multi-process +runs do not need a separate device-side naming scheme. + +Merge the host profile and each device architecture's profile +separately: + +.. code-block:: console + + $ llvm-profdata merge -o host.profdata pgo_data/default_*.profraw + $ llvm-profdata merge -o device.gfx1100.profdata pgo_data/gfx1100*.profraw + $ llvm-profdata merge -o device.gfx1101.profdata pgo_data/gfx1101*.profraw + +Use phase +--------- + +Host and device compilations consume different profiles, and each GPU +architecture consumes its own. ``-Xarch_host`` selects the host +profile and ``-Xarch_<gpu-arch>`` selects the per-architecture device +profile: + +.. code-block:: console + + $ clang++ -x hip demo.hip \ + --offload-arch=gfx1100 --offload-arch=gfx1101 \ + -Xarch_host -fprofile-use=host.profdata \ + -Xarch_gfx1100 -fprofile-use=device.gfx1100.profdata \ + -Xarch_gfx1101 -fprofile-use=device.gfx1101.profdata \ + -o demo + +For a single-arch build, ``-Xarch_device`` is a convenient shorthand +that applies the same profile to every offload architecture: + +.. code-block:: console + + $ clang++ -x hip demo.hip --offload-arch=gfx1101 \ + -Xarch_host -fprofile-use=host.profdata \ + -Xarch_device -fprofile-use=device.gfx1101.profdata \ + -o demo + +Notes +----- + +- The instrumented build is slower than a normal build; only the use + phase produces the optimized binary intended for deployment. + SPIR-V Support on HIPAMD ToolChain ================================== diff --git a/clang/docs/ReleaseNotes.rst b/clang/docs/ReleaseNotes.rst index 11cce36a0906c..9544c980f6339 100644 --- a/clang/docs/ReleaseNotes.rst +++ b/clang/docs/ReleaseNotes.rst @@ -809,6 +809,13 @@ CUDA/HIP Language Changes - The new offloading driver is now the default for HIP. Use `--no-oflfoad-new-driver` to return to the old behavior. +- Added IR-level profile-guided optimization (PGO) support for HIP + device code on AMD GPUs. ``-fprofile-generate`` now instruments both + host and device; running the instrumented binary writes host and + per-GPU-architecture device raw profiles, which are merged separately + with ``llvm-profdata`` and fed back via ``-Xarch_host`` / + ``-Xarch_<gpu-arch>`` ``-fprofile-use=``. See :doc:`HIPSupport` for + the full workflow. CUDA Support ^^^^^^^^^^^^ _______________________________________________ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
