Updated SRU template bug description as suggested @Andreas, thanks! ** Description changed:
- Whenever running tests for the miopen package multiple instances fail - with the same behavior: + [ Impact ] - ``` - [ RUN ] Full/GPU_TernaryTensorOps_FP64.TestDouble/2006 - In file included from <built-in>:1: - In file included from /usr/lib/llvm-21/lib/clang/21/include/__clang_hip_runtime_wrapper.h:111: - In file included from /usr/lib/llvm-21/lib/clang/21/include/cuda_wrappers/cmath:28: - /usr/lib/gcc/x86_64-linux-gnu/16/../../../../include/c++/16/cmath:55:15: fatal error: 'math.h' file not found - 55 | #include_next <math.h> - | ^~~~~~~~ - 1 error generated when compiling for gfx1151. - unknown file: Failure - C++ exception with description "src/hip/hip_build_utils.cpp:185: Failed cmd: '/usr/bin/clang++-21', args: '-DMIOPEN_TYPE=double -DMIOPEN_TENSOR_OP=miopenMin -DDIM_TYPE=uint32_t -DUSE_2D_TENSOR_GENERIC -Wno-everything --std=c++17 --cuda-gpu-arch=gfx1151 --cuda-device-only -c -O3 -Wno-unused-command-line-argument -I. -x hip -isystem /usr/include -D__HIP_PLATFORM_AMD__=1 -DUSE_PROF_API=1 -isystem /usr/include --hip-link -fno-offload-uniform-block -DHIP_PACKAGE_VERSION_FLAT=7001052801 MIOpenTensorKernelsHip.cpp -o /tmp/miopen-MIOpenTensorKernelsHip.cpp-c386-a2ea-e894-5438/MIOpenTensorKernelsHip.cpp.o'" thrown in the test body. + * When MIOpen is built without MIOPEN_USE_HIPRTC/MIOPEN_USE_COMGR, it + compiles HIP kernels at runtime by running the host compiler + (/usr/bin/clang++-21) with "-isystem /usr/include". That puts + /usr/include ahead of the libstdc++ include directory, so the + "#include_next <math.h>" in libstdc++'s <cmath> fails: - [ FAILED ] Full/GPU_TernaryTensorOps_FP64.TestDouble/2006, where GetParam() = 176-byte object <70-F4 F9-38 83-58 00-00 80-F4 F9-38 83-58 00-00 80-F4 F9-38 83-58 00-00 90-69 03-39 83-58 00-00 A0-69 03-39 83-58 00-00 A0-69 03-39 83-58 00-00 B0-E3 05-39 83-58 00-00 C8-E3 05-39 83-58 00-00 ... 68-22 83-38 83-58 00-00 90-1F 83-38 83-58 00-00 B8-1F 83-38 83-58 00-00 B8-1F 83-38 83-58 00-00 A0-4D 04-39 83-58 00-00 AC-4D 04-39 83-58 00-00 AC-4D 04-39 83-58 00-00 00-00 00-00 02-00 00-00> (282 ms) - [ RUN ] Full/GPU_TernaryTensorOps_FP64.TestDouble/2007 - In file included from <built-in>:1: - In file included from /usr/lib/llvm-21/lib/clang/21/include/__clang_hip_runtime_wrapper.h:111: - In file included from /usr/lib/llvm-21/lib/clang/21/include/cuda_wrappers/cmath:28: - /usr/lib/gcc/x86_64-linux-gnu/16/../../../../include/c++/16/cmath:55:15: fatal error: 'math.h' file not found - 55 | #include_next <math.h> - ``` + /usr/lib/gcc/x86_64-linux-gnu/16/../../../../include/c++/16/cmath:55:15: + fatal error: 'math.h' file not found + 55 | #include_next <math.h> + 1 error generated when compiling for gfx1151. + C++ exception with description "src/hip/hip_build_utils.cpp:185: + Failed cmd: '/usr/bin/clang++-21', args: '... -isystem /usr/include + ... MIOpenTensorKernelsHip.cpp ...'" thrown in the test body. + + Every operation that needs a runtime-compiled HIP kernel fails, which + shows up as mass failures in the test suite, e.g. + Full/GPU_TernaryTensorOps_FP64.TestDouble/*. Real workloads using + libmiopen1 hit the same failure. + + * The fix enables MIOPEN_USE_HIPRTC and MIOPEN_USE_COMGR in d/rules, so + kernels are compiled in-process through hipRTC/comgr instead of by + running the host clang++ with the broken include ordering. + + * This fix first landed in resolute in 7.1.0+dfsg-0ubuntu6 and is in + the current release version (7.1.1+dfsg-0ubuntu1). The resolute SRU to + 7.2.4+dfsg-1build1ubuntu0.1 (LP: #2157988) keeps it. This bug is + referenced here so the 7.2.4 SRU can be checked for not regressing it. + + [ Test Plan ] + + * This bug only shows up when MIOpen's runtime kernel compilation runs, + which the libmiopen1-tests autopkgtest exercises heavily. That suite + is the test plan and is run as part of the parent SRU (LP: #2157988). + + * On a machine with a supported AMD GPU, running resolute with + -proposed enabled: + 1. apt install libmiopen1/resolute-proposed libmiopen1-tests/resolute-proposed + 2. Run the autopkgtest suite (libmiopen1-tests), or run the TernaryTensorOps + tests directly: + /usr/libexec/rocm/libmiopen1-tests/run-tests \ + --gtest_filter='Full/GPU_TernaryTensorOps_FP64.*' + 3. Check that no test fails with "fatal error: 'math.h' file not found" + or "hip_build_utils.cpp: Failed cmd: '/usr/bin/clang++-21'". + + [ Where problems could occur ] + + * With HIPRTC/COMGR enabled, runtime kernel compilation depends on + libamd-comgr and the hipRTC library from the ROCm stack instead of the + clang++-21 binary. If those libraries are missing, mismatched, or + buggy on a system, kernel compilation fails at runtime. You would see + errors from comgr/hipRTC (not the 'math.h' error) on the first call to + an operation that needs a runtime-compiled kernel. + + * Kernels built by hipRTC may differ slightly from ones built by the + offline clang++ path in code generation. That could affect + performance, or which kernel the Find step picks. It should not affect + numerical correctness. + + [ Other Info ] + + * Parent SRU: LP: #2157988 (miopen 7.2.4, ROCm 7.2 stack). + * Proposed package: https://launchpad.net/ubuntu/+source/miopen/7.2.4+dfsg-1build1ubuntu0.1 + * Target: resolute 26.04 LTS -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2147985 Title: math.h not found by #include_next on 7.1.0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/miopen/+bug/2147985/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
