jinboci opened a new issue #18716: URL: https://github.com/apache/incubator-mxnet/issues/18716
## Problem 1: TVMOp doesn't work well with GPU builds [#17840](https://github.com/apache/incubator-mxnet/issues/17840) ### The error message: ```python >>> import mxnet as mx >>> x = mx.np.array([[0, 1], [1, 1], [2, 2]], ctx=mx.gpu()) >>> idx = x < 2 >>> x[idx] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/ubuntu/Documents/mxnet/python/mxnet/numpy/multiarray.py", line 1013, in __lt__ return less(self, other) File "/home/ubuntu/Documents/mxnet/python/mxnet/numpy/multiarray.py", line 8672, in less return _mx_nd_np.less(x1, x2, out) File "/home/ubuntu/Documents/mxnet/python/mxnet/ndarray/numpy/_op.py", line 6869, in less return _api_internal.less(x1, x2, out) File "/home/ubuntu/Documents/mxnet/python/mxnet/_ffi/_ctypes/function.py", line 115, in __call__ raise get_last_ffi_error() mxnet.base.MXNetError: Traceback (most recent call last): File "../3rdparty/tvm/src/runtime/module.cc", line 125 File "../3rdparty/tvm/src/runtime/library_module.cc", line 94 TVMError: Check failed: ret == 0 (-1 vs. 0) : Check failed: f != nullptr: Cannot find function less_scalar_gpufloat32_2bool_2_kernel0 in the imported modules or global registry ``` ### Root cause: In [mxnet/contrib/tvmop/compile.py](https://github.com/apache/incubator-mxnet/blob/master/contrib/tvmop/compile.py), only `function_binary` `(llvm Module)`is saved in `libtvmop.so`. The `imported_modules` `(cuda Module)`is not saved. So TVM cannot import any gpu functions and cannot find `less_scalar_gpufloat32_2bool_2_kernel0.` ```python (Pdb) func_binary Module(llvm, 55d7ce519d48) (Pdb) func_binary.imported_modules[0] Module(cuda, 55d7c7a09818) ``` ### Solution ([Github PR](https://github.com/apache/incubator-mxnet/pull/18678)): * Save `imported_modules[0]` to `libtvmop.cubin:` * Define `Import` function (using `TVMOpModule->Import` ): * Import `cubin_module` to `global_module`: * Outputs: ```python >>> import mxnet as mx >>> x = mx.np.array([[0, 1], [1, 1], [2, 2]], ctx=mx.gpu()) [10:19:41] ../src/base.cc:80: cuDNN lib mismatch: linked-against version 7605 != compiled-against version 7501. Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning. [10:19:41] ../src/base.cc:84: Upgrade advisory: this mxnet has been built against cuDNN lib version 7501, which is older than the oldest version tested by CI (7600). Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning. >>> idx = x < 2 >>> x[idx] array([0., 1., 1., 1.], ctx=gpu(0)) ``` ## Problem 2: CI Checks: libcuda.so does exist on the machine builds mxnet ### The error message: When running [unix-gpu](http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18526/18/pipeline) checks: ```python [2020-06-18T08:26:18.355Z] Traceback (most recent call last): [2020-06-18T08:26:18.355Z] File "/work/mxnet/contrib/tvmop/compile.py", line 20, in <module> [2020-06-18T08:26:18.355Z] import tvm [2020-06-18T08:26:18.355Z] File "/work/mxnet/3rdparty/tvm/python/tvm/__init__.py", line 27, in <module> [2020-06-18T08:26:18.355Z] from . import tensor [2020-06-18T08:26:18.355Z] File "/work/mxnet/3rdparty/tvm/python/tvm/tensor.py", line 20, in <module> [2020-06-18T08:26:18.355Z] from ._ffi.object import Object, register_object, ObjectGeneric, \ [2020-06-18T08:26:18.355Z] File "/work/mxnet/3rdparty/tvm/python/tvm/_ffi/object.py", line 24, in <module> [2020-06-18T08:26:18.355Z] from .base import _FFI_MODE, _RUNTIME_ONLY, check_call, _LIB, c_str [2020-06-18T08:26:18.355Z] File "/work/mxnet/3rdparty/tvm/python/tvm/_ffi/base.py", line 65, in <module> [2020-06-18T08:26:18.355Z] _LIB, _LIB_NAME = _load_lib() [2020-06-18T08:26:18.355Z] File "/work/mxnet/3rdparty/tvm/python/tvm/_ffi/base.py", line 57, in _load_lib [2020-06-18T08:26:18.355Z] lib = ctypes.CDLL(lib_path[0], ctypes.RTLD_GLOBAL) [2020-06-18T08:26:18.355Z] File "/usr/lib/python3.6/ctypes/__init__.py", line 348, in __init__ [2020-06-18T08:26:18.355Z] self._handle = _dlopen(self._name, mode) [2020-06-18T08:26:18.355Z] OSError: libcuda.so.1: cannot open shared object file: No such file or directory ``` ### Root cause: The unix-gpu machine that **builds** mxnet does not have libcuda.so ### Solution 1: Link `libtvm.so` with `stub/libcuda.so` on the machine that **builds** CI Checks. ### Solution 1 Pros/Cons/Workloads: * **Pros**: Solve the issue easily. * **Cons**: Violates the effort of removing `libcuda.so` totally, (would be great if someone can elaborate the motivation behind it). * **Workloads**: ~1 week ### Solution 2 (Possible) ([Github PR](https://github.com/jinboci/incubator-tvm/pull/2)) : TVM links libcuda.so because it invokes CUDA driver API during runtime. While these functions are not executed during compile-time. Therefore it is possible to remove them for compile-only purpose. I have made a prototype to remove the linkage of libcuda.so from libtvm.so: * Set `target_link_libraries` of tvm and tvm_runtime differently. (CMakeLists.txt) * Set an variable `CUDA_COMPILE_ONLY` to be `ON` to indicate “Building libtvm.so without libcuda.so” (CMakeLists.txt) * When `CUDA_COMPILE_ONLY` is `ON`, add compilation definition `-DCUDA_COMPILE_ONLY` (CMakeLists.txt) * When `CUDA_COMPILE_ONLY` is defined (when compiling libtvm.so), ignore any cuXXX CUDA Driver API functions: (cmake/modules/CUDA.cmake, src/runtime/cuda/cuda_common.h, src/runtime/cuda/cuda_device_api.cc, src/runtime/cuda/cuda_module.cc) ### Solution 2 Pros/Cons/Workloads: * **Pros**: Not depend on libcuda.so * **Cons**: After unlinking libtvm.so with libcuda.so, the CI checks still cannot pass. [GPU CUDA RTC](http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18678/8/pipeline) (When `-DUSE_TVM_OP=ON`) outputs the error message: ```python [2020-07-13T09:30:36.870Z] /usr/bin/ld: warning: libcuda.so.1, needed by /work/build/3rdparty/tvm/libtvm_runtime.so, not found (try using -rpath or -rpath-link) [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: undefined reference to `cuMemsetD32_v2' [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: undefined reference to `cuModuleLoadData' [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: undefined reference to `cuLaunchKernel' [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: undefined reference to `cuModuleGetFunction' [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: undefined reference to `cuModuleUnload' [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: undefined reference to `cuGetErrorName' [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: undefined reference to `cuDeviceGetName' [2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: undefined reference to `cuModuleGetGlobal_v2' [2020-07-13T09:30:36.870Z] collect2: error: ld returned 1 exit status [2020-07-13T09:30:36.870Z] ninja: build stopped: subcommand failed. [2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,154 - root - INFO - Waiting for status of container e95e5c4ca642 for 600 s. [2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,377 - root - INFO - Container exit status: {'Error': None, 'StatusCode': 1} [2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,377 - root - ERROR - Container exited with an error 😞 [2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,377 - root - INFO - Executed command for reproduction: [2020-07-13T09:30:36.870Z] ``` It seems unlinking `libcuda.so` from `libtvm.so` is not enough and we should also unlink `libcuda.so` from `libtvm_runtime.so`. However, `libtvm_runtime` does require `libcuda.so` during runtime. if we remove the linkage on build instances, `tvm_runtime` is not able to run on test instances. In order to fully address the problem, we have two options, 1. Build two versions of tvm, one links `libcuda.so` for compiling the tvm operators; another does not link `libcuda.so` which will be transferred to test instances for tvmop tests. 2. We do `dlopen(“libcuda.so”)` in tvm. * **Workloads**: * option 1: ~1 week to modify tvm, 1.5-2 weeks modify CI * option 2: ~2 weeks to modify tvm * Both options require big surgery on TVM, thus contributing back to upstream tvm might be difficult. Moreover, there’s also risk that tvm community would push back our suggestion. Even though they agree to do so, it might cost ~2 weeks to upstream the changes, and another ~1.5 weeks to sync mxnet’s tvm with updated apache/tvm. ## Proposal: Given the fact that it might take another 4 - 6 weeks to fully address the CI problem, we propose to, * Submit the fix for problem 1. * Link stub/libcuda.so and enable 1 instance for testing. If this is not acceptable, can we keep disabling the tvmop CI for now as it is not an essential component? * Open an issue/RFC in MXNet and TVM to track the remaining problems. * * * ## Comments: Also, When setting `-DUSE_TVM_OP=OFF` [the CI checks would be stuck](http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18678/10/pipeline). The output of GPU CUDA RTC looks like: ```python [2020-07-13T18:04:12.876Z] + md5sum build/3rdparty/tvm/libtvm_runtime.so [2020-07-13T18:04:12.876Z] md5sum: build/3rdparty/tvm/libtvm_runtime.so: No such file or directory [2020-07-13T18:04:12.876Z] + ls -lh build/3rdparty/tvm/libtvm_runtime.so [2020-07-13T18:04:12.876Z] ls: cannot access 'build/3rdparty/tvm/libtvm_runtime.so': No such file or directory [2020-07-13T18:04:12.876Z] + md5sum build/libtvmop.so [2020-07-13T18:04:12.876Z] md5sum: build/libtvmop.so: No such file or directory [2020-07-13T18:04:12.876Z] + ls -lh build/libtvmop.so [2020-07-13T18:04:12.876Z] ls: cannot access 'build/libtvmop.so': No such file or directory [2020-07-13T18:04:12.876Z] + md5sum build/tvmop.conf [2020-07-13T18:04:12.876Z] md5sum: build/tvmop.conf: No such file or directory [2020-07-13T18:04:12.876Z] + ls -lh build/tvmop.conf [2020-07-13T18:04:12.876Z] ls: cannot access 'build/tvmop.conf': No such file or directory [2020-07-13T18:04:12.876Z] + md5sum build/tests/mxnet_unit_tests [2020-07-13T18:04:12.876Z] 66aa8c8a37ffaaa9692ae98bda88491c build/tests/mxnet_unit_tests [2020-07-13T18:04:12.876Z] + ls -lh build/tests/mxnet_unit_tests [2020-07-13T18:04:12.876Z] -rwxr-xr-x 1 jenkins_slave jenkins_slave 34M Jul 13 18:04 build/tests/mxnet_unit_tests [2020-07-13T18:04:12.876Z] + md5sum build/3rdparty/openmp/runtime/src/libomp.so [2020-07-13T18:04:12.876Z] 819a0c986ae9e233b0a9525e71c906d9 build/3rdparty/openmp/runtime/src/libomp.so [2020-07-13T18:04:12.876Z] + ls -lh build/3rdparty/openmp/runtime/src/libomp.so [2020-07-13T18:04:12.876Z] -rwxr-xr-x 1 jenkins_slave jenkins_slave 1.1M Jul 13 17:56 build/3rdparty/openmp/runtime/src/libomp.so [2020-07-13T18:04:12.876Z] + return 0 ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
