## Problem 1: TVMOp doesn't work well with GPU builds 
[#17840](https://github.com/apache/incubator-mxnet/issues/17840)

### The error message:

```python
>>> import mxnet as mx
>>> x = mx.np.array([[0, 1], [1, 1], [2, 2]], ctx=mx.gpu())
>>> idx = x < 2
>>> x[idx]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/Documents/mxnet/python/mxnet/numpy/multiarray.py", line 
1013, in __lt__
    return less(self, other)
  File "/home/ubuntu/Documents/mxnet/python/mxnet/numpy/multiarray.py", line 
8672, in less
    return _mx_nd_np.less(x1, x2, out)
  File "/home/ubuntu/Documents/mxnet/python/mxnet/ndarray/numpy/_op.py", line 
6869, in less
    return _api_internal.less(x1, x2, out)
  File "/home/ubuntu/Documents/mxnet/python/mxnet/_ffi/_ctypes/function.py", 
line 115, in __call__
    raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
  File "../3rdparty/tvm/src/runtime/module.cc", line 125
  File "../3rdparty/tvm/src/runtime/library_module.cc", line 94
TVMError: Check failed: ret == 0 (-1 vs. 0) : Check failed: f != nullptr: 
Cannot find function less_scalar_gpufloat32_2bool_2_kernel0 in the imported 
modules or global registry
```

### Root cause:

In 
[mxnet/contrib/tvmop/compile.py](https://github.com/apache/incubator-mxnet/blob/master/contrib/tvmop/compile.py),
 only `function_binary` `(llvm Module)`is saved in `libtvmop.so`. The 
`imported_modules` `(cuda Module)`is not saved. So TVM cannot import any gpu 
functions and cannot find `less_scalar_gpufloat32_2bool_2_kernel0.`

```python
(Pdb) func_binary
Module(llvm, 55d7ce519d48)
(Pdb) func_binary.imported_modules[0]
Module(cuda, 55d7c7a09818)
```

### Solution ([Github 
PR](https://github.com/apache/incubator-mxnet/pull/18678)):

* Save `imported_modules[0]` to `libtvmop.cubin:`
* Define `Import` function (using `TVMOpModule->Import` ):
* Import `cubin_module` to `global_module`:
* Outputs:

```python
>>> import mxnet as mx
>>> x = mx.np.array([[0, 1], [1, 1], [2, 2]], ctx=mx.gpu())
[10:19:41] ../src/base.cc:80: cuDNN lib mismatch: linked-against version 7605 
!= compiled-against version 7501. Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this 
warning.
[10:19:41] ../src/base.cc:84: Upgrade advisory: this mxnet has been built 
against cuDNN lib version 7501, which is older than the oldest version tested 
by CI (7600). Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning.
>>> idx = x < 2
>>> x[idx]
array([0., 1., 1., 1.], ctx=gpu(0))
```

## Problem 2: CI Checks: libcuda.so does exist on the machine builds mxnet

### The error message:

When running 
[unix-gpu](http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18526/18/pipeline)
 checks:

```python
[2020-06-18T08:26:18.355Z] Traceback (most recent call last):
[2020-06-18T08:26:18.355Z]   File "/work/mxnet/contrib/tvmop/compile.py", line 
20, in <module>
[2020-06-18T08:26:18.355Z]     import tvm
[2020-06-18T08:26:18.355Z]   File 
"/work/mxnet/3rdparty/tvm/python/tvm/__init__.py", line 27, in <module>
[2020-06-18T08:26:18.355Z]     from . import tensor
[2020-06-18T08:26:18.355Z]   File 
"/work/mxnet/3rdparty/tvm/python/tvm/tensor.py", line 20, in <module>
[2020-06-18T08:26:18.355Z]     from ._ffi.object import Object, 
register_object, ObjectGeneric, \
[2020-06-18T08:26:18.355Z]   File 
"/work/mxnet/3rdparty/tvm/python/tvm/_ffi/object.py", line 24, in <module>
[2020-06-18T08:26:18.355Z]     from .base import _FFI_MODE, _RUNTIME_ONLY, 
check_call, _LIB, c_str
[2020-06-18T08:26:18.355Z]   File 
"/work/mxnet/3rdparty/tvm/python/tvm/_ffi/base.py", line 65, in <module>
[2020-06-18T08:26:18.355Z]     _LIB, _LIB_NAME = _load_lib()
[2020-06-18T08:26:18.355Z]   File 
"/work/mxnet/3rdparty/tvm/python/tvm/_ffi/base.py", line 57, in _load_lib
[2020-06-18T08:26:18.355Z]     lib = ctypes.CDLL(lib_path[0], 
ctypes.RTLD_GLOBAL)
[2020-06-18T08:26:18.355Z]   File "/usr/lib/python3.6/ctypes/__init__.py", line 
348, in __init__
[2020-06-18T08:26:18.355Z]     self._handle = _dlopen(self._name, mode)
[2020-06-18T08:26:18.355Z] OSError: libcuda.so.1: cannot open shared object 
file: No such file or directory
```

### Root cause:

The unix-gpu machine that **builds** mxnet does not have libcuda.so

### Solution 1:

Link `libtvm.so` with `stub/libcuda.so` on the machine that **builds** CI 
Checks.

### Solution 1 Pros/Cons/Workloads:

* **Pros**: Solve the issue easily.
* **Cons**: Violates the effort of removing `libcuda.so` totally, (would be 
great if someone can elaborate the motivation behind it).
* **Workloads**: ~1 week

### Solution 2 (Possible) ([Github 
PR](https://github.com/jinboci/incubator-tvm/pull/2)) :

TVM links libcuda.so because it invokes CUDA driver API during runtime. While 
these functions are not executed during compile-time. Therefore it is possible 
to remove them for compile-only purpose. 
I have made a prototype to remove the linkage of libcuda.so from libtvm.so:

* Set `target_link_libraries` of tvm and tvm_runtime differently. 
(CMakeLists.txt)
* Set an variable `CUDA_COMPILE_ONLY` to be `ON` to indicate “Building 
libtvm.so without libcuda.so” (CMakeLists.txt)
* When `CUDA_COMPILE_ONLY` is `ON`, add compilation definition 
`-DCUDA_COMPILE_ONLY` (CMakeLists.txt)
* When `CUDA_COMPILE_ONLY` is defined (when compiling libtvm.so), ignore any 
cuXXX CUDA Driver API functions: (cmake/modules/CUDA.cmake, 
src/runtime/cuda/cuda_common.h, src/runtime/cuda/cuda_device_api.cc, 
src/runtime/cuda/cuda_module.cc)

### Solution 2 Pros/Cons/Workloads:

* **Pros**: Not depend on libcuda.so

* **Cons**: After unlinking libtvm.so with libcuda.so, the CI checks still 
cannot pass. [GPU CUDA 
RTC](http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18678/8/pipeline)
 (When `-DUSE_TVM_OP=ON`) outputs the error message:

```python
[2020-07-13T09:30:36.870Z] /usr/bin/ld: warning: libcuda.so.1, needed by 
/work/build/3rdparty/tvm/libtvm_runtime.so, not found (try using -rpath or 
-rpath-link)
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: 
undefined reference to `cuMemsetD32_v2'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: 
undefined reference to `cuModuleLoadData'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: 
undefined reference to `cuLaunchKernel'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: 
undefined reference to `cuModuleGetFunction'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: 
undefined reference to `cuModuleUnload'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: 
undefined reference to `cuGetErrorName'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: 
undefined reference to `cuDeviceGetName'
[2020-07-13T09:30:36.870Z] /work/build/3rdparty/tvm/libtvm_runtime.so: 
undefined reference to `cuModuleGetGlobal_v2'
[2020-07-13T09:30:36.870Z] collect2: error: ld returned 1 exit status
[2020-07-13T09:30:36.870Z] ninja: build stopped: subcommand failed.
[2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,154 - root - INFO - Waiting for 
status of container e95e5c4ca642 for 600 s.
[2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,377 - root - INFO - Container 
exit status: {'Error': None, 'StatusCode': 1}
[2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,377 - root - ERROR - Container 
exited with an error 😞
[2020-07-13T09:30:36.870Z] 2020-07-13 09:30:30,377 - root - INFO - Executed 
command for reproduction:
[2020-07-13T09:30:36.870Z]
```

It seems unlinking `libcuda.so` from `libtvm.so` is not enough and we should 
also unlink `libcuda.so` from `libtvm_runtime.so`. However, `libtvm_runtime` 
does require `libcuda.so` during runtime. if we remove the linkage on build 
instances, `tvm_runtime` is not able to run on test instances.
In order to fully address the problem, we have two options,

1. Build two versions of tvm, one links `libcuda.so` for compiling the tvm 
operators; another does not link `libcuda.so` which will be transferred to test 
instances for tvmop tests.
2. We do `dlopen(“libcuda.so”)` in tvm.

* **Workloads**: 
    * option 1: ~1 week to modify tvm, 1.5-2 weeks modify CI
    * option 2: ~2 weeks to modify tvm
    * Both options require big surgery on TVM, thus contributing back to 
upstream tvm might be difficult. Moreover, there’s also risk that tvm community 
would push back our suggestion. Even though they agree to do so, it might cost 
~2 weeks to upstream the changes, and another ~1.5 weeks to sync mxnet’s tvm 
with updated apache/tvm.

## Proposal:

Given the fact that it might take another 4 - 6 weeks to fully address the CI 
problem, we propose to,

* Submit the fix for problem 1.
* Link stub/libcuda.so and enable 1 instance for testing. If this is not 
acceptable, can we keep disabling the tvmop CI for now as it is not an 
essential component?
* Open an issue/RFC in MXNet and TVM to track the remaining problems.

* * *

## Comments:

Also, When setting `-DUSE_TVM_OP=OFF` [the CI checks would be 
stuck](http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18678/10/pipeline).
 The output of GPU CUDA RTC looks like:

```python
[2020-07-13T18:04:12.876Z] + md5sum build/3rdparty/tvm/libtvm_runtime.so
[2020-07-13T18:04:12.876Z] md5sum: build/3rdparty/tvm/libtvm_runtime.so: No 
such file or directory
[2020-07-13T18:04:12.876Z] + ls -lh build/3rdparty/tvm/libtvm_runtime.so
[2020-07-13T18:04:12.876Z] ls: cannot access 
'build/3rdparty/tvm/libtvm_runtime.so': No such file or directory
[2020-07-13T18:04:12.876Z] + md5sum build/libtvmop.so
[2020-07-13T18:04:12.876Z] md5sum: build/libtvmop.so: No such file or directory
[2020-07-13T18:04:12.876Z] + ls -lh build/libtvmop.so
[2020-07-13T18:04:12.876Z] ls: cannot access 'build/libtvmop.so': No such file 
or directory
[2020-07-13T18:04:12.876Z] + md5sum build/tvmop.conf
[2020-07-13T18:04:12.876Z] md5sum: build/tvmop.conf: No such file or directory
[2020-07-13T18:04:12.876Z] + ls -lh build/tvmop.conf
[2020-07-13T18:04:12.876Z] ls: cannot access 'build/tvmop.conf': No such file 
or directory
[2020-07-13T18:04:12.876Z] + md5sum build/tests/mxnet_unit_tests
[2020-07-13T18:04:12.876Z] 66aa8c8a37ffaaa9692ae98bda88491c  
build/tests/mxnet_unit_tests
[2020-07-13T18:04:12.876Z] + ls -lh build/tests/mxnet_unit_tests
[2020-07-13T18:04:12.876Z] -rwxr-xr-x 1 jenkins_slave jenkins_slave 34M Jul 13 
18:04 build/tests/mxnet_unit_tests
[2020-07-13T18:04:12.876Z] + md5sum build/3rdparty/openmp/runtime/src/libomp.so
[2020-07-13T18:04:12.876Z] 819a0c986ae9e233b0a9525e71c906d9  
build/3rdparty/openmp/runtime/src/libomp.so
[2020-07-13T18:04:12.876Z] + ls -lh build/3rdparty/openmp/runtime/src/libomp.so
[2020-07-13T18:04:12.876Z] -rwxr-xr-x 1 jenkins_slave jenkins_slave 1.1M Jul 13 
17:56 build/3rdparty/openmp/runtime/src/libomp.so
[2020-07-13T18:04:12.876Z] + return 0
```


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/18716

Reply via email to