yangshuo0323 opened a new issue #19816:
URL: https://github.com/apache/incubator-mxnet/issues/19816
## Description
I want to use mxnet=1.7.0 with gluonnlp=0.10.0 for multi-GPU trainning and I
meet this error when I insall horovod.
Anyone can help this? Thanks in advance.
```
(qyx37) works@node121:/home/nishome/works/yangshuo_works$
HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITHOUT_GLOO=1 HOROVOD_WITH_MPI=1
HOROVOD_WITH_MXNET=1 HOROVOD_WITHOUT_TENSORFLOW=1 HOROVOD_WITHOUT_PYTORCH=1
python3 -m pip install horovod
Defaulting to user installation because normal site-packages is not writeable
Collecting horovod
Downloading
packages/d3/93/503830d0de5c7390d65c94ffa60db3719826f3deeb0d055fcfa35b4d7927/horovod-0.21.1.tar.gz
(3.2 MB)
|████████████████████████████████| 3.2 MB 1.9 MB/s
Requirement already satisfied: cloudpickle in
/home/nishome/works/.local/lib/python3.7/site-packages (from horovod) (1.6.0)
Requirement already satisfied: psutil in
/home/nishome/works/.local/lib/python3.7/site-packages (from horovod) (5.8.0)
Requirement already satisfied: pyyaml in
/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/site-packages (from
horovod) (5.3.1)
Building wheels for collected packages: horovod
Building wheel for horovod (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /home/nishome/works/miniconda3/envs/qyx37/bin/python3 -u -c
'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-wiooruf_/horovod_ae372f3eb30d402293d967812539ca14/setup.py'"'"';
__file__='"'"'/tmp/pip-install-wiooruf_/horovod_ae372f3eb30d402293d967812539ca14/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
bdist_wheel -d /tmp/pip-wheel-kri6ysyz
cwd:
/tmp/pip-install-wiooruf_/horovod_ae372f3eb30d402293d967812539ca14/
Complete output (218 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/horovod
copying horovod/__init__.py -> build/lib.linux-x86_64-3.7/horovod
creating build/lib.linux-x86_64-3.7/horovod/torch
copying horovod/torch/sync_batch_norm.py ->
build/lib.linux-x86_64-3.7/horovod/torch
copying horovod/torch/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/torch
copying horovod/torch/functions.py ->
build/lib.linux-x86_64-3.7/horovod/torch
copying horovod/torch/compression.py ->
build/lib.linux-x86_64-3.7/horovod/torch
copying horovod/torch/mpi_ops.py ->
build/lib.linux-x86_64-3.7/horovod/torch
copying horovod/torch/optimizer.py ->
build/lib.linux-x86_64-3.7/horovod/torch
creating build/lib.linux-x86_64-3.7/horovod/keras
copying horovod/keras/elastic.py ->
build/lib.linux-x86_64-3.7/horovod/keras
copying horovod/keras/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/keras
copying horovod/keras/callbacks.py ->
build/lib.linux-x86_64-3.7/horovod/keras
creating build/lib.linux-x86_64-3.7/horovod/ray
copying horovod/ray/elastic.py -> build/lib.linux-x86_64-3.7/horovod/ray
copying horovod/ray/__init__.py -> build/lib.linux-x86_64-3.7/horovod/ray
copying horovod/ray/runner.py -> build/lib.linux-x86_64-3.7/horovod/ray
copying horovod/ray/utils.py -> build/lib.linux-x86_64-3.7/horovod/ray
copying horovod/ray/driver_service.py ->
build/lib.linux-x86_64-3.7/horovod/ray
creating build/lib.linux-x86_64-3.7/horovod/_keras
copying horovod/_keras/elastic.py ->
build/lib.linux-x86_64-3.7/horovod/_keras
copying horovod/_keras/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/_keras
copying horovod/_keras/callbacks.py ->
build/lib.linux-x86_64-3.7/horovod/_keras
creating build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/run_task.py ->
build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/mpi_run.py ->
build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/js_run.py ->
build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/gloo_run.py ->
build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/task_fn.py ->
build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/launch.py ->
build/lib.linux-x86_64-3.7/horovod/runner
creating build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/elastic.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/sync_batch_norm.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/gradient_aggregation.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/functions.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/compression.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/mpi_ops.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/util.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/gradient_aggregation_eager.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
creating build/lib.linux-x86_64-3.7/horovod/mxnet
copying horovod/mxnet/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/mxnet
copying horovod/mxnet/functions.py ->
build/lib.linux-x86_64-3.7/horovod/mxnet
copying horovod/mxnet/mpi_ops.py ->
build/lib.linux-x86_64-3.7/horovod/mxnet
creating build/lib.linux-x86_64-3.7/horovod/common
copying horovod/common/elastic.py ->
build/lib.linux-x86_64-3.7/horovod/common
copying horovod/common/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/common
copying horovod/common/exceptions.py ->
build/lib.linux-x86_64-3.7/horovod/common
copying horovod/common/basics.py ->
build/lib.linux-x86_64-3.7/horovod/common
copying horovod/common/util.py -> build/lib.linux-x86_64-3.7/horovod/common
creating build/lib.linux-x86_64-3.7/horovod/spark
copying horovod/spark/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/spark
copying horovod/spark/conf.py -> build/lib.linux-x86_64-3.7/horovod/spark
copying horovod/spark/runner.py -> build/lib.linux-x86_64-3.7/horovod/spark
copying horovod/spark/mpi_run.py ->
build/lib.linux-x86_64-3.7/horovod/spark
copying horovod/spark/gloo_run.py ->
build/lib.linux-x86_64-3.7/horovod/spark
creating build/lib.linux-x86_64-3.7/horovod/torch/mpi_lib
copying horovod/torch/mpi_lib/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/torch/mpi_lib
creating build/lib.linux-x86_64-3.7/horovod/torch/mpi_lib_impl
copying horovod/torch/mpi_lib_impl/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/torch/mpi_lib_impl
creating build/lib.linux-x86_64-3.7/horovod/torch/elastic
copying horovod/torch/elastic/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/torch/elastic
copying horovod/torch/elastic/sampler.py ->
build/lib.linux-x86_64-3.7/horovod/torch/elastic
copying horovod/torch/elastic/state.py ->
build/lib.linux-x86_64-3.7/horovod/torch/elastic
creating build/lib.linux-x86_64-3.7/horovod/runner/task
copying horovod/runner/task/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/task
copying horovod/runner/task/task_service.py ->
build/lib.linux-x86_64-3.7/horovod/runner/task
creating build/lib.linux-x86_64-3.7/horovod/runner/http
copying horovod/runner/http/http_client.py ->
build/lib.linux-x86_64-3.7/horovod/runner/http
copying horovod/runner/http/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/http
copying horovod/runner/http/http_server.py ->
build/lib.linux-x86_64-3.7/horovod/runner/http
creating build/lib.linux-x86_64-3.7/horovod/runner/util
copying horovod/runner/util/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/util
copying horovod/runner/util/threads.py ->
build/lib.linux-x86_64-3.7/horovod/runner/util
copying horovod/runner/util/network.py ->
build/lib.linux-x86_64-3.7/horovod/runner/util
copying horovod/runner/util/remote.py ->
build/lib.linux-x86_64-3.7/horovod/runner/util
copying horovod/runner/util/lsf.py ->
build/lib.linux-x86_64-3.7/horovod/runner/util
copying horovod/runner/util/cache.py ->
build/lib.linux-x86_64-3.7/horovod/runner/util
creating build/lib.linux-x86_64-3.7/horovod/runner/common
copying horovod/runner/common/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common
creating build/lib.linux-x86_64-3.7/horovod/runner/driver
copying horovod/runner/driver/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/driver
copying horovod/runner/driver/driver_service.py ->
build/lib.linux-x86_64-3.7/horovod/runner/driver
creating build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/driver.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/rendezvous.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/registration.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/settings.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/constants.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/worker.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/discovery.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
creating build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/host_hash.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/settings.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/config_parser.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/secret.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/codec.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/hosts.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/tiny_shell_exec.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/timeout.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/network.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/safe_shell_exec.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/env.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
creating build/lib.linux-x86_64-3.7/horovod/runner/common/service
copying horovod/runner/common/service/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/service
copying horovod/runner/common/service/driver_service.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/service
copying horovod/runner/common/service/task_service.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/service
creating build/lib.linux-x86_64-3.7/horovod/tensorflow/keras
copying horovod/tensorflow/keras/elastic.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow/keras
copying horovod/tensorflow/keras/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow/keras
copying horovod/tensorflow/keras/callbacks.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow/keras
creating build/lib.linux-x86_64-3.7/horovod/spark/task
copying horovod/spark/task/task_info.py ->
build/lib.linux-x86_64-3.7/horovod/spark/task
copying horovod/spark/task/gloo_exec_fn.py ->
build/lib.linux-x86_64-3.7/horovod/spark/task
copying horovod/spark/task/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/spark/task
copying horovod/spark/task/task_service.py ->
build/lib.linux-x86_64-3.7/horovod/spark/task
copying horovod/spark/task/mpirun_exec_fn.py ->
build/lib.linux-x86_64-3.7/horovod/spark/task
creating build/lib.linux-x86_64-3.7/horovod/spark/torch
copying horovod/spark/torch/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/spark/torch
copying horovod/spark/torch/estimator.py ->
build/lib.linux-x86_64-3.7/horovod/spark/torch
copying horovod/spark/torch/remote.py ->
build/lib.linux-x86_64-3.7/horovod/spark/torch
copying horovod/spark/torch/util.py ->
build/lib.linux-x86_64-3.7/horovod/spark/torch
creating build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/tensorflow.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/bare.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/estimator.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/remote.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/util.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/optimizer.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
creating build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/store.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/serialization.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/_namedtuple_fix.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/backend.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/constants.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/estimator.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/params.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/cache.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/util.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
creating build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/rsh.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/host_discovery.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/rendezvous.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/driver_service.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/job_id.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/mpirun_rsh.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
running build_ext
-- The CXX compiler identification is GNU 7.3.0
-- Check for working CXX compiler:
/home/nishome/works/Downlaod/gcc7/bin/g++
-- Check for working CXX compiler:
/home/nishome/works/Downlaod/gcc7/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build architecture flags: -mf16c -mavx -mfma
-- Using command /home/nishome/works/miniconda3/envs/qyx37/bin/python3
-- Found MPI_CXX: /usr/local/lib/libmpi.so
-- Found CUDA: /usr/local/cuda-10.2 (found version "10.2")
-- Linking against static NCCL library
-- Found NCCL: /usr/include
-- Determining NCCL version from the header file: /usr/include/nccl.h
-- NCCL_MAJOR_VERSION: 2
-- Found NCCL (include: /usr/include, library:
/usr/lib/x86_64-linux-gnu/libnccl_static.a)
CMake Error at cmake/Modules/FindMxnet.cmake:39 (message):
MXNet was found with mkl-dnn support but mkldnn header files are missing.
Please, install MXNet with mkldnn header files.
Call Stack (most recent call first):
horovod/mxnet/CMakeLists.txt:12 (find_package)
-- Configuring incomplete, errors occurred!
See also
"/tmp/pip-install-wiooruf_/horovod_ae372f3eb30d402293d967812539ca14/build/temp.linux-x86_64-3.7/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
File "<string>", line 1, in <module>
File
"/tmp/pip-install-wiooruf_/horovod_ae372f3eb30d402293d967812539ca14/setup.py",
line 193, in <module>
'horovodrun = horovod.runner.launch:run_commandline'
File
"/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/site-packages/setuptools/__init__.py",
line 144, in setup
return distutils.core.setup(**attrs)
File
"/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/distutils/core.py",
line 148, in setup
dist.run_commands()
File
"/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/distutils/dist.py",
line 966, in run_commands
self.run_command(cmd)
File
"/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/distutils/dist.py",
line 985, in run_command
cmd_obj.run()
File
"/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/site-packages/wheel/bdist_wheel.py",
line 223, in run
self.run_command('build')
File
"/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/distutils/cmd.py",
line 313, in run_command
self.distribution.run_command(command)
File
"/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/distutils/dist.py",
line 985, in run_command
cmd_obj.run()
File
"/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/distutils/command/build.py",
line 135, in run
self.run_command(cmd_name)
File
"/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/distutils/cmd.py",
line 313, in run_command
self.distribution.run_command(command)
File
"/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/distutils/dist.py",
line 985, in run_command
cmd_obj.run()
File
"/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/site-packages/setuptools/command/build_ext.py",
line 87, in run
_build_ext.run(self)
File
"/home/nishome/works/.local/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py",
line 186, in run
_build_ext.build_ext.run(self)
File
"/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/distutils/command/build_ext.py",
line 340, in run
self.build_extensions()
File
"/tmp/pip-install-wiooruf_/horovod_ae372f3eb30d402293d967812539ca14/setup.py",
line 89, in build_extensions
cwd=self.build_temp)
File
"/home/nishome/works/miniconda3/envs/qyx37/lib/python3.7/subprocess.py", line
363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake',
'/tmp/pip-install-wiooruf_/horovod_ae372f3eb30d402293d967812539ca14',
'-DCMAKE_BUILD_TYPE=RelWithDebInfo',
'-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_RELWITHDEBINFO=/tmp/pip-install-wiooruf_/horovod_ae372f3eb30d402293d967812539ca14/build/lib.linux-x86_64-3.7',
'-DPYTHON_EXECUTABLE:FILEPATH=/home/nishome/works/miniconda3/envs/qyx37/bin/python3']'
returned non-zero exit status 1.
----------------------------------------
ERROR: Failed building wheel for horovod
Running setup.py clean for horovod
Failed to build horovod
Installing collected packages: horovod
Running setup.py install for horovod ... error
ERROR: Command errored out with exit status 1:
command: /home/nishome/works/miniconda3/envs/qyx37/bin/python3 -u -c
'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-wiooruf_/horovod_ae372f3eb30d402293d967812539ca14/setup.py'"'"';
__file__='"'"'/tmp/pip-install-wiooruf_/horovod_ae372f3eb30d402293d967812539ca14/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install
--record /tmp/pip-record-t2m4n2ij/install-record.txt
--single-version-externally-managed --user --prefix= --compile
--install-headers /home/nishome/works/.local/include/python3.7m/horovod
cwd:
/tmp/pip-install-wiooruf_/horovod_ae372f3eb30d402293d967812539ca14/
Complete output (220 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/horovod
copying horovod/__init__.py -> build/lib.linux-x86_64-3.7/horovod
creating build/lib.linux-x86_64-3.7/horovod/torch
copying horovod/torch/sync_batch_norm.py ->
build/lib.linux-x86_64-3.7/horovod/torch
copying horovod/torch/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/torch
copying horovod/torch/functions.py ->
build/lib.linux-x86_64-3.7/horovod/torch
copying horovod/torch/compression.py ->
build/lib.linux-x86_64-3.7/horovod/torch
copying horovod/torch/mpi_ops.py ->
build/lib.linux-x86_64-3.7/horovod/torch
copying horovod/torch/optimizer.py ->
build/lib.linux-x86_64-3.7/horovod/torch
creating build/lib.linux-x86_64-3.7/horovod/keras
copying horovod/keras/elastic.py ->
build/lib.linux-x86_64-3.7/horovod/keras
copying horovod/keras/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/keras
copying horovod/keras/callbacks.py ->
build/lib.linux-x86_64-3.7/horovod/keras
creating build/lib.linux-x86_64-3.7/horovod/ray
copying horovod/ray/elastic.py -> build/lib.linux-x86_64-3.7/horovod/ray
copying horovod/ray/__init__.py -> build/lib.linux-x86_64-3.7/horovod/ray
copying horovod/ray/runner.py -> build/lib.linux-x86_64-3.7/horovod/ray
copying horovod/ray/utils.py -> build/lib.linux-x86_64-3.7/horovod/ray
copying horovod/ray/driver_service.py ->
build/lib.linux-x86_64-3.7/horovod/ray
creating build/lib.linux-x86_64-3.7/horovod/_keras
copying horovod/_keras/elastic.py ->
build/lib.linux-x86_64-3.7/horovod/_keras
copying horovod/_keras/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/_keras
copying horovod/_keras/callbacks.py ->
build/lib.linux-x86_64-3.7/horovod/_keras
creating build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/run_task.py ->
build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/mpi_run.py ->
build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/js_run.py ->
build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/gloo_run.py ->
build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/task_fn.py ->
build/lib.linux-x86_64-3.7/horovod/runner
copying horovod/runner/launch.py ->
build/lib.linux-x86_64-3.7/horovod/runner
creating build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/elastic.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/sync_batch_norm.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/gradient_aggregation.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/functions.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/compression.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/mpi_ops.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/util.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
copying horovod/tensorflow/gradient_aggregation_eager.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow
creating build/lib.linux-x86_64-3.7/horovod/mxnet
copying horovod/mxnet/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/mxnet
copying horovod/mxnet/functions.py ->
build/lib.linux-x86_64-3.7/horovod/mxnet
copying horovod/mxnet/mpi_ops.py ->
build/lib.linux-x86_64-3.7/horovod/mxnet
creating build/lib.linux-x86_64-3.7/horovod/common
copying horovod/common/elastic.py ->
build/lib.linux-x86_64-3.7/horovod/common
copying horovod/common/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/common
copying horovod/common/exceptions.py ->
build/lib.linux-x86_64-3.7/horovod/common
copying horovod/common/basics.py ->
build/lib.linux-x86_64-3.7/horovod/common
copying horovod/common/util.py ->
build/lib.linux-x86_64-3.7/horovod/common
creating build/lib.linux-x86_64-3.7/horovod/spark
copying horovod/spark/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/spark
copying horovod/spark/conf.py -> build/lib.linux-x86_64-3.7/horovod/spark
copying horovod/spark/runner.py ->
build/lib.linux-x86_64-3.7/horovod/spark
copying horovod/spark/mpi_run.py ->
build/lib.linux-x86_64-3.7/horovod/spark
copying horovod/spark/gloo_run.py ->
build/lib.linux-x86_64-3.7/horovod/spark
creating build/lib.linux-x86_64-3.7/horovod/torch/mpi_lib
copying horovod/torch/mpi_lib/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/torch/mpi_lib
creating build/lib.linux-x86_64-3.7/horovod/torch/mpi_lib_impl
copying horovod/torch/mpi_lib_impl/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/torch/mpi_lib_impl
creating build/lib.linux-x86_64-3.7/horovod/torch/elastic
copying horovod/torch/elastic/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/torch/elastic
copying horovod/torch/elastic/sampler.py ->
build/lib.linux-x86_64-3.7/horovod/torch/elastic
copying horovod/torch/elastic/state.py ->
build/lib.linux-x86_64-3.7/horovod/torch/elastic
creating build/lib.linux-x86_64-3.7/horovod/runner/task
copying horovod/runner/task/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/task
copying horovod/runner/task/task_service.py ->
build/lib.linux-x86_64-3.7/horovod/runner/task
creating build/lib.linux-x86_64-3.7/horovod/runner/http
copying horovod/runner/http/http_client.py ->
build/lib.linux-x86_64-3.7/horovod/runner/http
copying horovod/runner/http/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/http
copying horovod/runner/http/http_server.py ->
build/lib.linux-x86_64-3.7/horovod/runner/http
creating build/lib.linux-x86_64-3.7/horovod/runner/util
copying horovod/runner/util/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/util
copying horovod/runner/util/threads.py ->
build/lib.linux-x86_64-3.7/horovod/runner/util
copying horovod/runner/util/network.py ->
build/lib.linux-x86_64-3.7/horovod/runner/util
copying horovod/runner/util/remote.py ->
build/lib.linux-x86_64-3.7/horovod/runner/util
copying horovod/runner/util/lsf.py ->
build/lib.linux-x86_64-3.7/horovod/runner/util
copying horovod/runner/util/cache.py ->
build/lib.linux-x86_64-3.7/horovod/runner/util
creating build/lib.linux-x86_64-3.7/horovod/runner/common
copying horovod/runner/common/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common
creating build/lib.linux-x86_64-3.7/horovod/runner/driver
copying horovod/runner/driver/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/driver
copying horovod/runner/driver/driver_service.py ->
build/lib.linux-x86_64-3.7/horovod/runner/driver
creating build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/driver.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/rendezvous.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/registration.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/settings.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/constants.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/worker.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
copying horovod/runner/elastic/discovery.py ->
build/lib.linux-x86_64-3.7/horovod/runner/elastic
creating build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/host_hash.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/settings.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/config_parser.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/secret.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/codec.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/hosts.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/tiny_shell_exec.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/timeout.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/network.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/safe_shell_exec.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
copying horovod/runner/common/util/env.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/util
creating build/lib.linux-x86_64-3.7/horovod/runner/common/service
copying horovod/runner/common/service/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/service
copying horovod/runner/common/service/driver_service.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/service
copying horovod/runner/common/service/task_service.py ->
build/lib.linux-x86_64-3.7/horovod/runner/common/service
creating build/lib.linux-x86_64-3.7/horovod/tensorflow/keras
copying horovod/tensorflow/keras/elastic.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow/keras
copying horovod/tensorflow/keras/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow/keras
copying horovod/tensorflow/keras/callbacks.py ->
build/lib.linux-x86_64-3.7/horovod/tensorflow/keras
creating build/lib.linux-x86_64-3.7/horovod/spark/task
copying horovod/spark/task/task_info.py ->
build/lib.linux-x86_64-3.7/horovod/spark/task
copying horovod/spark/task/gloo_exec_fn.py ->
build/lib.linux-x86_64-3.7/horovod/spark/task
copying horovod/spark/task/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/spark/task
copying horovod/spark/task/task_service.py ->
build/lib.linux-x86_64-3.7/horovod/spark/task
copying horovod/spark/task/mpirun_exec_fn.py ->
build/lib.linux-x86_64-3.7/horovod/spark/task
creating build/lib.linux-x86_64-3.7/horovod/spark/torch
copying horovod/spark/torch/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/spark/torch
copying horovod/spark/torch/estimator.py ->
build/lib.linux-x86_64-3.7/horovod/spark/torch
copying horovod/spark/torch/remote.py ->
build/lib.linux-x86_64-3.7/horovod/spark/torch
copying horovod/spark/torch/util.py ->
build/lib.linux-x86_64-3.7/horovod/spark/torch
creating build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/tensorflow.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/bare.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/estimator.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/remote.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/util.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
copying horovod/spark/keras/optimizer.py ->
build/lib.linux-x86_64-3.7/horovod/spark/keras
creating build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/store.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/serialization.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/_namedtuple_fix.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/backend.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/constants.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/estimator.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/params.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/cache.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
copying horovod/spark/common/util.py ->
build/lib.linux-x86_64-3.7/horovod/spark/common
creating build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/rsh.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/host_discovery.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/rendezvous.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/__init__.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/driver_service.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/job_id.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
copying horovod/spark/driver/mpirun_rsh.py ->
build/lib.linux-x86_64-3.7/horovod/spark/driver
running build_ext
-- The CXX compiler identification is GNU 7.3.0
-- Check for working CXX compiler:
/home/nishome/works/Downlaod/gcc7/bin/g++
-- Check for working CXX compiler:
/home/nishome/works/Downlaod/gcc7/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build architecture flags: -mf16c -mavx -mfma
-- Using command /home/nishome/works/miniconda3/envs/qyx37/bin/python3
-- Found MPI_CXX: /usr/local/lib/libmpi.so
-- Found CUDA: /usr/local/cuda-10.2 (found version "10.2")
-- Linking against static NCCL library
-- Found NCCL: /usr/include
-- Determining NCCL version from the header file: /usr/include/nccl.h
-- NCCL_MAJOR_VERSION: 2
-- Found NCCL (include: /usr/include, library:
/usr/lib/x86_64-linux-gnu/libnccl_static.a)
CMake Error at cmake/Modules/FindMxnet.cmake:39 (message):
MXNet was found with mkl-dnn support but mkldnn header files are
missing.
Please, install MXNet with mkldnn header files.
Call Stack (most recent call first):
horovod/mxnet/CMakeLists.txt:12 (find_package)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
