[GitHub] [incubator-mxnet] cjolivier01 edited a comment on issue #17641: OpenMP Error
cjolivier01 edited a comment on issue #17641: OpenMP Error URL: https://github.com/apache/incubator-mxnet/issues/17641#issuecomment-589812034 btw, cmake files have min cmake at 3.13, but default 18.04 cmake install is cmake 3.10. Does anyone know what the deal is with 3.13? Ubuntu 18.04 is a pretty widely-used release... I changed back to 3.10 and it seems to build ok. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] cjolivier01 edited a comment on issue #17641: OpenMP Error
cjolivier01 edited a comment on issue #17641: OpenMP Error URL: https://github.com/apache/incubator-mxnet/issues/17641#issuecomment-589836549 Actually, i don;t see this behavior when it does pull in mkl/pulling in the other omp (this is Ubuntu 18.04): ``` [chriso@chriso-ripper:~/src/mxnet/build (master)]ldd libmxnet.so linux-vdso.so.1 (0x7ffcbdf3b000) libmkl_rt.so => /opt/intel/mkl/lib/intel64/libmkl_rt.so (0x7fb399dd8000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7fb399bd) libomp.so => /home/chriso/src/mxnet/build/3rdparty/openmp/runtime/src/libomp.so (0x7fb3998ea000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7fb3996e6000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7fb3994c7000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x7fb39913e000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7fb398da) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7fb398b88000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7fb398797000) /lib64/ld-linux-x86-64.so.2 (0x7fb3a078c000) ``` I don;t show libmkl_rt.so pulling in libiomp5: ``` [chriso@chriso-ripper:~/src/mxnet/build (master)]ldd /opt/intel/mkl/lib/intel64/libmkl_rt.so linux-vdso.so.1 (0x7ffd6c5cc000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7fc85058d000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7fc85019c000) /lib64/ld-linux-x86-64.so.2 (0x7fc850e71000) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] cjolivier01 edited a comment on issue #17641: OpenMP Error
cjolivier01 edited a comment on issue #17641: OpenMP Error URL: https://github.com/apache/incubator-mxnet/issues/17641#issuecomment-590521574 I can reproduce now. However, even if I remove the openmp build in CMakeLists.txt and build with clang, I get that warning, since it pulls in libomp from clang (I am using clang8): ```bash [chriso@chriso-ripper:~/src/mxnet/build (master)]ldd libmxnet.so linux-vdso.so.1 (0x7ffd55ab4000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f084c8cd000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f084c6ae000) libmkl_rt.so => /opt/intel/mkl/lib/intel64/libmkl_rt.so (0x7f084bfce000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f084bdc6000) liblapack.so.3 => /usr/lib/x86_64-linux-gnu/liblapack.so.3 (0x7f084b54) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x7f084b1b7000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f084ae19000) libomp.so => /usr/local/lib/libomp.so (0x7f084ab56000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f084a93e000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f084a54d000) /lib64/ld-linux-x86-64.so.2 (0x7f0855365000) libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x7f08482a7000) libgfortran.so.4 => /usr/lib/x86_64-linux-gnu/libgfortran.so.4 (0x7f0847ec8000) libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x7f0847c88000) ``` ```bash [chriso@chriso-ripper:~/src/mxnet (master)]PYTHONPATH=$(pwd)/python python3 test.py OMP: Error #15: Initializing libiomp5.so, but found libomp.so already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/. Aborted (core dumped) ``` ```[chriso@chriso-ripper:~/src/mxnet/build (master)]ldd libmxnet.so linux-vdso.so.1 (0x7ffd55ab4000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f084c8cd000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f084c6ae000) libmkl_rt.so => /opt/intel/mkl/lib/intel64/libmkl_rt.so (0x7f084bfce000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f084bdc6000) liblapack.so.3 => /usr/lib/x86_64-linux-gnu/liblapack.so.3 (0x7f084b54) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x7f084b1b7000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f084ae19000) libomp.so => /usr/local/lib/libomp.so (0x7f084ab56000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f084a93e000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f084a54d000) /lib64/ld-linux-x86-64.so.2 (0x7f0855365000) libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x7f08482a7000) libgfortran.so.4 => /usr/lib/x86_64-linux-gnu/libgfortran.so.4 (0x7f0847ec8000) libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x7f0847c88000) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] cjolivier01 edited a comment on issue #17641: OpenMP Error
cjolivier01 edited a comment on issue #17641: OpenMP Error URL: https://github.com/apache/incubator-mxnet/issues/17641#issuecomment-590521574 I can reproduce now. However, even if I remove the openmp build in CMakeLists.txt and build with clang, I get that warning, since it pulls in libomp from clang (I am using clang8): ```bash [chriso@chriso-ripper:~/src/mxnet/build (master)]ldd libmxnet.so linux-vdso.so.1 (0x7ffd55ab4000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f084c8cd000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f084c6ae000) libmkl_rt.so => /opt/intel/mkl/lib/intel64/libmkl_rt.so (0x7f084bfce000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f084bdc6000) liblapack.so.3 => /usr/lib/x86_64-linux-gnu/liblapack.so.3 (0x7f084b54) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x7f084b1b7000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f084ae19000) libomp.so => /usr/local/lib/libomp.so (0x7f084ab56000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f084a93e000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f084a54d000) /lib64/ld-linux-x86-64.so.2 (0x7f0855365000) libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x7f08482a7000) libgfortran.so.4 => /usr/lib/x86_64-linux-gnu/libgfortran.so.4 (0x7f0847ec8000) libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x7f0847c88000) ``` ```bash [chriso@chriso-ripper:~/src/mxnet (master)]PYTHONPATH=$(pwd)/python python3 test.py OMP: Error #15: Initializing libiomp5.so, but found libomp.so already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/. Aborted (core dumped) ``` ```[chriso@chriso-ripper:~/src/mxnet/build (master)]ldd libmxnet.so linux-vdso.so.1 (0x7ffd55ab4000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f084c8cd000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f084c6ae000) libmkl_rt.so => /opt/intel/mkl/lib/intel64/libmkl_rt.so (0x7f084bfce000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f084bdc6000) liblapack.so.3 => /usr/lib/x86_64-linux-gnu/liblapack.so.3 (0x7f084b54) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x7f084b1b7000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f084ae19000) libomp.so => /usr/local/lib/libomp.so (0x7f084ab56000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f084a93e000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f084a54d000) /lib64/ld-linux-x86-64.so.2 (0x7f0855365000) libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x7f08482a7000) libgfortran.so.4 => /usr/lib/x86_64-linux-gnu/libgfortran.so.4 (0x7f0847ec8000) libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x7f0847c88000) ``` so this seems like a systemic problem with mkl to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] cjolivier01 edited a comment on issue #17641: OpenMP Error
cjolivier01 edited a comment on issue #17641: OpenMP Error URL: https://github.com/apache/incubator-mxnet/issues/17641#issuecomment-590521574 I can reproduce now. However, even if I remove the openmp build in CMakeLists.txt and build with clang, I get that warning, since it pulls in libomp from clang (I am using clang8): ```bash [chriso@chriso-ripper:~/src/mxnet (master)]PYTHONPATH=$(pwd)/python python3 test.py OMP: Error #15: Initializing libiomp5.so, but found libomp.so already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/. Aborted (core dumped) ``` ``` [chriso@chriso-ripper:~/src/mxnet/build (master)]ldd libmxnet.so linux-vdso.so.1 (0x7ffd55ab4000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f084c8cd000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f084c6ae000) libmkl_rt.so => /opt/intel/mkl/lib/intel64/libmkl_rt.so (0x7f084bfce000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f084bdc6000) liblapack.so.3 => /usr/lib/x86_64-linux-gnu/liblapack.so.3 (0x7f084b54) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x7f084b1b7000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f084ae19000) libomp.so => /usr/local/lib/libomp.so (0x7f084ab56000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x7f084a93e000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f084a54d000) /lib64/ld-linux-x86-64.so.2 (0x7f0855365000) libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x7f08482a7000) libgfortran.so.4 => /usr/lib/x86_64-linux-gnu/libgfortran.so.4 (0x7f0847ec8000) libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x7f0847c88000) ``` so this seems like a systemic problem with mkl to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] cjolivier01 edited a comment on issue #17641: OpenMP Error
cjolivier01 edited a comment on issue #17641: OpenMP Error URL: https://github.com/apache/incubator-mxnet/issues/17641#issuecomment-590527708 Another thing to note is that ompenmp with llvm also installs libgomp.so as a symlink to libomp.so, so there's a good chance that libomp.so will be loaded no matter what, depending upon whether a system had clang/openmp installed at all and where that is in the link order. So unless I am missing some clever logic, what mkl doing with its dynamic loading is a cause for concern. Also of note, clang seems to also put a symlink to libiomp5: ```bash [chriso@chriso-ripper:~/src/mxnet (master)]ls -l /usr/local/lib/lib*omp*.so* lrwxrwxrwx 1 root root 9 Feb 20 14:37 /usr/local/lib/libgomp.so -> libomp.so lrwxrwxrwx 1 root root 9 Feb 20 14:37 /usr/local/lib/libiomp5.so -> libomp.so -rw-r--r-- 1 root root 953376 Feb 20 14:36 /usr/local/lib/libomp.so -rw-r--r-- 1 root root 66072 Feb 20 14:36 /usr/local/lib/libomptarget.so ``` So it seems link order is important? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] cjolivier01 edited a comment on issue #17641: OpenMP Error
cjolivier01 edited a comment on issue #17641: OpenMP Error URL: https://github.com/apache/incubator-mxnet/issues/17641#issuecomment-590527708 Another thing to note is that ompenmp with llvm also installs libgomp.so as a symlink to libomp.so, so there's a good chance that libomp.so will be loaded no matter what, depending upon whether a system had clang/openmp installed at all and where that is in the link order. So unless I am missing some clever logic, what mkl doing with its dynamic loading is a cause for concern. Also of note, clang seems to also put a symlink to libiomp5 (in addition to libgomp): ```bash [chriso@chriso-ripper:~/src/mxnet (master)]ls -l /usr/local/lib/lib*omp*.so* lrwxrwxrwx 1 root root 9 Feb 20 14:37 /usr/local/lib/libgomp.so -> libomp.so lrwxrwxrwx 1 root root 9 Feb 20 14:37 /usr/local/lib/libiomp5.so -> libomp.so -rw-r--r-- 1 root root 953376 Feb 20 14:36 /usr/local/lib/libomp.so -rw-r--r-- 1 root root 66072 Feb 20 14:36 /usr/local/lib/libomptarget.so ``` So it seems link order is important? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] cjolivier01 edited a comment on issue #17641: OpenMP Error
cjolivier01 edited a comment on issue #17641: OpenMP Error URL: https://github.com/apache/incubator-mxnet/issues/17641#issuecomment-590997673 > -DMKL_USE_STATIC_LIBS=ON How broken? When I built with clang, it seems to run ok. Nevermind, scratch that, it wasn;t finding mkl This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] cjolivier01 edited a comment on issue #17641: OpenMP Error
cjolivier01 edited a comment on issue #17641: OpenMP Error URL: https://github.com/apache/incubator-mxnet/issues/17641#issuecomment-591207663 > @leezu what's the error of statically linking MKL libraries? for me it was some link error on some secondary thing like cpp unit test or something like that. libmxnet.so built successfully and the test script was successful. probably not too hard to fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] cjolivier01 edited a comment on issue #17641: OpenMP Error
cjolivier01 edited a comment on issue #17641: OpenMP Error URL: https://github.com/apache/incubator-mxnet/issues/17641#issuecomment-591513111 i will post a pr in the next day or two that addresses this and also clang issue as well as transitive omp dependencies which may also cause the error due to mkl behaving foolishly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
