matteosal opened a new issue #20675: URL: https://github.com/apache/incubator-mxnet/issues/20675
I am building master branch (v2.0) from commit fabcd145cd496628791f9f2ea813048360ac33ca and 1.x branch (v1.6) from commit 6eec9da55c5096079355d1f1a5fa58dcf35d6752 and our internal benchmarks show a severe performance loss of v2.0 on Windows. Build settings for v2.0 are: ``` cmake -G"Visual Studio 15 2017 Win64" -T host=x64 ^ %= GENERAL FLAGS =% ^ -DCMAKE_INSTALL_PREFIX=%output_dir% ^ -DCMAKE_BUILD_TYPE=Release ^ -DCMAKE_SKIP_BUILD_RPATH=On ^ -DUSE_OPENCV=OFF ^ -DUSE_JEMALLOC=OFF ^ -DUSE_F16C=Off %= float16 support =%^ -DUSE_INT64_TENSOR_SIZE=ON ^ -DCMAKE_C_FLAGS="-D_WIN32" ^ -DCMAKE_CXX_FLAGS="-D_WIN32" ^ -DCMAKE_C_FLAGS_RELEASE="/MT -DNDEBUG" ^ -DCMAKE_CXX_FLAGS_RELEASE="/MT -DNDEBUG" ^ -DMXNET_FORCE_SHARED_CRT=OFF %= link statically to C runtime =%^ -DUSE_MXNET_LIB_NAMING=OFF ^ %= MATH BACKENDS =% ^ -DBLAS=MKL ^ -DUSE_LAPACK=OFF ^ -DUSE_ONEDNN=OFF ^ -DBLA_VENDOR="Intel10_64ilp" ^ -DBLA_STATIC=OFF ^ -DMKL_USE_SINGLE_DYNAMIC_LIBRARY=OFF ^ -DMKL_INCLUDE_DIR=%mkl_dir% ^ -DBLAS_LIBRARIES="%mkl_dir%/libiomp5md.lib;%mkl_dir%/mkl_core_dll.lib;%mkl_dir%/mkl_intel_ilp64_dll.lib;%mkl_dir%/mkl_intel_thread_dll.lib" ^ %= OPENMP =% ^ -DUSE_OPENMP=ON ^ -DOpenMP_C_FLAGS="-I%mkl_dir% /openmp:experimental" ^ -DOpenMP_CXX_FLAGS="-I%mkl_dir% /openmp:experimental" ^ -DOpenMP_C_LIB_NAMES="libiomp5" ^ -DOpenMP_CXX_LIB_NAMES="libiomp5" ^ -DOpenMP_libiomp5_LIBRARY="%mkl_dir%/libiomp5md.lib" ^ %= CUDA =% ^ -DUSE_CUDA=OFF ^ %mxnet_dir% ``` Build settings for v1.6 are: ``` cmake -G"Visual Studio 15 2017 Win64" -T host=x64 ^ %= GENERAL FLAGS =% ^ -DCMAKE_INSTALL_PREFIX=%output_dir% ^ -DCMAKE_BUILD_TYPE=Release ^ -DCMAKE_SKIP_BUILD_RPATH=On ^ -DUSE_OPENCV=OFF ^ -DUSE_JEMALLOC=OFF ^ -DUSE_F16C=Off %= float16 support =%^ -DCMAKE_C_FLAGS="-D_WIN32" ^ -DCMAKE_CXX_FLAGS="-D_WIN32" ^ -DCMAKE_C_FLAGS_RELEASE="/MT -DNDEBUG" ^ -DCMAKE_CXX_FLAGS_RELEASE="/MT -DNDEBUG" ^ -DUSE_MXNET_LIB_NAMING=OFF ^ %= MATH BACKENDS =% ^ -DBLAS=MKL ^ -DUSE_LAPACK=OFF ^ -DUSE_MKLDNN=OFF ^ -DUSE_MKL_IF_AVAILABLE=ON ^ -DMKL_USE_ILP64=ON ^ -DMKL_USE_SINGLE_DYNAMIC_LIBRARY=ON ^ -DMKL_ROOT=%mkl_dir% ^ -DMKL_INCLUDE_DIR=%mkl_dir% ^ -DMKL_RT_LIBRARY="%mkl_dir%/libiomp5md.lib;%mkl_dir%/mkl_rt.lib" ^ %= OPENMP =% ^ -DUSE_OPENMP=ON ^ -DOpenMP_C_FLAGS="-I%mkl_dir% /openmp" ^ -DOpenMP_C_LIB_NAMES="libiomp5" ^ -DOpenMP_CXX_FLAGS="-I%mkl_dir% /openmp" ^ -DOpenMP_CXX_LIB_NAMES="libiomp5" ^ -DOpenMP_libiomp5_LIBRARY="%mkl_dir%/libiomp5md.lib" ^ %= CUDA =% ^ -DUSE_CUDA=OFF ^ %mxnet_dir% ``` Some comments/observations: * Linux and Mac (built with analogous settings) do not suffer from this problem * MKL version is 2021.3 for all builds * Performance loss varies from about 1.4x slower to 10x slower on a set of models spanning from convnets to rnn to transformers, except for a small stack of 3x3 convolutions which is about 1.4x faster (but large convnets are 1.5x - 2x slower) * Removing the flag `/openmp:experimental` passed to `OpenMP_C_FLAGS` and `OpenMP_CXX_FLAGS` in the build settings of mxnet 2.0 doesn't have any effect, but removing the analogous `/openmp` flag from v1.6 DOES worsen its performance (although it still doesn't get as bad as v2.0). I would consider this to be a bug in the CMake code because one shouldn't need extra stuff besides `USE_OPENMP=ON` to fully leverage openMP. Also, Linux and Mac need the analogous of those flags (`-fopenmp` on Linux, `-Xpreprocessor -fopenmp` on Mac) on both v1.6 and v2.0 to get optimal performance. This is not evidence that the v2.0 slowdown problem on Windows is due to openMP but I think it's worth mentioning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@mxnet.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@mxnet.apache.org For additional commands, e-mail: issues-h...@mxnet.apache.org