rongzha1 edited a comment on issue #16891: Upgrading MKLDNN to 1.0 causes performance regression. URL: https://github.com/apache/incubator-mxnet/issues/16891#issuecomment-558656729 cpu test on both v1.5.x and v1.6.x mkldnn + openblas, but no regression issue was found. So can you try to use USE_BLAS=mkl as Taolv said above and test again? I have tried to use build.sh but failed for: CMake Error at simd/CMakeLists.txt:41 (enable_language): No CMAKE_ASM_NASM_COMPILER could be found. So for v1.5 and v1.6 I build use cmd: make -j USE_MKLDNN=1 USE_BLAS=openblas USE_GPERFTOOLS=0 and setting openblas include and lib directory. platform: skx-8180 1.5: [rongzha1@mlt-ace ds2_training_inference]$ cd mxnet_1.5/ [rongzha1@mlt-ace mxnet_1.5]$ ldd lib/libmxnet.so | grep open libopenblas.so.0 => /lib64/libopenblas.so.0 (0x00007f8db5ff9000) libopencv_highgui.so.2.4 => /lib64/libopencv_highgui.so.2.4 (0x00007f8dacdaf000) libopencv_imgproc.so.2.4 => /lib64/libopencv_imgproc.so.2.4 (0x00007f8dac931000) libopencv_core.so.2.4 => /lib64/libopencv_core.so.2.4 (0x00007f8dac4f7000) [rongzha1@mlt-ace mxnet_1.5]$ ldd lib/libmxnet.so | grep mkl libmklml_intel.so => /home/rongzha1/project/mxnet/ds2_training_inference/mxnet_1.5/lib/libmklml_intel.so (0x00007f9707c8d000) libmkldnn.so.0 => /home/rongzha1/project/mxnet/ds2_training_inference/mxnet_1.5/lib/libmkldnn.so.0 (0x00007f970671d000) (mxnet) [rongzha1@mlt-ace mxnet_1.5]$ ldd lib/libmxnet.so | grep omp libiomp5.so => /home/rongzha1/project/mxnet/ds2_training_inference/mxnet_1.5/lib/libiomp5.so (0x00007f75cbc42000) libXcomposite.so.1 => /lib64/libXcomposite.so.1 (0x00007f75c2647000) 1.6.x: [rongzha1@mlt-skx141 perf_regression]$ ldd lib/libmxnet.so | grep open libopenblas.so.0 => /usr/lib64/libopenblas.so.0 (0x00007fc101c03000) libopencv_highgui.so.2.4 => /usr/lib64/libopencv_highgui.so.2.4 (0x00007fc1004cf000) libopencv_imgproc.so.2.4 => /usr/lib64/libopencv_imgproc.so.2.4 (0x00007fc100051000) libopencv_core.so.2.4 => /usr/lib64/libopencv_core.so.2.4 (0x00007fc0ffc18000) [rongzha1@mlt-skx141 perf_regression]$ ldd lib/libmxnet.so | grep mkl libmkldnn.so.1 => /home/rongzha1/project/mxnet/ds2_training_inference/perf_regression/lib/libmkldnn.so.1 (0x00007f8378240000) [rongzha1@mlt-skx141 perf_regression]$ ldd lib/libmxnet.so | grep omp libgomp.so.1 => /usr/lib64/libgomp.so.1 (0x00007f1357b17000) libXcomposite.so.1 => /usr/lib64/libXcomposite.so.1 (0x00007f13509a1000) v1.5.x: 1 [21:43:26] src/io/iter_image_recordio_2.cc:172: ImageRecordIOParser2: data/cifar/train.rec, use 4 threads for decoding.. 2 [21:43:26] src/io/iter_image_recordio_2.cc:172: ImageRecordIOParser2: data/cifar/test.rec, use 4 threads for decoding.. 3 INFO:root:Epoch[0] Batch [0-50] Speed: 1668.60 samples/sec accuracy=0.273897 4 INFO:root:Epoch[0] Batch [50-100] Speed: 1699.64 samples/sec accuracy=0.380312 5 INFO:root:Epoch[0] Batch [100-150] Speed: 1692.57 samples/sec accuracy=0.425000 6 INFO:root:Epoch[0] Batch [150-200] Speed: 1696.67 samples/sec accuracy=0.444063 7 INFO:root:Epoch[0] Batch [200-250] Speed: 1698.27 samples/sec accuracy=0.465000 8 INFO:root:Epoch[0] Batch [250-300] Speed: 1693.87 samples/sec accuracy=0.497812 9 INFO:root:Epoch[0] Batch [300-350] Speed: 1698.26 samples/sec accuracy=0.505625 10 INFO:root:Epoch[0] Batch [350-400] Speed: 1691.21 samples/sec accuracy=0.520000 11 INFO:root:Epoch[0] Batch [400-450] Speed: 1694.42 samples/sec accuracy=0.538750 12 INFO:root:Epoch[0] Batch [450-500] Speed: 1693.73 samples/sec accuracy=0.576875 13 INFO:root:Epoch[0] Batch [500-550] Speed: 1688.67 samples/sec accuracy=0.579063 14 INFO:root:Epoch[0] Batch [550-600] Speed: 1686.91 samples/sec accuracy=0.585313 15 INFO:root:Epoch[0] Batch [600-650] Speed: 1691.39 samples/sec accuracy=0.605313 16 INFO:root:Epoch[0] Batch [650-700] Speed: 1693.22 samples/sec accuracy=0.612812 17 INFO:root:Epoch[0] Batch [700-750] Speed: 1692.32 samples/sec accuracy=0.603750 18 INFO:root:Epoch[0] Train-accuracy=0.511549 19 INFO:root:Epoch[0] Time cost=29.955 20 INFO:root:Epoch[0] Validation-accuracy=0.642317 v1.6.x: 1 [22:02:24] src/io/iter_image_recordio_2.cc:178: ImageRecordIOParser2: data/cifar/train.rec, use 4 threads for decoding.. 2 [22:02:25] src/io/iter_image_recordio_2.cc:178: ImageRecordIOParser2: data/cifar/test.rec, use 4 threads for decoding.. 3 [22:02:25] src/executor/graph_executor.cc:1979: Subgraph backend MKLDNN is activated. 4 /home/rongzha1/anaconda3/envs/mxnet/lib/python3.6/site-packages/scipy/__init__.py:115: UserWarning: Numpy 1.13.3 or above is required for this version of scipy (detected version 1.13.1) 5 UserWarning) 6 INFO:root:Epoch[0] Batch [0-50] Speed: 2119.74 samples/sec accuracy=0.280025 7 INFO:root:Epoch[0] Batch [50-100] Speed: 2161.65 samples/sec accuracy=0.392500 8 INFO:root:Epoch[0] Batch [100-150] Speed: 2145.79 samples/sec accuracy=0.425938 9 INFO:root:Epoch[0] Batch [150-200] Speed: 2145.72 samples/sec accuracy=0.448125 10 INFO:root:Epoch[0] Batch [200-250] Speed: 2158.03 samples/sec accuracy=0.461250 11 INFO:root:Epoch[0] Batch [250-300] Speed: 2151.47 samples/sec accuracy=0.498125 12 INFO:root:Epoch[0] Batch [300-350] Speed: 2157.60 samples/sec accuracy=0.515312 13 INFO:root:Epoch[0] Batch [350-400] Speed: 2133.91 samples/sec accuracy=0.530625 14 INFO:root:Epoch[0] Batch [400-450] Speed: 2143.35 samples/sec accuracy=0.545625 15 INFO:root:Epoch[0] Batch [450-500] Speed: 2153.24 samples/sec accuracy=0.577187 16 INFO:root:Epoch[0] Batch [500-550] Speed: 2154.20 samples/sec accuracy=0.577500 17 INFO:root:Epoch[0] Batch [550-600] Speed: 2151.89 samples/sec accuracy=0.580625 18 INFO:root:Epoch[0] Batch [600-650] Speed: 2162.29 samples/sec accuracy=0.596250 19 INFO:root:Epoch[0] Batch [650-700] Speed: 2161.74 samples/sec accuracy=0.609062 20 INFO:root:Epoch[0] Batch [700-750] Speed: 2156.80 samples/sec accuracy=0.597812 21 INFO:root:Epoch[0] Train-accuracy=0.512828 22 INFO:root:Epoch[0] Time cost=23.642 23 INFO:root:Epoch[0] Validation-accuracy=0.613455
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
