Hi Pedro, I'm looking at this case, and using the script of "incubator-mxnet/example/image-classification/train_cifar10.py" to get the timing data, but seems there's not much difference between mxnet 1.4.1.rc0 and 1.5.0.rc1 on C5.18xlarge.
Not sure if there's any difference in the python script, can you point me the link to get your script (cifar10.py)? Or you can also have a try with MXNet's script (train_cifar10.py) and see the performance. Here's the command I used to collect the time: python train_cifar10.py --num-epoch=5 1) 1.5.0.rc1 (4d9667121ae6fb643f2a02ab15e25231ed756cde) real 9m4.880s user 333m13.340s sys 14m36.100s 2) 1.4.1.rc0 (1a7199691f5cbc6012bb53eecbf884bed5ae6590) real 9m2.155s user 329m37.092s sys 16m8.668s -Ciyong -----Original Message----- From: Pedro Larroy [mailto:pedro.larroy.li...@gmail.com] Sent: Wednesday, June 26, 2019 6:28 AM To: dev@mxnet.incubator.apache.org Cc: d...@mxnet.apache.org Subject: Re: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc1 Hi these were my build flags and system info: --- # CMake configuration USE_CUDA: "OFF" # Build with CUDA support USE_OLDCMAKECUDA: "OFF" # Build with old cmake cuda USE_NCCL: "OFF" # Use NVidia NCCL with CUDA USE_OPENCV: "ON" # Build with OpenCV support USE_OPENMP: "ON" # Build with Openmp support USE_CUDNN: "ON" # Build with cudnn support) # one could set CUDNN_ROOT for search path USE_SSE: "ON" # Build with x86 SSE instruction support IF NOT ARM USE_F16C: "ON" # Build with x86 F16C instruction support) # autodetects support if "ON" USE_LAPACK: "ON" # Build with lapack support USE_MKL_IF_AVAILABLE: "ON" # Use MKL if found USE_MKLML_MKL: "ON" # Use MKLDNN variant of MKL (if MKL found) IF USE_MKL_IF_AVAILABLE AND (NOT APPLE) USE_MKLDNN: "ON" # Use MKLDNN variant of MKL (if MKL found) IF USE_MKL_IF_AVAILABLE AND (NOT APPLE) USE_OPERATOR_TUNING: "ON" # Enable auto-tuning of operators IF NOT MSVC USE_GPERFTOOLS: "ON" # Build with GPerfTools support (if found) USE_JEMALLOC: "ON" # Build with Jemalloc support USE_PROFILER: "ON" # Build with Profiler support USE_DIST_KVSTORE: "OFF" # Build with DIST_KVSTORE support USE_PLUGINS_WARPCTC: "OFF" # Use WARPCTC Plugins USE_PLUGIN_CAFFE: "OFF" # Use Caffe Plugin USE_CPP_PACKAGE: "OFF" # Build C++ Package USE_MXNET_LIB_NAMING: "ON" # Use MXNet library naming conventions. USE_GPROF: "OFF" # Compile with gprof (profiling) flag USE_CXX14_IF_AVAILABLE: "OFF" # Build with C++14 if the compiler supports it USE_VTUNE: "OFF" # Enable use of Intel Amplifier XE (VTune)) # one could set VTUNE_ROOT for search path ENABLE_CUDA_RTC: "ON" # Build with CUDA runtime compilation support BUILD_CPP_EXAMPLES: "ON" # Build cpp examples INSTALL_EXAMPLES: "OFF" # Install the example source files. USE_SIGNAL_HANDLER: "ON" # Print stack traces on segfaults. USE_TENSORRT: "OFF" # Enable infeference optimization with TensorRT. USE_ASAN: "OFF" # Enable Clang/GCC ASAN sanitizers. ENABLE_TESTCOVERAGE: "OFF" # Enable compilation with test coverage metric output CMAKE_BUILD_TYPE: "Release" CMAKE_CUDA_COMPILER_LAUNCHER: "ccache" CMAKE_C_COMPILER_LAUNCHER: "ccache" CMAKE_CXX_COMPILER_LAUNCHER: "ccache" commit 4d9667121ae6fb643f2a02ab15e25231ed756cde (HEAD, tag: 1.5.0.rc1, upstream/v1.5.x) commit 1a7199691f5cbc6012bb53eecbf884bed5ae6590 (HEAD, tag: 1.4.1.rc0, upstream/v1.4.x) curl http://169.254.169.254/latest/meta-data/instance-type c5d.18xlarge Version : 3.6.7 Compiler : GCC 8.2.0 Build : ('default', 'Oct 22 2018 11:32:17') Arch : ('64bit', 'ELF') ------------Pip Info----------- Version : 19.1.1 Directory : /home/piotr/mxnet_1.5/py3_venv/lib/python3.6/site-packages/pip ----------MXNet Info----------- Version : 1.5.0 Directory : /home/piotr/mxnet_1.5/python/mxnet Hashtag not found. Not installed from pre-built package. ----------System Info---------- Platform : Linux-4.15.0-1035-aws-x86_64-with-Ubuntu-18.04-bionic system : Linux node : ip-172-31-63-171 release : 4.15.0-1035-aws version : #37-Ubuntu SMP Mon Mar 18 16:15:14 UTC 2019 ----------Hardware Info---------- machine : x86_64 processor : x86_64 Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 72 On-line CPU(s) list: 0-71 Thread(s) per core: 2 Core(s) per socket: 18 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz Stepping: 4 CPU MHz: 1326.446 BogoMIPS: 6000.00 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 25344K NUMA node0 CPU(s): 0-17,36-53 NUMA node1 CPU(s): 18-35,54-71 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke ----------Network Test---------- ----------Python Info---------- Version : 3.6.7 Compiler : GCC 8.2.0 Build : ('default', 'Oct 22 2018 11:32:17') Arch : ('64bit', 'ELF') ------------Pip Info----------- Version : 19.1.1 Directory : /home/piotr/mxnet_1.4/py3_venv/lib/python3.6/site-packages/pip ----------MXNet Info----------- Version : 1.4.1 Directory : /home/piotr/mxnet_1.4/python/mxnet Hashtag not found. Not installed from pre-built package. ----------System Info---------- Platform : Linux-4.15.0-1035-aws-x86_64-with-Ubuntu-18.04-bionic system : Linux node : ip-172-31-63-171 release : 4.15.0-1035-aws version : #37-Ubuntu SMP Mon Mar 18 16:15:14 UTC 2019 ----------Hardware Info---------- machine : x86_64 processor : x86_64 Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 72 On-line CPU(s) list: 0-71 Thread(s) per core: 2 Core(s) per socket: 18 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz Stepping: 4 CPU MHz: 1223.344 BogoMIPS: 6000.00 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 25344K NUMA node0 CPU(s): 0-17,36-53 NUMA node1 CPU(s): 18-35,54-71 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke ----------Network Test---------- On Tue, Jun 25, 2019 at 2:35 PM Pedro Larroy <pedro.larroy.li...@gmail.com> wrote: > > I did a training of cifar10 in CPU and seems there's some regressions > in the range of 7% increase of training time against 1.4.1: > > (py3_venv) piotr@ip-172-31-63-171:0:~/deeplearning-benchmark/dawnbench > (master)+$ time python cifar10.py --epochs 5 > real 11m30.388s > user 417m7.766s > sys 16m57.315s > > VS 1.4.1: > real 10m41.994s > user 392m40.646s > sys 12m30.601s > > > On Thu, Jun 20, 2019 at 10:15 PM Lai Wei <roywei...@gmail.com> wrote: > > > > Hi Anirudh, > > > > Thanks for jumping into this quickly, I followed up on the issue. > > > > I was meant for sockeye developer/maintainers to help setup nightly > > tests and raise issues early. > > > > Thanks! > > > > On Fri, Jun 21, 2019 at 10:10 AM Haibin Lin > > <haibin.lin....@gmail.com> > > wrote: > > > > > In GluonNLP we are testing with MXNET nightly build for each PR, > > > and we did find some MXNet related issue caught by the CI. > > > I recommend other toolkits also add integration tests with MXNet nightly. > > > It helps identify issues early. > > > > > > Best, > > > Haibin > > > > > > On Thu, Jun 20, 2019 at 18:52 Zhao, Patric <patric.z...@intel.com> wrote: > > > > > > > Thanks to raise the issue and we will take a look ASAP. > > > > > > > > The downstream cases is not in the MXNet CI so it's hard to > > > > catch the potential bugs or performance degradation for MXNet > > > > developers. > > > > > > > > In the future, I suggest adding the major downstream test cases, > > > > like > > > from > > > > sockeye, GluonNLP, GLuonCV, DGL, Gluon-TS, into the nightly test. > > > > If it's still too heavy, maybe testing it weekly or monthly :) > > > > > > > > Thanks, > > > > > > > > --Patric > > > > > > > > > -----Original Message----- > > > > > From: Anirudh Subramanian [mailto:anirudh2...@gmail.com] > > > > > Sent: Friday, June 21, 2019 9:31 AM > > > > > To: dev@mxnet.incubator.apache.org > > > > > Cc: d...@mxnet.apache.org > > > > > Subject: Re: [VOTE] Release Apache MXNet (incubating) version > > > > > 1.5.0.rc1 > > > > > > > > > > Hi Lai, > > > > > > > > > > I have opened an issue: > > > > > https://github.com/apache/incubator-mxnet/issues/15297 > > > > > I came to know about this issue only today and I have not been > > > monitoring > > > > > sockeye. > > > > > I jumped onto this issue to make sure it wasn't caused by the > > > > > dlpack > > > > changes. > > > > > Also, I don't think sockeye CI checks against master, it is > > > > > using > > > 1.4.1. > > > > > > > > > > Anirudh > > > > > > > > > > > > > > > On Thu, Jun 20, 2019 at 6:17 PM Lai Wei <roywei...@gmail.com> wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > Could you share which test failed and what’s the crash? How > > > > > > to reproduce it? > > > > > > > > > > > > I was able to install sockeye and run all tests passed. > > > > > > Using python setup.py test > > > > > > > > > > > > I have tested both nightly pip package and 1.5.0.rc1 > > > > > > > > > > > > It would be great to create an issue with reproducible steps > > > > > > and move the discussion there. > > > > > > > > > > > > Also I see sockeye nightly build[1] has been failing for > > > > > > some time, > > > if > > > > > > it’s due to MXNet change, please raise this early so we can > > > > > > track and solve it in time rather than block the release during > > > > > > vote time. > > > > > > > > > > > > [1] https://travis-ci.org/awslabs/sockeye > > > > > > > > > > > > > > > > > > On Fri, Jun 21, 2019 at 7:01 AM Anirudh Subramanian > > > > > > <anirudh2...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > I was able to reproduce a crash with the commit > > > > > > > 09202f7f261954383aa387144524d38f83f18d06 but not with the > > > > > > > commit a862270beb2d796c1ba311183f7f4a766a18ad6c. > > > > > > > > > > > > > > Anirudh > > > > > > > > > > > > > > On Thu, Jun 20, 2019 at 3:53 PM Lai Wei > > > > > > > <roywei...@gmail.com> > > > wrote: > > > > > > > > > > > > > > > Hi Przemyslaw, > > > > > > > > > > > > > > > > Is there an issue with more details to track the problem? > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 21, 2019 at 6:04 AM Przemysław Trędak > > > > > > > > <ptre...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > -1 > > > > > > > > > > > > > > > > > > There is a crash in sockeye unit test (python setup.py > > > > > > > > > test) observed starting with nightly 1.5 build from > > > > > > > > > 6/13 and still occuring in > > > > > > > 1.5rc1. I > > > > > > > > > don't yet have the exact commit that is responsible > > > > > > > > > for it, but it is either > > > > > > > > > a862270beb2d796c1ba311183f7f4a766a18ad6c (dlpack > > > > > > > > > related) or > > > > > > > > > 09202f7f261954383aa387144524d38f83f18d06 (cached op > > > > > optimization). > > > > > > > > > > > > > > > > > > On 2019/06/20 06:36:22, Lai Wei <roywei...@gmail.com> wrote: > > > > > > > > > > Dear MXNet community, > > > > > > > > > > > > > > > > > > > > This is the 3-day vote to release Apache MXNet > > > > > > > > > > (incubating) version > > > > > > > > > 1.5.0. > > > > > > > > > > Voting on dev@ will start June 19, 23:59:59(PST) > > > > > > > > > > and close > > > on > > > > > > June > > > > > > > > 22, > > > > > > > > > > 23:59:59. > > > > > > > > > > > > > > > > > > > > 1) Link to release notes: > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+No > > > te > > > > > > > s > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2) Link to release candidate: > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-mxnet/releases/tag/1.5.0.r > > > > > > > > > > c1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3) Link to source and signatures on apache dist server: > > > > > > > > > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.5.0.r > > > > > > > > > > c1/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please remember to TEST first before voting accordingly: > > > > > > > > > > > > > > > > > > > > +1 = approve > > > > > > > > > > +0 = no opinion > > > > > > > > > > -1 = disapprove (provide reason) > > > > > > > > > > -- > > > > > > > > > > Best Regards > > > > > > > > > > > > > > > > > > > > Lai > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Best Regards > > > > > > > > > > > > > > > > Lai > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best Regards > > > > > > > > > > > > Lai > > > > > > > > > > > > > > > -- > > Best Regards > > > > Lai