Re: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc1
Hi all,
I ran the same cifar10.py script as Pedro, but for 20 epochs. Considering
the first 10 epochs for warm-up, I averaged time per epoch for the last 10
epochs.
With MXNet 1.4.1 average time is 164.23 s
With MXNet 1.5.0 average time is 174.59 s (~6.3% regression)
For a second data point, I ran Gluon speed test benchmark script -
https://github.com/apache/incubator-mxnet/blob/master/benchmark/python/gluon/benchmark_gluon.py
using the following command:
python3 benchmark_gluon.py --model 'resnet152_v2' --batch-size 128
--num-batches 200 --type 'training'
I got the following speeds:
With MXNet 1.4.1, average speed is 25.677534 img/s
With MXNet 1.5.0, average speed is 25.082130 img/s (~2.3% regression)
Note:
For 1.4.1 version, I used pip install mxnet-mkl==1.4.1
For 1.5.0 version, I used pip install mxnet-mkl==1.5.0b20190619 which
corresponds to commit# ccbbf6b4b76ea536a6583c99497c83b65a20817b which is
behind 1.5.x branch by 4 commits
Best,
Manu
On 6/27/19, 3:37 PM, "sandeep krishnamurthy"
wrote:
Hello Ciyong/Pedro,
Ran operator benchmarks on 1.4.1 and 1.5.0.rc2. (Not complete, doesn’t
cover all MXNet operators, not presented in best possible way, still
WIP)
https://gist.github.com/sandeep-krishnamurthy/e0a2be893c8c4d484390c9c8813bdf50
Following operators looks slower in 1.5 compared to 1.4.1:
- BatchNorm
- Pooling
- FullyConnected
- batch_dot
- Dot
- broadcast_mul
- log_softmax
and few other operators
Also, several operators runs a lot faster on 1.5 compared to 1.4.1. For
example - Convolution, flatten, elementwise operators etc. So I see that
likely few operators have regressed noticeably, however, due to other
operator performance improvements, the end effect is not that
significant
hiding a lot of regression. We need more detailed analysis per operator
performance. We will not be able to do this for current release, we
should
have a more concrete way to determining such performance regression
before
next release.
Setup:
1.5 => Build from source (head of 1.5.rc2 tag), built with MKLDNN
1.4.1 => PyPi mxnet-mkl==1.4.1
Machine: C5.18X
No explicit environment variable were set
Operator benchmark code -
https://github.com/apache/incubator-mxnet/tree/master/benchmark/opperf
Best,
Sandeep
On Thu, Jun 27, 2019 at 10:42 AM Pedro Larroy <
pedro.larroy.li...@gmail.com>
wrote:
> I will try to run a few benchmarks in a bare metal instance tonight to
> remove virtualization variance for the measurements and provide some
> numbers.
>
> Please propose a set of models / examples that would be desirable to
> run before the release and provide a link to an easy to run script
> with instructions so we can validate the release better.
>
> Thank you.
>
> On Thu, Jun 27, 2019 at 10:01 AM Lai Wei wrote:
> >
> > Dear @dev,
> >
> > I m cancelling the vote for cached op fix:
> >
> > https://github.com/apache/incubator-mxnet/pull/15298
> >
> > As for the possible cpu training regression, it looks like not a
blocker
> > for now.
> >
> > I will start a new rc2 vote, please help to validate.
> >
> > Thanks!
> >
> >
> > On Thu, Jun 27, 2019 at 10:06 PM Chen, Ciyong
> wrote:
> >
> > > Hi Pedro,
> > >
> > > I was able to reproduced the similar result (v1.5 is ~%5.6 slower
than
> > > v1.4, I was using 18 cores for computing) with your script on
> C5.18xlarge.
> > > But need to bind the cores with below command when running the
script,
> > > (without setting the env variables, I got a close time (<1%) with
v1.5
> and
> > > v1.4)
> > > export
KMP_AFFINITY=granularity=fine,noduplicates,compact,1,0
> > > export OMP_NUM_THREADS=18
> > >
> > > Did you set any env variables during running?
> > >
> > > The performance result I got as below:
> > > 1) 1.4.1.rc0 (1a7199691f5cbc6012bb53eecbf884bed5ae6590)
> > > real12m10.856s
> > > user234m49.576s
> > > sys 4m38.044s
> > >
> > > 2) 1.5.0.rc1 (4d9667121ae6fb643f2a02ab15e25231ed756cde)
> > > real12m52.140s
> > > user246m30.740s
> > > sys 5m8.188s
> > >
> > > As I looked at the profiling data, most of the ops have same perf
> between
> > > v1.4 and v1.5. But some ops like " _backward_BatchNorm" and
"Pooling"
> is
> > > ~1.37x slower on v1.5 compared with v1.4.
> > > Will do further analysis on these ops.
> > >
> > > Here's the hardware/OS info from my side:
> > > --Python Info--
> > > Version : 3.6.8
> > > Compiler : GCC 7.3.0
> > > Build: ('default', 'Dec 30 2018 01:22:34')
> > > Arch : ('64bit', '')
> > > Pip Info---
> > > Version : 19.0.3
> > > Directory:
> > >
Re: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc1
Hello Ciyong/Pedro,
Ran operator benchmarks on 1.4.1 and 1.5.0.rc2. (Not complete, doesn’t
cover all MXNet operators, not presented in best possible way, still WIP)
https://gist.github.com/sandeep-krishnamurthy/e0a2be893c8c4d484390c9c8813bdf50
Following operators looks slower in 1.5 compared to 1.4.1:
- BatchNorm
- Pooling
- FullyConnected
- batch_dot
- Dot
- broadcast_mul
- log_softmax
and few other operators
Also, several operators runs a lot faster on 1.5 compared to 1.4.1. For
example - Convolution, flatten, elementwise operators etc. So I see that
likely few operators have regressed noticeably, however, due to other
operator performance improvements, the end effect is not that significant
hiding a lot of regression. We need more detailed analysis per operator
performance. We will not be able to do this for current release, we should
have a more concrete way to determining such performance regression before
next release.
Setup:
1.5 => Build from source (head of 1.5.rc2 tag), built with MKLDNN
1.4.1 => PyPi mxnet-mkl==1.4.1
Machine: C5.18X
No explicit environment variable were set
Operator benchmark code -
https://github.com/apache/incubator-mxnet/tree/master/benchmark/opperf
Best,
Sandeep
On Thu, Jun 27, 2019 at 10:42 AM Pedro Larroy
wrote:
> I will try to run a few benchmarks in a bare metal instance tonight to
> remove virtualization variance for the measurements and provide some
> numbers.
>
> Please propose a set of models / examples that would be desirable to
> run before the release and provide a link to an easy to run script
> with instructions so we can validate the release better.
>
> Thank you.
>
> On Thu, Jun 27, 2019 at 10:01 AM Lai Wei wrote:
> >
> > Dear @dev,
> >
> > I m cancelling the vote for cached op fix:
> >
> > https://github.com/apache/incubator-mxnet/pull/15298
> >
> > As for the possible cpu training regression, it looks like not a blocker
> > for now.
> >
> > I will start a new rc2 vote, please help to validate.
> >
> > Thanks!
> >
> >
> > On Thu, Jun 27, 2019 at 10:06 PM Chen, Ciyong
> wrote:
> >
> > > Hi Pedro,
> > >
> > > I was able to reproduced the similar result (v1.5 is ~%5.6 slower than
> > > v1.4, I was using 18 cores for computing) with your script on
> C5.18xlarge.
> > > But need to bind the cores with below command when running the script,
> > > (without setting the env variables, I got a close time (<1%) with v1.5
> and
> > > v1.4)
> > > export KMP_AFFINITY=granularity=fine,noduplicates,compact,1,0
> > > export OMP_NUM_THREADS=18
> > >
> > > Did you set any env variables during running?
> > >
> > > The performance result I got as below:
> > > 1) 1.4.1.rc0 (1a7199691f5cbc6012bb53eecbf884bed5ae6590)
> > > real12m10.856s
> > > user234m49.576s
> > > sys 4m38.044s
> > >
> > > 2) 1.5.0.rc1 (4d9667121ae6fb643f2a02ab15e25231ed756cde)
> > > real12m52.140s
> > > user246m30.740s
> > > sys 5m8.188s
> > >
> > > As I looked at the profiling data, most of the ops have same perf
> between
> > > v1.4 and v1.5. But some ops like " _backward_BatchNorm" and "Pooling"
> is
> > > ~1.37x slower on v1.5 compared with v1.4.
> > > Will do further analysis on these ops.
> > >
> > > Here's the hardware/OS info from my side:
> > > --Python Info--
> > > Version : 3.6.8
> > > Compiler : GCC 7.3.0
> > > Build: ('default', 'Dec 30 2018 01:22:34')
> > > Arch : ('64bit', '')
> > > Pip Info---
> > > Version : 19.0.3
> > > Directory:
> > > /home/ubuntu/anaconda3/envs/perf-mxnet/lib/python3.6/site-packages/pip
> > > --MXNet Info---
> > > Version : 1.5.0
> > > Directory: /home/ubuntu/ws/incubator-mxnet/python/mxnet
> > > Hashtag not found. Not installed from pre-built package.
> > > --System Info--
> > > Platform : Linux-4.4.0-1085-aws-x86_64-with-debian-stretch-sid
> > > system : Linux
> > > node : ip-172-31-32-129
> > > release : 4.4.0-1085-aws
> > > version : #96-Ubuntu SMP Tue Jun 11 09:08:32 UTC 2019
> > > --Hardware Info--
> > > machine : x86_64
> > > processor: x86_64
> > > Architecture: x86_64
> > > CPU op-mode(s):32-bit, 64-bit
> > > Byte Order:Little Endian
> > > CPU(s):72
> > > On-line CPU(s) list: 0-71
> > > Thread(s) per core:2
> > > Core(s) per socket:18
> > > Socket(s): 2
> > > NUMA node(s): 2
> > > Vendor ID: GenuineIntel
> > > CPU family:6
> > > Model: 85
> > > Model name:Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
> > > Stepping: 3
> > > CPU MHz: 3000.000
> > > BogoMIPS: 6000.00
> > > Hypervisor vendor: KVM
> > > Virtualization type: full
> > > L1d cache: 32K
> > > L1i cache: 32K
> > > L2 cache: 1024K
> > > L3 cache: 25344K
> > > NUMA node0 CPU(s):
Re: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc1
I will try to run a few benchmarks in a bare metal instance tonight to
remove virtualization variance for the measurements and provide some
numbers.
Please propose a set of models / examples that would be desirable to
run before the release and provide a link to an easy to run script
with instructions so we can validate the release better.
Thank you.
On Thu, Jun 27, 2019 at 10:01 AM Lai Wei wrote:
>
> Dear @dev,
>
> I m cancelling the vote for cached op fix:
>
> https://github.com/apache/incubator-mxnet/pull/15298
>
> As for the possible cpu training regression, it looks like not a blocker
> for now.
>
> I will start a new rc2 vote, please help to validate.
>
> Thanks!
>
>
> On Thu, Jun 27, 2019 at 10:06 PM Chen, Ciyong wrote:
>
> > Hi Pedro,
> >
> > I was able to reproduced the similar result (v1.5 is ~%5.6 slower than
> > v1.4, I was using 18 cores for computing) with your script on C5.18xlarge.
> > But need to bind the cores with below command when running the script,
> > (without setting the env variables, I got a close time (<1%) with v1.5 and
> > v1.4)
> > export KMP_AFFINITY=granularity=fine,noduplicates,compact,1,0
> > export OMP_NUM_THREADS=18
> >
> > Did you set any env variables during running?
> >
> > The performance result I got as below:
> > 1) 1.4.1.rc0 (1a7199691f5cbc6012bb53eecbf884bed5ae6590)
> > real12m10.856s
> > user234m49.576s
> > sys 4m38.044s
> >
> > 2) 1.5.0.rc1 (4d9667121ae6fb643f2a02ab15e25231ed756cde)
> > real12m52.140s
> > user246m30.740s
> > sys 5m8.188s
> >
> > As I looked at the profiling data, most of the ops have same perf between
> > v1.4 and v1.5. But some ops like " _backward_BatchNorm" and "Pooling" is
> > ~1.37x slower on v1.5 compared with v1.4.
> > Will do further analysis on these ops.
> >
> > Here's the hardware/OS info from my side:
> > --Python Info--
> > Version : 3.6.8
> > Compiler : GCC 7.3.0
> > Build: ('default', 'Dec 30 2018 01:22:34')
> > Arch : ('64bit', '')
> > Pip Info---
> > Version : 19.0.3
> > Directory:
> > /home/ubuntu/anaconda3/envs/perf-mxnet/lib/python3.6/site-packages/pip
> > --MXNet Info---
> > Version : 1.5.0
> > Directory: /home/ubuntu/ws/incubator-mxnet/python/mxnet
> > Hashtag not found. Not installed from pre-built package.
> > --System Info--
> > Platform : Linux-4.4.0-1085-aws-x86_64-with-debian-stretch-sid
> > system : Linux
> > node : ip-172-31-32-129
> > release : 4.4.0-1085-aws
> > version : #96-Ubuntu SMP Tue Jun 11 09:08:32 UTC 2019
> > --Hardware Info--
> > machine : x86_64
> > processor: x86_64
> > Architecture: x86_64
> > CPU op-mode(s):32-bit, 64-bit
> > Byte Order:Little Endian
> > CPU(s):72
> > On-line CPU(s) list: 0-71
> > Thread(s) per core:2
> > Core(s) per socket:18
> > Socket(s): 2
> > NUMA node(s): 2
> > Vendor ID: GenuineIntel
> > CPU family:6
> > Model: 85
> > Model name:Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
> > Stepping: 3
> > CPU MHz: 3000.000
> > BogoMIPS: 6000.00
> > Hypervisor vendor: KVM
> > Virtualization type: full
> > L1d cache: 32K
> > L1i cache: 32K
> > L2 cache: 1024K
> > L3 cache: 25344K
> > NUMA node0 CPU(s): 0-17,36-53
> > NUMA node1 CPU(s): 18-35,54-71
> > Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> > pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb
> > rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc
> > aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1
> > sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand
> > hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase
> > tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f rdseed adx
> > smap clflushopt clwb avx512cd xsaveopt xsavec xgetbv1 ida arat pku
> > --Network Test--
> >
> >
> > -Ciyong
> >
> >
> > -Original Message-
> > From: Zhao, Patric [mailto:patric.z...@intel.com]
> > Sent: Thursday, June 27, 2019 9:55 AM
> > To: dev@mxnet.incubator.apache.org
> > Cc: d...@mxnet.apache.org
> > Subject: RE: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc1
> >
> > Could we run more epochs to see the performance difference or profiling
> > the difference between good and bad run?
> >
> > > -Original Message-
> > > From: Pedro Larroy [mailto:pedro.larroy.li...@gmail.com]
> > > Sent: Thursday, June 27, 2019 9:35 AM
> > > To: dev@mxnet.incubator.apache.org
> > > Cc: d...@mxnet.apache.org
> > > Subject: Re: [VOTE] Release Apache MXNet (incubating) version
> > > 1.5.0.rc1
> > >
> > > I run again and the gap is again bigger, I guess we
[VOTE] Release Apache MXNet (incubating) version 1.5.0.rc2
Dear MXNet community, This is the 3-day vote to release Apache MXNet (incubating) version 1.5.0. Voting on dev@ will start June 26, 23:59:59(PST) and close on June 29, 23:59:59. 1) Link to release notes: https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Notes 2) Link to release candidate: https://github.com/apache/incubator-mxnet/releases/tag/1.5.0.rc2 3) Link to source and signatures on apache dist server: https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.5.0.rc2/ Please remember to TEST first before voting accordingly: +1 = approve +0 = no opinion -1 = disapprove (provide reason) -- Best Regards Lai
Re: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc1
Dear @dev,
I m cancelling the vote for cached op fix:
https://github.com/apache/incubator-mxnet/pull/15298
As for the possible cpu training regression, it looks like not a blocker
for now.
I will start a new rc2 vote, please help to validate.
Thanks!
On Thu, Jun 27, 2019 at 10:06 PM Chen, Ciyong wrote:
> Hi Pedro,
>
> I was able to reproduced the similar result (v1.5 is ~%5.6 slower than
> v1.4, I was using 18 cores for computing) with your script on C5.18xlarge.
> But need to bind the cores with below command when running the script,
> (without setting the env variables, I got a close time (<1%) with v1.5 and
> v1.4)
> export KMP_AFFINITY=granularity=fine,noduplicates,compact,1,0
> export OMP_NUM_THREADS=18
>
> Did you set any env variables during running?
>
> The performance result I got as below:
> 1) 1.4.1.rc0 (1a7199691f5cbc6012bb53eecbf884bed5ae6590)
> real12m10.856s
> user234m49.576s
> sys 4m38.044s
>
> 2) 1.5.0.rc1 (4d9667121ae6fb643f2a02ab15e25231ed756cde)
> real12m52.140s
> user246m30.740s
> sys 5m8.188s
>
> As I looked at the profiling data, most of the ops have same perf between
> v1.4 and v1.5. But some ops like " _backward_BatchNorm" and "Pooling" is
> ~1.37x slower on v1.5 compared with v1.4.
> Will do further analysis on these ops.
>
> Here's the hardware/OS info from my side:
> --Python Info--
> Version : 3.6.8
> Compiler : GCC 7.3.0
> Build: ('default', 'Dec 30 2018 01:22:34')
> Arch : ('64bit', '')
> Pip Info---
> Version : 19.0.3
> Directory:
> /home/ubuntu/anaconda3/envs/perf-mxnet/lib/python3.6/site-packages/pip
> --MXNet Info---
> Version : 1.5.0
> Directory: /home/ubuntu/ws/incubator-mxnet/python/mxnet
> Hashtag not found. Not installed from pre-built package.
> --System Info--
> Platform : Linux-4.4.0-1085-aws-x86_64-with-debian-stretch-sid
> system : Linux
> node : ip-172-31-32-129
> release : 4.4.0-1085-aws
> version : #96-Ubuntu SMP Tue Jun 11 09:08:32 UTC 2019
> --Hardware Info--
> machine : x86_64
> processor: x86_64
> Architecture: x86_64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Little Endian
> CPU(s):72
> On-line CPU(s) list: 0-71
> Thread(s) per core:2
> Core(s) per socket:18
> Socket(s): 2
> NUMA node(s): 2
> Vendor ID: GenuineIntel
> CPU family:6
> Model: 85
> Model name:Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
> Stepping: 3
> CPU MHz: 3000.000
> BogoMIPS: 6000.00
> Hypervisor vendor: KVM
> Virtualization type: full
> L1d cache: 32K
> L1i cache: 32K
> L2 cache: 1024K
> L3 cache: 25344K
> NUMA node0 CPU(s): 0-17,36-53
> NUMA node1 CPU(s): 18-35,54-71
> Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb
> rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc
> aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1
> sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand
> hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase
> tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f rdseed adx
> smap clflushopt clwb avx512cd xsaveopt xsavec xgetbv1 ida arat pku
> --Network Test--
>
>
> -Ciyong
>
>
> -Original Message-
> From: Zhao, Patric [mailto:patric.z...@intel.com]
> Sent: Thursday, June 27, 2019 9:55 AM
> To: dev@mxnet.incubator.apache.org
> Cc: d...@mxnet.apache.org
> Subject: RE: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc1
>
> Could we run more epochs to see the performance difference or profiling
> the difference between good and bad run?
>
> > -Original Message-
> > From: Pedro Larroy [mailto:pedro.larroy.li...@gmail.com]
> > Sent: Thursday, June 27, 2019 9:35 AM
> > To: dev@mxnet.incubator.apache.org
> > Cc: d...@mxnet.apache.org
> > Subject: Re: [VOTE] Release Apache MXNet (incubating) version
> > 1.5.0.rc1
> >
> > I run again and the gap is again bigger, I guess we need to average
> > out the times across several runs:
> >
> > piotr@ip-172-31-63-171:0:~/deeplearning-benchmark/dawnbench
> > (master)+$ time ~/mxnet_1.4/py3_venv/bin/python cifar10.py --epochs 5
> > && time ~/mxnet_1.5/py3_venv/bin/python cifar10.py --epochs 5
> > [23:17:09] ../src/io/iter_image_recordio_2.cc:172:
> > ImageRecordIOParser2:
> > /home/piotr/deeplearning-benchmark/data/cifar/train.rec, use 4 threads
> > for decoding..
> > [23:17:09] ../src/io/iter_image_recordio_2.cc:230: Load mean image
> > from /home/piotr/deeplearning-benchmark/data/cifar/mean.bin
> > [23:17:09] ../src/io/iter_image_recordio_2.cc:248: Load mean image
> >
RE: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc1
Hi Pedro,
I was able to reproduced the similar result (v1.5 is ~%5.6 slower than v1.4, I
was using 18 cores for computing) with your script on C5.18xlarge.
But need to bind the cores with below command when running the script, (without
setting the env variables, I got a close time (<1%) with v1.5 and v1.4)
export KMP_AFFINITY=granularity=fine,noduplicates,compact,1,0
export OMP_NUM_THREADS=18
Did you set any env variables during running?
The performance result I got as below:
1) 1.4.1.rc0 (1a7199691f5cbc6012bb53eecbf884bed5ae6590)
real12m10.856s
user234m49.576s
sys 4m38.044s
2) 1.5.0.rc1 (4d9667121ae6fb643f2a02ab15e25231ed756cde)
real12m52.140s
user246m30.740s
sys 5m8.188s
As I looked at the profiling data, most of the ops have same perf between v1.4
and v1.5. But some ops like " _backward_BatchNorm" and "Pooling" is ~1.37x
slower on v1.5 compared with v1.4.
Will do further analysis on these ops.
Here's the hardware/OS info from my side:
--Python Info--
Version : 3.6.8
Compiler : GCC 7.3.0
Build: ('default', 'Dec 30 2018 01:22:34')
Arch : ('64bit', '')
Pip Info---
Version : 19.0.3
Directory:
/home/ubuntu/anaconda3/envs/perf-mxnet/lib/python3.6/site-packages/pip
--MXNet Info---
Version : 1.5.0
Directory: /home/ubuntu/ws/incubator-mxnet/python/mxnet
Hashtag not found. Not installed from pre-built package.
--System Info--
Platform : Linux-4.4.0-1085-aws-x86_64-with-debian-stretch-sid
system : Linux
node : ip-172-31-32-129
release : 4.4.0-1085-aws
version : #96-Ubuntu SMP Tue Jun 11 09:08:32 UTC 2019
--Hardware Info--
machine : x86_64
processor: x86_64
Architecture: x86_64
CPU op-mode(s):32-bit, 64-bit
Byte Order:Little Endian
CPU(s):72
On-line CPU(s) list: 0-71
Thread(s) per core:2
Core(s) per socket:18
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family:6
Model: 85
Model name:Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
Stepping: 3
CPU MHz: 3000.000
BogoMIPS: 6000.00
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 25344K
NUMA node0 CPU(s): 0-17,36-53
NUMA node1 CPU(s): 18-35,54-71
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm
constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc aperfmperf
tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic
movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm
abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep
bmi2 erms invpcid rtm mpx avx512f rdseed adx smap clflushopt clwb avx512cd
xsaveopt xsavec xgetbv1 ida arat pku
--Network Test--
-Ciyong
-Original Message-
From: Zhao, Patric [mailto:patric.z...@intel.com]
Sent: Thursday, June 27, 2019 9:55 AM
To: dev@mxnet.incubator.apache.org
Cc: d...@mxnet.apache.org
Subject: RE: [VOTE] Release Apache MXNet (incubating) version 1.5.0.rc1
Could we run more epochs to see the performance difference or profiling the
difference between good and bad run?
> -Original Message-
> From: Pedro Larroy [mailto:pedro.larroy.li...@gmail.com]
> Sent: Thursday, June 27, 2019 9:35 AM
> To: dev@mxnet.incubator.apache.org
> Cc: d...@mxnet.apache.org
> Subject: Re: [VOTE] Release Apache MXNet (incubating) version
> 1.5.0.rc1
>
> I run again and the gap is again bigger, I guess we need to average
> out the times across several runs:
>
> piotr@ip-172-31-63-171:0:~/deeplearning-benchmark/dawnbench
> (master)+$ time ~/mxnet_1.4/py3_venv/bin/python cifar10.py --epochs 5
> && time ~/mxnet_1.5/py3_venv/bin/python cifar10.py --epochs 5
> [23:17:09] ../src/io/iter_image_recordio_2.cc:172:
> ImageRecordIOParser2:
> /home/piotr/deeplearning-benchmark/data/cifar/train.rec, use 4 threads
> for decoding..
> [23:17:09] ../src/io/iter_image_recordio_2.cc:230: Load mean image
> from /home/piotr/deeplearning-benchmark/data/cifar/mean.bin
> [23:17:09] ../src/io/iter_image_recordio_2.cc:248: Load mean image
> from /home/piotr/deeplearning-benchmark/data/cifar/mean.bin completed
> [23:17:09] ../src/io/iter_image_recordio_2.cc:172:
> ImageRecordIOParser2:
> /home/piotr/deeplearning-benchmark/data/cifar/test.rec, use 4 threads
> for decoding..
> [23:17:09] ../src/io/iter_image_recordio_2.cc:230: Load mean image
> from /home/piotr/deeplearning-benchmark/data/cifar/mean.bin
> [23:17:09] ../src/io/iter_image_recordio_2.cc:248: Load mean image
> from /home/piotr/deeplearning-benchmark/data/cifar/mean.bin completed
>
Re: Podling Report Reminder - July 2019
A draft has been posted: https://cwiki.apache.org/confluence/display/INCUBATOR/July2019#mxnet -sz On 2019/06/23 23:51:12, Sheng Zha wrote: > A few of us started drafing it and will post the draft for community review > by 6/26. Thanks. > > -sz > > On 2019/06/22 02:31:17, jmcl...@apache.org wrote: > > Dear podling, > > > > This email was sent by an automated system on behalf of the Apache > > Incubator PMC. It is an initial reminder to give you plenty of time to > > prepare your quarterly board report. > > > > The board meeting is scheduled for Wed, 17 July 2019, 10:30 am PDT. > > The report for your podling will form a part of the Incubator PMC > > report. The Incubator PMC requires your report to be submitted 2 weeks > > before the board meeting, to allow sufficient time for review and > > submission (Wed, July 03). > > > > Please submit your report with sufficient time to allow the Incubator > > PMC, and subsequently board members to review and digest. Again, the > > very latest you should submit your report is 2 weeks prior to the board > > meeting. > > > > Candidate names should not be made public before people are actually > > elected, so please do not include the names of potential committers or > > PPMC members in your report. > > > > Thanks, > > > > The Apache Incubator PMC > > > > Submitting your Report > > > > -- > > > > Your report should contain the following: > > > > * Your project name > > * A brief description of your project, which assumes no knowledge of > > the project or necessarily of its field > > * A list of the three most important issues to address in the move > > towards graduation. > > * Any issues that the Incubator PMC or ASF Board might wish/need to be > > aware of > > * How has the community developed since the last report > > * How has the project developed since the last report. > > * How does the podling rate their own maturity. > > > > This should be appended to the Incubator Wiki page at: > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/July2019 > > > > Note: This is manually populated. You may need to wait a little before > > this page is created from a template. > > > > Note: The format of the report has changed to use markdown. > > > > Mentors > > --- > > > > Mentors should review reports for their project(s) and sign them off on > > the Incubator wiki page. Signing off reports shows that you are > > following the project - projects that are not signed may raise alarms > > for the Incubator PMC. > > > > Incubator PMC > > >
