piiswrong closed pull request #10761: update the performance page of MXNet. URL: https://github.com/apache/incubator-mxnet/pull/10761
This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/docs/faq/perf.md b/docs/faq/perf.md index b5d73f69a03..ce74391122d 100644 --- a/docs/faq/perf.md +++ b/docs/faq/perf.md @@ -29,65 +29,70 @@ Note that _MXNet_ treats all CPUs on a single machine as a single device. So whether you specify `cpu(0)` or `cpu()`, _MXNet_ will use all CPU cores on the machine. ### Scoring results -The following table shows performance, +The following table shows performance of [MXNet-1.2.0.rc1](https://github.com/apache/incubator-mxnet/releases/download/1.2.0.rc1/apache-mxnet-src-1.2.0.rc1-incubating.tar.gz), namely number of images that can be predicted per second. We used [example/image-classification/benchmark_score.py](https://github.com/dmlc/mxnet/blob/master/example/image-classification/benchmark_score.py) to measure the performance on different AWS EC2 machines. -AWS EC2 C4.8xlarge: - -| Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | -| --- | --- | --- | --- | --- | --- | --- | -| 1 | 119.57 | 34.23 | 111.36 | 54.42 | 42.83 | 19.51 | -| 2 | 210.58 | 51.63 | 137.10 | 67.30 | 57.54 | 23.56 | -| 4 | 318.54 | 70.00 | 187.21 | 76.53 | 63.64 | 25.80 | -| 8 | 389.34 | 77.39 | 211.90 | 84.26 | 63.89 | 28.11 | -| 16 | 489.12 | 85.26 | 220.52 | 82.00 | 63.93 | 27.08 | -| 32 | 564.04 | 87.15 | 208.21 | 83.05 | 62.19 | 25.76 | - -AWS EC2 C4.4xlarge: - -| Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | -| --- | --- | --- | --- | --- | --- | --- | -| 1 | 109.96 | 23.00 | 71.82 | 28.10 | 30.66 | 11.81 | -| 2 | 124.56 | 24.86 | 81.61 | 31.32 | 32.73 | 12.82 | -| 4 | 157.01 | 26.60 | 86.77 | 32.94 | 33.32 | 13.16 | -| 8 | 178.40 | 30.67 | 88.58 | 33.52 | 33.32 | 13.32 | -| 16 | 189.52 | 35.61 | 90.36 | 33.63 | 32.94 | 13.18 | -| 32 | 196.61 | 38.98 | 105.27 | 33.77 | 32.65 | 13.00 | - -AWS EC2 C4.2xlarge: - -| Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | -| --- | --- | --- | --- | --- | --- | --- | -| 1 | 70.75 | 12.87 | 42.86 | 16.53 | 18.14 | 7.01 | -| 2 | 71.53 | 13.08 | 45.66 | 17.38 | 18.53 | 7.18 | -| 4 | 84.72 | 15.38 | 47.50 | 17.80 | 18.96 | 7.35 | -| 8 | 93.44 | 18.33 | 48.08 | 17.93 | 18.99 | 7.40 | -| 16 | 97.03 | 20.12 | 55.73 | 18.00 | 18.91 | 7.36 | -| 32 | 113.90 | 21.10 | 62.54 | 17.98 | 18.80 | 7.33 | - -AWS EC2 C4.xlarge: - -| Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | -| --- | --- | --- | --- | --- | --- | --- | -| 1 | 37.92 | 6.57 | 23.09 | 8.79 | 9.65 | 3.73 | -| 2 | 36.77 | 7.31 | 24.00 | 9.00 | 9.84 | 3.78 | -| 4 | 43.18 | 8.94 | 24.42 | 9.12 | 9.91 | 3.83 | -| 8 | 47.05 | 10.01 | 28.32 | 9.13 | 9.88 | 3.83 | -| 16 | 55.74 | 10.61 | 31.96 | 9.14 | 9.86 | 3.80 | -| 32 | 65.05 | 10.91 | 33.86 | 9.34 | 10.31 | 3.86 | - -AWS EC2 C4.large: - -| Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | -| --- | --- | --- | --- | --- | --- | --- | -| 1 | 19.86 | 3.67 | 12.20 | 4.59 | 5.11 | 1.97 | -| 2 | 19.37 | 4.24 | 12.41 | 4.64 | 5.15 | 1.98 | -| 4 | 22.64 | 4.89 | 14.34 | 4.66 | 5.16 | 2.00 | -| 8 | 27.19 | 5.25 | 16.17 | 4.66 | 5.16 | 1.99 | -| 16 | 31.82 | 5.46 | 17.24 | 4.76 | 5.35 | OOM | -| 32 | 34.67 | 5.55 | 17.64 | 4.88 | OOM | OOM | +AWS EC2 C5.18xlarge: + +| Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | +|-------|---------|--------|--------------|--------------|-----------|------------| +| 1 | 390.53 | 81.57 | 124.13 | 62.26 | 76.22 | 32.92 | +| 2 | 596.45 | 100.84 | 206.58 | 93.36 | 119.55 | 46.80 | +| 4 | 710.77 | 119.04 | 275.55 | 127.86 | 148.62 | 59.36 | +| 8 | 921.40 | 120.38 | 380.82 | 157.11 | 167.95 | 70.78 | +| 16 | 1018.43 | 115.30 | 411.67 | 168.71 | 178.54 | 75.13 | +| 32 | 1290.31 | 107.19 | 483.34 | 179.38 | 193.47 | 85.86 | + + +AWS EC2 C5.9xlarge: + +| Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | +|-------|---------|-------|--------------|--------------|-----------|------------| +| 1 | 257.77 | 50.61 | 130.99 | 66.95 | 75.38 | 32.33 | +| 2 | 410.60 | 63.02 | 195.14 | 87.84 | 102.67 | 41.57 | +| 4 | 462.59 | 62.64 | 263.15 | 109.87 | 127.15 | 50.69 | +| 8 | 573.79 | 63.95 | 309.99 | 121.36 | 140.84 | 59.01 | +| 16 | 709.47 | 67.79 | 350.19 | 128.26 | 147.41 | 64.15 | +| 32 | 831.46 | 69.58 | 354.91 | 129.92 | 149.18 | 64.25 | + + +AWS EC2 C5.4xlarge: + +| Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | +|-------|---------|-------|--------------|--------------|-----------|------------| +| 1 | 214.15 | 29.32 | 114.97 | 47.96 | 61.01 | 23.92 | +| 2 | 310.04 | 34.81 | 150.09 | 60.89 | 71.16 | 27.92 | +| 4 | 330.69 | 34.56 | 186.63 | 74.15 | 86.86 | 34.37 | +| 8 | 378.88 | 35.46 | 204.89 | 77.05 | 91.10 | 36.93 | +| 16 | 424.00 | 36.49 | 211.55 | 78.39 | 91.23 | 37.34 | +| 32 | 481.95 | 37.23 | 213.71 | 78.23 | 91.68 | 37.26 | + + +AWS EC2 C5.2xlarge: + +| Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | +|-------|---------|-------|--------------|--------------|-----------|------------| +| 1 | 131.01 | 15.67 | 78.75 | 31.12 | 37.30 | 14.75 | +| 2 | 182.29 | 18.01 | 98.59 | 39.13 | 45.98 | 17.84 | +| 4 | 189.31 | 18.25 | 110.26 | 41.35 | 49.21 | 19.32 | +| 8 | 211.75 | 18.57 | 115.46 | 42.53 | 49.98 | 19.81 | +| 16 | 236.06 | 19.11 | 117.18 | 42.59 | 50.20 | 19.92 | +| 32 | 261.13 | 19.46 | 116.20 | 42.72 | 49.95 | 19.80 | + + +AWS EC2 C5.xlarge: + +| Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | +|-------|---------|------|--------------|--------------|-----------|------------| +| 1 | 36.64 | 3.93 | 27.06 | 10.09 | 12.98 | 5.06 | +| 2 | 49.21 | 4.49 | 29.67 | 10.80 | 12.94 | 5.14 | +| 4 | 50.12 | 4.50 | 30.31 | 10.83 | 13.17 | 5.19 | +| 8 | 54.71 | 4.58 | 30.22 | 10.89 | 13.19 | 5.20 | +| 16 | 60.23 | 4.70 | 30.20 | 10.91 | 13.23 | 5.19 | +| 32 | 66.37 | 4.76 | 30.10 | 10.90 | 13.22 | 5.15 | + ## Other CPU @@ -101,88 +106,90 @@ We suggest always checking to make sure that a recent cuDNN version is used. Setting the environment `export MXNET_CUDNN_AUTOTUNE_DEFAULT=1` sometimes also helps. -We show results when using various GPUs including K80 (EC2 p2.2xlarge), M40, -and P100 (DGX-1). +We show results when using various GPUs including K80 (EC2 p2.2xlarge), M60 (EC2 g3.4xlarge), +and V100 (EC2 p3.2xlarge). ### Scoring results Based on [example/image-classification/benchmark_score.py](https://github.com/dmlc/mxnet/blob/master/example/image-classification/benchmark_score.py) -and MXNet commit `0a03417`, with cuDNN 5.1 +and [MXNet-1.2.0.rc1](https://github.com/apache/incubator-mxnet/releases/download/1.2.0.rc1/apache-mxnet-src-1.2.0.rc1-incubating.tar.gz), with cuDNN 7.0.5 - K80 (single GPU) - | Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | - | --- | --- | --- | --- | --- | --- | --- | - | 1 | 202.66 | 70.76 | 74.91 | 42.61 | 70.94 | 24.87 | - | 2 | 233.76 | 63.53 | 119.60 | 60.09 | 92.28 | 34.23 | - | 4 | 367.91 | 78.16 | 164.41 | 72.30 | 116.68 | 44.76 | - | 8 | 624.14 | 119.06 | 195.24 | 79.62 | 129.37 | 50.96 | - | 16 | 1071.19 | 195.83 | 256.06 | 99.38 | 160.40 | 66.51 | - | 32 | 1443.90 | 228.96 | 287.93 | 106.43 | 167.12 | 69.73 | - -- M40 - - | Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | - | --- | --- | --- | --- | --- | --- | --- | - | 1 | 412.09 | 142.10 | 115.89 | 64.40 | 126.90 | 46.15 | - | 2 | 743.49 | 212.21 | 205.31 | 108.06 | 202.17 | 75.05 | - | 4 | 1155.43 | 280.92 | 335.69 | 161.59 | 266.53 | 106.83 | - | 8 | 1606.87 | 332.76 | 491.12 | 224.22 | 317.20 | 128.67 | - | 16 | 2070.97 | 400.10 | 618.25 | 251.87 | 335.62 | 134.60 | - | 32 | 2694.91 | 466.95 | 624.27 | 258.59 | 373.35 | 152.71 | - -- P100 - - | Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | - | --- | --- | --- | --- | --- | --- | --- | - | 1 | 624.84 | 294.6 | 139.82 | 80.17 | 162.27 | 58.99 | - | 2 | 1226.85 | 282.3 | 267.41 | 142.63 | 278.02 | 102.95 | - | 4 | 1934.97 | 399.3 | 463.38 | 225.56 | 423.63 | 168.91 | - | 8 | 2900.54 | 522.9 | 709.30 | 319.52 | 529.34 | 210.10 | - | 16 | 4063.70 | 755.3 | 949.22 | 444.65 | 647.43 | 270.07 | - | 32 | 4883.77 | 854.4 | 1197.74 | 493.72 | 713.17 | 294.17 | +| Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | +|-------|---------|--------|--------------|--------------|-----------|------------| +| 1 | 243.93 | 43.59 | 68.62 | 35.52 | 67.41 | 23.65 | +| 2 | 338.16 | 49.14 | 113.41 | 56.29 | 93.35 | 33.88 | +| 4 | 478.92 | 53.44 | 159.61 | 74.43 | 119.18 | 45.23 | +| 8 | 683.52 | 70.50 | 190.49 | 86.23 | 131.32 | 50.54 | +| 16 | 1004.66 | 109.01 | 254.20 | 105.70 | 155.40 | 62.55 | +| 32 | 1238.55 | 114.98 | 285.49 | 116.79 | 159.42 | 64.99 | + +- M60 + +| Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | +|-------|---------|--------|--------------|--------------|-----------|------------| +| 1 | 243.49 | 59.95 | 101.97 | 48.30 | 95.46 | 39.29 | +| 2 | 491.04 | 69.14 | 170.35 | 80.27 | 142.61 | 60.17 | +| 4 | 711.54 | 78.94 | 257.89 | 123.09 | 182.36 | 76.51 | +| 8 | 1077.73 | 109.34 | 343.42 | 152.82 | 208.74 | 87.27 | +| 16 | 1447.21 | 144.93 | 390.25 | 166.32 | 220.73 | 92.41 | +| 32 | 1797.66 | 151.86 | 416.69 | 176.56 | 230.19 | 97.03 | + + +- V100 + +| Batch | Alexnet | VGG | Inception-BN | Inception-v3 | Resnet 50 | Resnet 152 | +|-------|---------|--------|--------------|--------------|-----------|------------| +| 1 | 659.51 | 205.16 | 136.91 | 76.54 | 162.15 | 61.38 | +| 2 | 1248.21 | 265.40 | 261.85 | 144.23 | 293.74 | 116.30 | +| 4 | 2122.41 | 333.97 | 477.22 | 270.03 | 479.14 | 195.17 | +| 8 | 3894.30 | 420.26 | 831.09 | 450.68 | 699.39 | 294.19 | +| 16 | 5815.58 | 654.16 | 1332.26 | 658.97 | 947.45 | 398.79 | +| 32 | 7906.09 | 708.43 | 1784.23 | 817.33 | 1076.81 | 451.82 | + ### Training results Based on [example/image-classification/train_imagenet.py](https://github.com/dmlc/mxnet/blob/master/example/image-classification/train_imagenet.py) -and MXNet commit `0a03417`, with CUDNN 5.1. The benchmark script is available at +and [MXNet-1.2.0.rc1](https://github.com/apache/incubator-mxnet/releases/download/1.2.0.rc1/apache-mxnet-src-1.2.0.rc1-incubating.tar.gz), with CUDNN 7.0.5. The benchmark script is available at [here](https://github.com/mli/mxnet-benchmark/blob/master/run_vary_batch.sh), -where the batch size for Alexnet is increased by 8x. +where the batch size for Alexnet is increased by 16x. - K80 (single GPU) | Batch | Alexnet(\*8) | Inception-v3 | Resnet 50 | | --- | --- | --- | --- | - | 1 | 230.69 | 9.81 | 13.83 | - | 2 | 348.10 | 15.31 | 21.85 | - | 4 | 457.28 | 20.48 | 29.58 | - | 8 | 533.51 | 24.47 | 36.83 | - | 16 | 582.36 | 28.46 | 43.60 | - | 32 | 483.37 | 29.62 | 45.52 | + | 1 | 300.30 | 10.48 | 15.61 | + | 2 | 406.08 | 16.00 | 23.88 | + | 4 | 461.01 | 22.10 | 32.26 | + | 8 | 484.00 | 26.80 | 39.42 | + | 16 | 490.45 | 31.62 | 46.69 | + | 32 | 414.72 | 33.78 | 49.48 | -- M40 +- M60 - | Batch | Alexnet(\*8) | Inception-v3 | Resnet 50 | + | Batch | Alexnet(\*16) | Inception-v3 | Resnet 50 | | --- | --- | --- | --- | - | 1 | 405.17 | 14.35 | 21.56 | - | 2 | 606.32 | 23.96 | 36.48 | - | 4 | 792.66 | 37.38 | 52.96 | - | 8 | 1016.51 | 52.69 | 70.21 | - | 16 | 1105.18 | 62.35 | 83.13 | - | 32 | 1046.23 | 68.87 | 90.74 | + | 1 | 380.96 | 14.06 | 20.55 | + | 2 | 530.53 | 21.90 | 32.65 | + | 4 | 600.17 | 31.96 | 45.57 | + | 8 | 633.60 | 40.58 | 54.92 | + | 16 | 639.37 | 46.88 | 64.44 | + | 32 | 576.54 | 50.05 | 68.34 | -- P100 +- V100 - | Batch | Alexnet(\*8) | Inception-v3 | Resnet 50 | + | Batch | Alexnet(\*16) | Inception-v3 | Resnet 50 | | --- | --- | --- | --- | - | 1 | 809.94 | 15.14 | 27.20 | - | 2 | 1202.93 | 30.34 | 49.55 | - | 4 | 1631.37 | 50.59 | 78.31 | - | 8 | 1882.74 | 77.75 | 122.45 | - | 16 | 2012.04 | 111.11 | 156.79 | - | 32 | 1869.69 | 129.98 | 181.53 | + | 1 | 1629.52 | 21.83 | 34.54 | + | 2 | 2359.73 | 40.11 | 65.01 | + | 4 | 2687.89 | 72.79 | 113.49 | + | 8 | 2919.02 | 118.43 | 174.81 | + | 16 | 2994.32 | 173.15 | 251.22 | + | 32 | 2585.61 | 214.48 | 298.51 | ## Multiple Devices ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
