[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-372415466 @marcoabreu I have rebased with the master branch. Could you help to verify whether my changes in Jenkinsfile and runtime_functions.sh are correct? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-371556298 @KellenSunderland Thank you very much for such a careful proofreading. All are good catches and comments, I will address them later. :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-370205130 @wentingj Yes, I noticed that too. I think there is still value of doing this. We can always keep improving the performance in the long run. I am waiting for @marcoabreu to setup the P3 instance properly. The current os image used in the P3 does not have cudnn lib installed. It may take 3-4 weeks to get this PR merged. Please plan accordingly. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-370122535 @wentingj Thanks for the benchmark results. It aligns with what we observed cudnn: int8 conv costs more runtime than fp32 conv. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-364294870 Loading `libcudnn.so.7` failed. @marcoabreu @KellenSunderland Do you know what went wrong with the configuration? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-363243755 @KellenSunderland cudnn6 should be alright. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-362779633 @pengzhao-intel I will definitely let you know if there are breaking changes. For testing inference, you can use the script `example/quantization/imagenet_gen_qsym.py` to generate quantized models (resnet-152 and inception w/ bn) and run the inference using `example/quantization/imagenet_inference.py`. Remember to change the `ctx` to `mx.cpu` since it's currently default to `mx.gpu(0)`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-362742835 @marcoabreu It looks like the cuDNN version (5.0) is too low for building quantization implementation. Do we have plan to upgrade the lib? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-362642270 @KellenSunderland Thank you for the note. I will either cherry pick the PR or rebase with the master once your PR is merged. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-362662671 @pengzhao-intel Thank you guys for implementing quantized ops for CPU computing. We look forward to seeing and benchmarking the implementation. I propose that your team work on top this PR and submit a separate PR of your work after this one is merged. This is already a big PR (>3000 lines of code) and adding more code would make the review process overwhelming. Please also know that we still need to wait for P3 instances in the CI being officially ready to fully test the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-362642270 @KellenSunderland Thank you for the note. I will either cherry pick the PR or rebased with the master once your PR is merged. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-362483572 Thank @marcoabreu for setting up the testing environment for the PR, I will try to run the tests on it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-360965769 @marcoabreu Oh, I see. Since calibration is conducted offline, it's not constrained by the hardware resources of edge devices. I believe there is an optimal value of num_bins for each layer. It could become a hyperparameter for users to tune. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-360964981 @marcoabreu The optimal values are determined by the calibration datasets. So they are independent of platforms. So long as the platform supports int8 basic addition and multiplication, it would be able to run quantized models. We would of course need to write dedicated int8 operators for a specific platform. The current implementation only works on Nvidia GPUs with dp4a instruction. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-360964778 @jinhuang415 1. The parameters are quantized offline, which means the min/max values were pre-calculated before inference. 2. In theory, if the calibration dataset is representative enough of the real inference image sets, more examples used for calibration should lead to less accuracy loss. The purpose of using entropy calibration is to keep the accuracy loss stable with respect to the number of examples used for calibration. The naive calibration approach suffers from more calibration examples leads to bigger accuracy loss as you can see the trend in the last two tables. My guess is that if the calibration dataset contains examples that are not similar to real inference images, the quantization thresholds might be biased by those examples and result in a little drop down of accuracy. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration
reminisce commented on issue #9552: [REQUEST FOR REVIEW | DO NOT MERGE] Model Quantization with Calibration URL: https://github.com/apache/incubator-mxnet/pull/9552#issuecomment-360963839 @wentingj The quantized ops used in the benchmarks are convolution, fully-connected, avg_pooling, max_pooling, and flatten. The quantize, dequantize, and requantize each takes up about 5-10% runtime per epoch. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
