masahi edited a comment on issue #5061: [Torch, QNN] Remove FP32 piggy back and use QNN add/mul/concatenate URL: https://github.com/apache/incubator-tvm/pull/5061#issuecomment-598902899 @anijain2305 @jwfromm @jjohnson-arm Here is the current result on mobilenet v2, using QNN `add` and post training calibration (**which is wrong**). ``` Model name: mobilenet_v2, per channel quantization PyTorch accuracy: Top1 = 67.87, Top5 = 88.15 TVM accuracy: Top1 = 62.47, Top5 = 84.67 PyTorch top5 label: [101 386 51 385 69] TVM top5 label: [101 386 51 385 340] PyTorch top5 raw output: [18.233843 16.314491 15.674707 13.115572 12.795679] TVM top5 raw output: [27.510712 26.231144 21.752655 20.153194 17.274168] max abs diff: 9.916653 mean abs_diff: 2.0649028 50 in 1000 raw outputs correct. ``` We lost 5 point accuracy compared to Torch. And here is **without** post training calibration, also using QNN `add`. Now the top1 accuracy is much better and almost the same as Torch. Moreover, the raw output of the network, 1000 floating point values, are much closer to Torch. The former has only 50 out of 1000 outputs identical, while the latter, correct one has 376/1000. ``` Model name: mobilenet_v2, per channel quantization PyTorch accuracy: Top1 = 71.32, Top5 = 89.86 TVM accuracy: Top1 = 71.27, Top5 = 89.86 PyTorch top5 label: [101 386 385 51 340] TVM top5 label: [101 386 385 51 340] PyTorch top5 raw output: [20.168097 18.80845 17.222195 13.59647 9.290921] TVM top5 raw output: [19.941488 18.581842 16.995586 13.823077 9.064313] max abs diff: 0.9064312 mean abs_diff: 0.17562106 376 in 1000 raw outputs correct. ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
