comaniac commented on a change in pull request #64: URL: https://github.com/apache/tvm-rfcs/pull/64#discussion_r832440524
########## File path: rfcs/0064-autotvm-tune-with-subgraph-granularity.md ########## @@ -0,0 +1,135 @@ +- Feature Name: AutoTVM tuning with Subgraph Granularity +- Start Date: 2022-3-17 +- RFC PR: [apache/tvm-rfcs#0064](https://github.com/apache/tvm-rfcs/pull/0064) +- GitHub Issue: N/A + +# Summary +[summary]: #summary + +This RFC introduces why and how we tune with subgraph granularity. + +# Motivation +[motivation]: #motivation + +During performance optimization for platform Xavier which has a Volta GPU in it, we found that tuning by AutoTVM with subgraph as granularity could bring performance improvement. Because the data type of the subgraph's output may be different from the data type of the subgraph's anchor operator's output, this may change the task from memory-bound to compute-bound or change in reverse. +Let's take the subgraph in the figure below as an example. If we tune with the single convolution the output data type is 'Int32' but if we tune with the subgraph the output data type is 'Int8'. The former's data size is four times the latter one. But in the actual inference, the data type is 'Int8' same as the latter one. So the best config searched by tuning with a single operator maybe not be the best for the subgraph. + +We also run an example to verify the theory above. + +- We wrote a schedule marked as ```ScheduleA``` for the subgraph above by hardcode and the latency of subgraph inferencing is 104 microseconds. Then we tuned the subgraph with single op as granularity. In the tuning log we found ```ScheduleA``` and the latency recorded in the measurement result is 329 microseconds. +- The best schedule from the tuned log in the step above is marked as ```ScheduleB``` and the latency recorded in the measurement result is 237 microseconds. The latency of subgraph inferencing with ```ScheduleB``` is 224 microseconds. + +From the example above we can tell AutoTVM would not find ```ScheduleA```, the obvious better schedule. This means the tuning result is distorted, the distortion would be more obvious if the shape of the output became bigger. Review comment: This is a great motivating example. I believe this is also an important reason that auto-scheduler could have end-to-end speedup over AutoTVM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
