Sandeep, Thanks for initiating work on individual operator performance. However I find the proposed approach(ie., a separate libary/framework) to unnecessary and increases maintenance overhead for the project. Also, have you considered alternate approaches to achieve the same goal.?
Many of the requirements/motivations you have mentioned typically should be covered in unit-tests(different data-types/ different dimensions), so instead of having to rewrite for all operators measuring performance, consider writing a @timeit routine(using Python decorators) which can be called on individual unit tests. Also even if you call the performance script from Python, typically you want to measure as close to the kernel as possible and avoid any other variables. I left some comments on the doc itself. Happy to discuss further. -Naveen On Mon, Apr 29, 2019 at 1:57 PM sandeep krishnamurthy < sandeep.krishn...@gmail.com> wrote: > Hello Community, > > I am currently working on building a utility/library to help us easily do > individual operator benchmarking in MXNet. I have documented the proposal > in > this cwiki > < > https://cwiki.apache.org/confluence/display/MXNET/MXNet+Operator+Benchmarks > >, > and staging the current development in this github repository > <https://github.com/sandeep-krishnamurthy/dl-operator-benchmark>. Proposal > is to get this library under incubator-mxnet/benchmark/ > <https://github.com/apache/incubator-mxnet/tree/master/benchmark>. Please > do review and provide your feedback and suggestions. > > Thanks to fellow MXNet community members - Lin, Sam, Rohit for providing > initial ideas and suggestion. > > Best, > Sandeep > > > > > -- > Sandeep Krishnamurthy >
