That's a great question! Since we record branch-coverage as well, we would now reliably notice these test-gaps and could make targeted tests. Additionally, we could define test-coverage thresholds for certain files to enforce this - e.g. 100% for all operators.
While the stochastic approach certainly covers a wider area, failures are currently rather ignored and rerun instead of properly analyzed. Proper tooling like a consistent test-coverage would then enable us to properly investigate these inconsistencies if we encounter them. -Marco On Sun, Jan 20, 2019 at 5:55 PM Chris Olivier <cjolivie...@gmail.com> wrote: > wouldn’t abandoning stochastic testing mean that execution paths would no > longer be tested and therefore increase issues found in production? > > On Sun, Jan 20, 2019 at 5:13 AM Marco de Abreu <marco.g.ab...@gmail.com> > wrote: > > > Hello everyone, > > > > a few months ago, I have enabled test coverage recording for the MXNet > > repository [1]. The next step would be to report the coverage changes to > > pull requests so we can make them part of the review. For this, I've > > created a pull request here at [2]. > > > > Before this feature can be enabled though, we have to stabilize the test > > coverage. Due to the extensive usage of random inputs in our tests, the > > execution takes different paths and thus, the coverage results undergo a > > heavy variance. If we would enable the coverage report for PRs, they > would > > be marked as increased/decreased coverage without actually doing so. To > get > > a few examples, visit [3]. > > > > I'd like to provide some detailed examples or screenshots, but with big > > reports [4], CodeCov's webinterface tends to time out. I currently have > an > > open ticket with CodeCov that will hopefully improve the situation soon, > > but we have to live with the timeouts until then. > > > > My proposal to improve the situation would be to divide the work across > the > > community in two stages: > > 1. Replace random inputs in tests with deterministic ones if applicable. > > 2. If randomness cannot be prevented or this 'flakiness' persists, we > could > > run the same test multiple times with coverage, look at the coverage diff > > and then write targeted unittests for the inconsistent parts. > > > > If there is more interest towards test coverage, I'd be happy to write a > > guide that explains how to measure test coverage and detect flaky > coverage > > areas so everybody can help contribute towards a stable test coverage. > > Please let me know what you think. > > > > Best regards, > > Marco > > > > [1]: https://codecov.io/gh/apache/incubator-mxnet > > [2]: https://github.com/apache/incubator-mxnet/pull/12648 > > [3]: https://codecov.io/gh/apache/incubator-mxnet/pulls > > [4]: https://codecov.io/gh/apache/incubator-mxnet/pull/13849 > > >
