I'd like to disable flaky tests until they're fixed. What would the process be for fixing a failure if the tests are done nightly? Would the commit be reverted? Won't we end up in the same situation with so many flaky tests?
I'd like to see if we can separate the test pipelines based on the content of the commit. I think that md, html, and js updates should fly through and not have to go through GPU tests. Maybe some special flag added to the comment? Is this possible? On Wed, Jun 6, 2018 at 10:37 PM, Pedro Larroy <pedro.larroy.li...@gmail.com> wrote: > Hi Team > > The time to validate a PR is growing, due to our number of supported > platforms and increased time spent in testing and running models. We are > at approximately 3h for a full successful run. > > This is compounded with the failure rate of builds due to flaky tests of > more than 50% which is a big drag in developer productivity if you can only > get one or two CI runs to a change per day. > > I would want a turnaround time of less than 30 minutes and 0% failure rate > on master. > > For this I propose working towards moving tests from CI to nightly, > specially the ones that take most time or do black box testing with full > training of models. And addressing flaky tests by either fixing them or > disabling them. > > I would like to check if there's consensus on this previous plan so we are > aligned on pursuing this common goal as a shared effort. > > Pedro. >
