Yeah, I think we are at the point at which we have to disable tests.. If a test fails in nightly, the commit would not be reverted since it's hard to pin a failure to a specific PR. We will have reporting for failures on nightly (they have proven to be stable, so we can enable it right from the beginning). I'm also currently working on a system that tracks all test failures, so this will also cover nightly tests. This will give us actionable data which allows us to define acceptance criteria for a release. E.g. if the test success rate is below X%, a release can not be made. This could be set to 100% for nightly, for example.
It would definitely be good if we could determine which tests are required to run and which ones are unnecessary. I don't really like the flag in the comment (and also it's hard to integrate). A good idea would be some analytics on the changed file content. If we have this data, we could easily enable and disable different jobs. Since this behaviour is entirely defined in GitHub, I'd like to invite everybody to submit a PR. -Marco On Thu, Jun 7, 2018 at 5:20 PM Aaron Markham <aaron.s.mark...@gmail.com> wrote: > I'd like to disable flaky tests until they're fixed. > What would the process be for fixing a failure if the tests are done > nightly? Would the commit be reverted? Won't we end up in the same > situation with so many flaky tests? > > I'd like to see if we can separate the test pipelines based on the content > of the commit. I think that md, html, and js updates should fly through and > not have to go through GPU tests. > > Maybe some special flag added to the comment? > Is this possible? > > > On Wed, Jun 6, 2018 at 10:37 PM, Pedro Larroy < > pedro.larroy.li...@gmail.com> > wrote: > > > Hi Team > > > > The time to validate a PR is growing, due to our number of supported > > platforms and increased time spent in testing and running models. We are > > at approximately 3h for a full successful run. > > > > This is compounded with the failure rate of builds due to flaky tests of > > more than 50% which is a big drag in developer productivity if you can > only > > get one or two CI runs to a change per day. > > > > I would want a turnaround time of less than 30 minutes and 0% failure > rate > > on master. > > > > For this I propose working towards moving tests from CI to nightly, > > specially the ones that take most time or do black box testing with full > > training of models. And addressing flaky tests by either fixing them or > > disabling them. > > > > I would like to check if there's consensus on this previous plan so we > are > > aligned on pursuing this common goal as a shared effort. > > > > Pedro. > > >
