Hi Team The time to validate a PR is growing, due to our number of supported platforms and increased time spent in testing and running models. We are at approximately 3h for a full successful run.
This is compounded with the failure rate of builds due to flaky tests of more than 50% which is a big drag in developer productivity if you can only get one or two CI runs to a change per day. I would want a turnaround time of less than 30 minutes and 0% failure rate on master. For this I propose working towards moving tests from CI to nightly, specially the ones that take most time or do black box testing with full training of models. And addressing flaky tests by either fixing them or disabling them. I would like to check if there's consensus on this previous plan so we are aligned on pursuing this common goal as a shared effort. Pedro.
