Quick update: The issue has been resolved. It was actually not as bad as I thought, because flinkbot was showing the build as "FAILED", so people probably saw that something is wrong.
On Tue, Oct 27, 2020 at 3:51 PM Robert Metzger <rmetz...@apache.org> wrote: > Hi all, > > Quick update on our CI system during the "hot phase" of the release: > 1. We are (for the first time in this release cycle) *hitting our e2e > test processing capacity limit:* All our 20 slots are in use, 13 builds > are in the queue. The other tests are running without delays. I will > monitor this closely. > Please try to use your personal Azure accounts as much as possible for > testing, and only push to your PR once you know the change should pass. > > 2. I probably made a *mistake in our e2e test scripts, causing broken e2e > tests to not fail the build*. > I'm tracking the issue here: > https://issues.apache.org/jira/browse/FLINK-19839 I'm trying to fix this > in the next hours. > You can see if the e2e test failed if there's a warning with the cache > upload, as with this example: > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8385&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529 > > I have checked the recent master builds, and the e2e tests on master seem > to be stable. Until I've fixed this, please take a quick look onto the CI > overview page of your PR before merging! Thanks & sorry! > > > > On Mon, Oct 19, 2020 at 11:15 AM Till Rohrmann <trohrm...@apache.org> > wrote: > >> Thanks for the reminder Robert. I think you are completely right. Since we >> are getting closer to the feature freeze and therefore also the testing >> period, having green builds and only merging PRs where we are sure that CI >> passes will decrease the likelihood of introducing new bugs and, hence, it >> will decrease the time we need for hardening the release. Of course, this >> should also be the modus operandi for not only shortly before the feature >> freeze. >> >> Cheers, >> Till >> >> On Mon, Oct 19, 2020 at 11:00 AM Robert Metzger <rmetz...@apache.org> >> wrote: >> >> > Hi folks, >> > >> > As we are approaching the final stages of the Flink 1.12 release cycle, >> I >> > would like to remind everybody to *only merge pull requests when the CI >> > system gives green light*! >> > I know that the CI system sometimes shows "FAILED" due to CI system >> > instabilities, but please carefully check those cases to make sure it is >> > really the CI systems fault, and not an unstable or failing test. >> > It is much easier to identify and fix an unstable or failing test in the >> > context of a pull request then after it's been merged. >> > >> > An essential part of keeping our CI system useful is helping to address >> > build instabilities early and proactively. >> > In this phase, I'll try to proactively revert commits introducing severe >> > issues, or disable failing/unstable tests early to keep the CI system >> > meaningful. >> > >> > Please post in this thread or reach out to me personally if you are >> > uncertain about a build failure or the overall state of the CI system. >> > >> > Best, >> > Robert >> > >> >
