FYI, there's also this which I just reported: https://issues.apache.org/jira/browse/INFRA-16808
Patrick On Fri, Jul 20, 2018 at 12:01 AM Patrick Hunt <ph...@apache.org> wrote: > Something that's significantly different about the 3.4 and 3.5/master > Jenkins jobs is that 3.5/master has > > test.junit.threads=8 > > set while this is not supported in 3.4 (see build.xml). It's very likely > that the paralyzation of the tests is causing the discrepancy. > > setting threads > 1 significantly improves the speed of the jobs, that's > why it was originally added to 3.5+. > See a358280fb2b3cc7852cded3fe67769765a519beb > > Perhaps we should try one/more of the 3.5/master jobs with threads=1 and > see? > > Patrick > > > > On Thu, Jul 19, 2018 at 1:26 PM Molnár Andor <an...@nu.hu> wrote: > >> Sorry guys for this aweful email. Looks like Apache converted my nicely >> illustrated email into plain text. :( >> >> Maybe I could attach the test reports as images, but I think you already >> got the idea. >> >> >> Andor >> >> >> >> On 07/18/2018 05:42 PM, Andor Molnar wrote: >> > Hi, >> > >> > *branch-3.4* >> > >> > I've taken a quick look at our Jenkins builds and in terms of flaky >> tests, >> > it looks like branch-3.4 is in a pretty good shape. The build hasn't >> failed >> > for 5-6 days on all JDKs which I think is pretty awesome. >> > >> > *branch-3.5* >> > >> > This branch is in very bad condition. Which is quite unfortunate given >> > we're in the middle of stabilising it. :) >> > Especially on JDK8, last successful build was 11 days ago. JDK9 (50% >> > failing) and JDK10 (30% failing) are looking better in the last 10 >> builds. >> > >> > Interestingly (apart from a few quite rare ones) it looks there's only 1 >> > test which is quite nasty on this branch: testManyChildWatchersAutoReset >> > >> > There's a Jira about fixing it and a fix has been merged by increasing >> the >> > timeout of the test, but having a bug on the branch is also possible >> > causing the test to fail even with 10 min timeout. >> > >> > I wasn't able to repro the failing test on my machine (Mac and >> CentOS7), it >> > always finished in 30-40 seconds maximum. On jenkins slaves it shows the >> > following: >> > >> > *JDK 8:* >> > >> > Report creation timed out. >> > >> > >> > *JDK 9:* >> > >> > New Failures >> > Chart >> > See children >> > Build Number ⇒ >> > Package-Class-Testmethod names ⇓ >> > 351 >> > 350 >> > 349 >> > 348 >> > 347 >> > 346 >> > 345 >> > 344 >> > 343 >> > 342 >> > 341 >> > 340 >> > 339 >> > 338 >> > 337 >> > 336 >> > 335 >> > 334 >> > testManyChildWatchersAutoReset >> > 45.604 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/351/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.337 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/350/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 21.904 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/349/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 583.063 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/348/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.325 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/347/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.383 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/346/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.362 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/345/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 21.139 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/344/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 24.031 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/343/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 584.200 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/342/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.327 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/341/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.323 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/340/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 23.737 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/339/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.406 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/338/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 547.004 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/337/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.393 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/336/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > N/A >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/test_results_analyzer/ >> > >> > 373.955 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java9/334/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > >> > >> > *JDK 10:* >> > >> > >> > New Failures >> > Chart >> > See children >> > Build Number ⇒ >> > Package-Class-Testmethod names ⇓ >> > 110 >> > 109 >> > 108 >> > 107 >> > 106 >> > 105 >> > 104 >> > 103 >> > 102 >> > 101 >> > 100 >> > 99 >> > 98 >> > 97 >> > 96 >> > 95 >> > 94 >> > 93 >> > 92 >> > testManyChildWatchersAutoReset >> > 364.945 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/110/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 543.983 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/109/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 388.182 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/108/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.446 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/107/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.025 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/106/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 535.046 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/105/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.306 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/104/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 474.005 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/103/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 560.925 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/102/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.328 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/101/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 558.547 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/100/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.397 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/99/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.414 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/98/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 430.383 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/97/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 564.064 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/96/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 600.357 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/95/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 432.435 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/94/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 596.378 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/93/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > 39.242 >> > < >> https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper_branch35_java10/92/testReport/org.apache.zookeeper.test/DisconnectedWatcherTest/testManyChildWatchersAutoReset >> > >> > >> > >> > It takes ages to complete on Jenkins for some reason and it looks like >> it >> > ends quite frequently close to the limit, so I suspect it should succeed >> > eventually if we were to increase the timeout even more. But is that >> > correct? >> > Bug or infrastructure issue? >> > >> > *master / 3.6* >> > >> > Pretty much the same as 3.5. I haven't seen >> testManyChildWatchersAutoReset >> > failing on this branch with JDK8 which is a bit confusing, but other >> then >> > that I see the same pattern on JDK9 and JDK10. Unable to generate the >> above >> > reports here, because Test Result Analyzer keep timeouting for me, but >> I'll >> > follow-up when I have them. >> > >> > Btw. Flaky Test report has been broken for 10 days, I'm going to raise a >> > ticket on that if somebody willing to fix it. (I'm planning to do so.) >> > It would be nice to see the report working again, because if my >> > observations are correct, we don't have too many annoying tests apart >> from >> > the one mentioned. >> > >> > Thanks, >> > Andor >> > >> >>
