Re: Trying to find pattern in Flaky Tests

2018-08-01 Thread Patrick Hunt
Looks like 16808 has been resolved - I haven't noticed it after the recent changes. Note that INFRA recently added openjdk10 to Jenkins and I added a job or two which seem to be working OK. Java 11 is failing on 3.4 due to broken libraries (according to Rakesh on another thread) but we're also

Re: Trying to find pattern in Flaky Tests

2018-07-24 Thread Patrick Hunt
FYI, there's also this which I just reported: https://issues.apache.org/jira/browse/INFRA-16808 Patrick On Fri, Jul 20, 2018 at 12:01 AM Patrick Hunt wrote: > Something that's significantly different about the 3.4 and 3.5/master > Jenkins jobs is that 3.5/master has > > test.junit.threads=8 >

Re: Trying to find pattern in Flaky Tests

2018-07-20 Thread Patrick Hunt
Something that's significantly different about the 3.4 and 3.5/master Jenkins jobs is that 3.5/master has test.junit.threads=8 set while this is not supported in 3.4 (see build.xml). It's very likely that the paralyzation of the tests is causing the discrepancy. setting threads > 1

Re: Trying to find pattern in Flaky Tests

2018-07-19 Thread Molnár Andor
Sorry guys for this aweful email. Looks like Apache converted my nicely illustrated email into plain text. :( Maybe I could attach the test reports as images, but I think you already got the idea. Andor On 07/18/2018 05:42 PM, Andor Molnar wrote: > Hi, > > *branch-3.4* > > I've taken a quick

Re: Trying to find pattern in Flaky Tests

2018-07-19 Thread Andor Molnár
Thanks Pat for fixing the report. Very useful. I think it was unable to retrieve the test report, because there was no test report available for those builds at the time it was running. We might want to expose the error code in the below logs, but if you take a look at #104 and #100: test report

Re: Trying to find pattern in Flaky Tests

2018-07-19 Thread Andor Molnár
Thanks Bogdan. Your analysis is much appreciated. According to your findings, it looks like it must be an infrastructure issue. But that doesn't explain why the test is a lot more stable on the 3.4 branch. I'd like to setup new jenkins jobs on a private infrastructure (other than Apache) on

Re: Trying to find pattern in Flaky Tests

2018-07-19 Thread Bogdan Kanivets
Hi Andor, For testManyChildWatchersAutoReset, that was me who put 10min timeout on the test itself. I wanted to see the logs and the problem is that when test is timed out by ant (default 15min) logs aren't captured. I agree that it became much flakier. I've pushed the PR right now to increase to

Re: Trying to find pattern in Flaky Tests

2018-07-18 Thread Michael Han
Thanks Pat for promptly fixing this! I have no idea of the "failed to get" symptoms. Probably we could give it more days and see if the pattern recurs? If not might be a transient infra issue... On Wed, Jul 18, 2018 at 11:16 AM, Patrick Hunt wrote: > Ok, I committed a change that seems to

Re: Trying to find pattern in Flaky Tests

2018-07-18 Thread Patrick Hunt
Ok, I committed a change that seems to address the main failure: https://github.com/apache/zookeeper/commit/06b9507ab78a1a055b8f467846c15791600b72ee https://builds.apache.org/view/S-Z/view/ZooKeeper/job/ZooKeeper-Find-Flaky-Tests/lastSuccessfulBuild/artifact/report.html However I do notice some

Re: Trying to find pattern in Flaky Tests

2018-07-18 Thread Patrick Hunt
FYI, created this: https://issues.apache.org/jira/browse/INFRA-16785 for the security warnings, not sure if that's causing the issue. Likely it's the recent jenkins upgrade, looking into it a bit... Patrick On Wed, Jul 18, 2018 at 9:48 AM Michael Han wrote: > Hi Andor, > > >> I suspect it

Re: Trying to find pattern in Flaky Tests

2018-07-18 Thread Michael Han
Hi Andor, >> I suspect it should succeed eventually if we were to increase the timeout even more. But is that correct? Bug or infrastructure issue? You could set up a dedicated git branch with all patches (e.g. the one in ZOOKEEPER-2251) you want to apply and I can set up a dedicated Jenkins job

Trying to find pattern in Flaky Tests

2018-07-18 Thread Andor Molnar
Hi, *branch-3.4* I've taken a quick look at our Jenkins builds and in terms of flaky tests, it looks like branch-3.4 is in a pretty good shape. The build hasn't failed for 5-6 days on all JDKs which I think is pretty awesome. *branch-3.5* This branch is in very bad condition. Which is quite