[ https://issues.apache.org/jira/browse/BEAM-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16726665#comment-16726665 ]
Etienne Chauchot commented on BEAM-6279: ---------------------------------------- Please note for point 2 that we have the ability now to set the timeout at the Elasticsearch client level . For point one, jenkins might be overloader and ES embedded cluster became unhealthy. As a defensive action I would recommend configuring the embedded cluster for IO v5 and IO v6 to fewer nodes. No need for IO v2, because there is no cluster because I could use embedded node that was removed by the ES team in favor ES test framework that provides an embedded cluster > Failures in ElasticSearchIOTest: testWriteRetryValidRequest (versions 2 & 5) > testDefaultRetryPredicate (version 5) > ------------------------------------------------------------------------------------------------------------------ > > Key: BEAM-6279 > URL: https://issues.apache.org/jira/browse/BEAM-6279 > Project: Beam > Issue Type: Bug > Components: io-java-elasticsearch, test-failures > Reporter: Kenneth Knowles > Assignee: Tim Robertson > Priority: Critical > Labels: flake > > [https://builds.apache.org/job/beam_PreCommit_Java_Cron/730/] > [https://scans.gradle.com/s/6rpbvgwx2nk7c] > I don't see recent changes, so thinking this is a flake. > testDefaultRetryPredicate: > {code:java} > java.lang.AssertionError > : > All incoming requests on node [node_s0] should have finished. Expected 0 but > got 2540 Expected: <0L> but: was <2540L> > at __randomizedtesting.SeedInfo.seed([4CF46FDC92DC07B8:B39A5DDFBC85274E]:0) > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) > at org.junit.Assert.assertThat(Assert.java:956) > at > org.elasticsearch.test.InternalTestCluster.lambda$assertRequestsFinished$17(InternalTestCluster.java:2086) > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:705) > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:679) > at > org.elasticsearch.test.InternalTestCluster.assertRequestsFinished(InternalTestCluster.java:2083) > at > org.elasticsearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2062) > at > org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:581) > at > org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2054) > ... > {code} > This looks like an internal assertion (aka {{checkState}}) failed and the > cluster became unhealthy, maybe? > testWriteRetryValidRequest: > {code:java} > org.apache.beam.sdk.Pipeline$PipelineExecutionException > : > java.io.IOException: listener timeout after waiting for [90000] ms > Open stacktrace > Caused by: > java.io.IOException > : > listener timeout after waiting for [90000] ms > Close stacktrace > at > org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:899) > at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227) > at org.elasticsearch.client.RestClient.performRequest(RestClient.java:321) > at > org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1285) > at > org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.finishBundle(ElasticsearchIO.java:1260){code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)