[jira] [Commented] (KAFKA-4801) Transient test failure (part 2): ConsumerBounceTest.testConsumptionWithBrokerFailures
[ https://issues.apache.org/jira/browse/KAFKA-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995863#comment-15995863 ] Jason Gustafson commented on KAFKA-4801: After my patch was merged, this seems to now be showing up as this: {code} kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures FAILED org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:776) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:722) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:797) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:778) at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204) at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167) at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:486) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:346) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:260) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:206) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:182) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:589) at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1104) at kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:113) at kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:84) {code} This suggests that the consumer is falling out of the group in the test case for whatever reason. > Transient test failure (part 2): > ConsumerBounceTest.testConsumptionWithBrokerFailures > - > > Key: KAFKA-4801 > URL: https://issues.apache.org/jira/browse/KAFKA-4801 > Project: Kafka > Issue Type: Sub-task >Reporter: Armin Braun >Assignee: Jason Gustafson >Priority: Minor > Labels: transient-system-test-failure > > There is still some (but very little ... when reproducing this you need more > than 100 runs in half the cases statistically) instability left in the test > {code} > ConsumerBounceTest.testConsumptionWithBrokerFailures > {code} > Resulting in this exception being thrown at a relatively low rate (I'd say > def less than 0.5% of all runs on my machine). > {code} > kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures FAILED > java.lang.IllegalArgumentException: You can only check the position for > partitions assigned to this consumer. > at > org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1271) > at > kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:96) > at > kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:69) > {code} > this was also reported in a comment to the original KAFKA-4198 > https://issues.apache.org/jira/browse/KAFKA-4198?focusedCommentId=15765468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15765468 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4801) Transient test failure (part 2): ConsumerBounceTest.testConsumptionWithBrokerFailures
[ https://issues.apache.org/jira/browse/KAFKA-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995872#comment-15995872 ] Guozhang Wang commented on KAFKA-4801: -- [~hachikuji] Saw this happen again but with a different stack trace: {code} kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures FAILED java.lang.AssertionError: expected: but was: at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:834) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:105) at kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:84) {code} > Transient test failure (part 2): > ConsumerBounceTest.testConsumptionWithBrokerFailures > - > > Key: KAFKA-4801 > URL: https://issues.apache.org/jira/browse/KAFKA-4801 > Project: Kafka > Issue Type: Sub-task >Reporter: Armin Braun >Assignee: Jason Gustafson >Priority: Minor > Labels: transient-system-test-failure > > There is still some (but very little ... when reproducing this you need more > than 100 runs in half the cases statistically) instability left in the test > {code} > ConsumerBounceTest.testConsumptionWithBrokerFailures > {code} > Resulting in this exception being thrown at a relatively low rate (I'd say > def less than 0.5% of all runs on my machine). > {code} > kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures FAILED > java.lang.IllegalArgumentException: You can only check the position for > partitions assigned to this consumer. > at > org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1271) > at > kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:96) > at > kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:69) > {code} > this was also reported in a comment to the original KAFKA-4198 > https://issues.apache.org/jira/browse/KAFKA-4198?focusedCommentId=15765468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15765468 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4801) Transient test failure (part 2): ConsumerBounceTest.testConsumptionWithBrokerFailures
[ https://issues.apache.org/jira/browse/KAFKA-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996438#comment-15996438 ] Ismael Juma commented on KAFKA-4801: [~original-brownbear], feel free to try to reproduce this. Given how frequently it's happening in Jenkins now, we may have to disable it if we can't figure out the reason soon. > Transient test failure (part 2): > ConsumerBounceTest.testConsumptionWithBrokerFailures > - > > Key: KAFKA-4801 > URL: https://issues.apache.org/jira/browse/KAFKA-4801 > Project: Kafka > Issue Type: Sub-task >Reporter: Armin Braun >Assignee: Jason Gustafson >Priority: Minor > Labels: transient-system-test-failure > > There is still some (but very little ... when reproducing this you need more > than 100 runs in half the cases statistically) instability left in the test > {code} > ConsumerBounceTest.testConsumptionWithBrokerFailures > {code} > Resulting in this exception being thrown at a relatively low rate (I'd say > def less than 0.5% of all runs on my machine). > {code} > kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures FAILED > java.lang.IllegalArgumentException: You can only check the position for > partitions assigned to this consumer. > at > org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1271) > at > kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:96) > at > kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:69) > {code} > this was also reported in a comment to the original KAFKA-4198 > https://issues.apache.org/jira/browse/KAFKA-4198?focusedCommentId=15765468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15765468 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4801) Transient test failure (part 2): ConsumerBounceTest.testConsumptionWithBrokerFailures
[ https://issues.apache.org/jira/browse/KAFKA-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996442#comment-15996442 ] Armin Braun commented on KAFKA-4801: [~ijuma] will take a look very soon then (~2h :)) > Transient test failure (part 2): > ConsumerBounceTest.testConsumptionWithBrokerFailures > - > > Key: KAFKA-4801 > URL: https://issues.apache.org/jira/browse/KAFKA-4801 > Project: Kafka > Issue Type: Sub-task >Reporter: Armin Braun >Assignee: Armin Braun >Priority: Minor > Labels: transient-system-test-failure > > There is still some (but very little ... when reproducing this you need more > than 100 runs in half the cases statistically) instability left in the test > {code} > ConsumerBounceTest.testConsumptionWithBrokerFailures > {code} > Resulting in this exception being thrown at a relatively low rate (I'd say > def less than 0.5% of all runs on my machine). > {code} > kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures FAILED > java.lang.IllegalArgumentException: You can only check the position for > partitions assigned to this consumer. > at > org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1271) > at > kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:96) > at > kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:69) > {code} > this was also reported in a comment to the original KAFKA-4198 > https://issues.apache.org/jira/browse/KAFKA-4198?focusedCommentId=15765468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15765468 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4801) Transient test failure (part 2): ConsumerBounceTest.testConsumptionWithBrokerFailures
[ https://issues.apache.org/jira/browse/KAFKA-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951392#comment-15951392 ] Armin Braun commented on KAFKA-4801: Just for the record, I stopped being able to reproduce this one. Unless this is still happening on Jenkins I think ti can be closed. > Transient test failure (part 2): > ConsumerBounceTest.testConsumptionWithBrokerFailures > - > > Key: KAFKA-4801 > URL: https://issues.apache.org/jira/browse/KAFKA-4801 > Project: Kafka > Issue Type: Sub-task >Reporter: Armin Braun >Priority: Minor > Labels: transient-system-test-failure > > There is still some (but very little ... when reproducing this you need more > than 100 runs in half the cases statistically) instability left in the test > {code} > ConsumerBounceTest.testConsumptionWithBrokerFailures > {code} > Resulting in this exception being thrown at a relatively low rate (I'd say > def less than 0.5% of all runs on my machine). > {code} > kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures FAILED > java.lang.IllegalArgumentException: You can only check the position for > partitions assigned to this consumer. > at > org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1271) > at > kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:96) > at > kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:69) > {code} > this was also reported in a comment to the original KAFKA-4198 > https://issues.apache.org/jira/browse/KAFKA-4198?focusedCommentId=15765468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15765468 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4801) Transient test failure (part 2): ConsumerBounceTest.testConsumptionWithBrokerFailures
[ https://issues.apache.org/jira/browse/KAFKA-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987843#comment-15987843 ] Jason Gustafson commented on KAFKA-4801: Recent occurrence: https://jenkins.confluent.io/job/kafka-trunk/1847/console. > Transient test failure (part 2): > ConsumerBounceTest.testConsumptionWithBrokerFailures > - > > Key: KAFKA-4801 > URL: https://issues.apache.org/jira/browse/KAFKA-4801 > Project: Kafka > Issue Type: Sub-task >Reporter: Armin Braun >Priority: Minor > Labels: transient-system-test-failure > > There is still some (but very little ... when reproducing this you need more > than 100 runs in half the cases statistically) instability left in the test > {code} > ConsumerBounceTest.testConsumptionWithBrokerFailures > {code} > Resulting in this exception being thrown at a relatively low rate (I'd say > def less than 0.5% of all runs on my machine). > {code} > kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures FAILED > java.lang.IllegalArgumentException: You can only check the position for > partitions assigned to this consumer. > at > org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1271) > at > kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:96) > at > kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:69) > {code} > this was also reported in a comment to the original KAFKA-4198 > https://issues.apache.org/jira/browse/KAFKA-4198?focusedCommentId=15765468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15765468 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4801) Transient test failure (part 2): ConsumerBounceTest.testConsumptionWithBrokerFailures
[ https://issues.apache.org/jira/browse/KAFKA-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987910#comment-15987910 ] Armin Braun commented on KAFKA-4801: Will try to reproduce again tomorrow ... > Transient test failure (part 2): > ConsumerBounceTest.testConsumptionWithBrokerFailures > - > > Key: KAFKA-4801 > URL: https://issues.apache.org/jira/browse/KAFKA-4801 > Project: Kafka > Issue Type: Sub-task >Reporter: Armin Braun >Assignee: Armin Braun >Priority: Minor > Labels: transient-system-test-failure > > There is still some (but very little ... when reproducing this you need more > than 100 runs in half the cases statistically) instability left in the test > {code} > ConsumerBounceTest.testConsumptionWithBrokerFailures > {code} > Resulting in this exception being thrown at a relatively low rate (I'd say > def less than 0.5% of all runs on my machine). > {code} > kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures FAILED > java.lang.IllegalArgumentException: You can only check the position for > partitions assigned to this consumer. > at > org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1271) > at > kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:96) > at > kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:69) > {code} > this was also reported in a comment to the original KAFKA-4198 > https://issues.apache.org/jira/browse/KAFKA-4198?focusedCommentId=15765468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15765468 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4801) Transient test failure (part 2): ConsumerBounceTest.testConsumptionWithBrokerFailures
[ https://issues.apache.org/jira/browse/KAFKA-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988003#comment-15988003 ] Jason Gustafson commented on KAFKA-4801: [~original-brownbear] I couldn't reproduce the failure locally either. At a glance, the test case doesn't look totally safe since we don't actually ensure that the partition was assigned to the consumer before calling {{position}}. A simple way to make the test more resilient would be to only execute the {{commitSync}} block if the call to {{poll}} returned a non-empty record set. That said, it's unclear in what situation we would lose the partition assignment, so it would be good to understand that before patching the test case if possible. > Transient test failure (part 2): > ConsumerBounceTest.testConsumptionWithBrokerFailures > - > > Key: KAFKA-4801 > URL: https://issues.apache.org/jira/browse/KAFKA-4801 > Project: Kafka > Issue Type: Sub-task >Reporter: Armin Braun >Assignee: Armin Braun >Priority: Minor > Labels: transient-system-test-failure > > There is still some (but very little ... when reproducing this you need more > than 100 runs in half the cases statistically) instability left in the test > {code} > ConsumerBounceTest.testConsumptionWithBrokerFailures > {code} > Resulting in this exception being thrown at a relatively low rate (I'd say > def less than 0.5% of all runs on my machine). > {code} > kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures FAILED > java.lang.IllegalArgumentException: You can only check the position for > partitions assigned to this consumer. > at > org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1271) > at > kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:96) > at > kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:69) > {code} > this was also reported in a comment to the original KAFKA-4198 > https://issues.apache.org/jira/browse/KAFKA-4198?focusedCommentId=15765468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15765468 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-4801) Transient test failure (part 2): ConsumerBounceTest.testConsumptionWithBrokerFailures
[ https://issues.apache.org/jira/browse/KAFKA-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988049#comment-15988049 ] Jason Gustafson commented on KAFKA-4801: I submitted a patch to make the expectation that the given topic partition is assigned explicit: https://github.com/apache/kafka/pull/2930. I expect the new assertion to now fail instead of the call to {{position}}. > Transient test failure (part 2): > ConsumerBounceTest.testConsumptionWithBrokerFailures > - > > Key: KAFKA-4801 > URL: https://issues.apache.org/jira/browse/KAFKA-4801 > Project: Kafka > Issue Type: Sub-task >Reporter: Armin Braun >Assignee: Armin Braun >Priority: Minor > Labels: transient-system-test-failure > > There is still some (but very little ... when reproducing this you need more > than 100 runs in half the cases statistically) instability left in the test > {code} > ConsumerBounceTest.testConsumptionWithBrokerFailures > {code} > Resulting in this exception being thrown at a relatively low rate (I'd say > def less than 0.5% of all runs on my machine). > {code} > kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures FAILED > java.lang.IllegalArgumentException: You can only check the position for > partitions assigned to this consumer. > at > org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1271) > at > kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:96) > at > kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:69) > {code} > this was also reported in a comment to the original KAFKA-4198 > https://issues.apache.org/jira/browse/KAFKA-4198?focusedCommentId=15765468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15765468 -- This message was sent by Atlassian JIRA (v6.3.15#6346)