Coordinator issues with 0.9 consumer
Hi, I'm tracking the 0.9.0.0 Git tag and have a Java consumer using the new API, but I'm seeing some strange issues. I run ZooKeeper and Kafka on my own machine using the settings files in config/ and no authentication. Build is done using Oracle JDK 8. I have 13 topics, each created with a replication factor of 1. The first time my consumer connects to the newly created topics (before anything has been written to the topic) it emits the following errors: [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : 0.9.0.0 [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : fe855f98263cb7b8 [main] ERROR org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto offset commit failed. org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException: The group coordinator is not available. [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Marking the coordinator 2147483647 dead. [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Attempt to join group xxx-yyy-reader failed due to obsolete coordinator information, retrying. [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Marking the coordinator 2147483647 dead. [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Attempt to join group xxx-yyy-reader failed due to obsolete coordinator information, retrying. If I restart my consumer then it starts without issue. When I start writing to the topics then my consumer reads data, but will often log messages like these: [main] ERROR org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto offset commit failed. org.apache.kafka.clients.consumer.internals.SendFailedException And occasionally these: [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Attempt to heart beat failed since member id is not valid, reset it and try to re-join group. [main] ERROR org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Error ILLEGAL_GENERATION occurred while committing offsets for group audience-indexer-reader [main] ERROR org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto offset commit failed. org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to group rebalance at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:497) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:438) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:673) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:646) at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167) at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:350) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:288) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:303) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:197) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:187) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:157) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:352) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.maybeAutoCommitOffsetsSync(ConsumerCoordinator.java:384) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:236) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:209) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:304) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:861) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:829) Does anyone have any input on what might be happening? -- Martin Skøtt
Re: Coordinator issues with 0.9 consumer
Hi Martin, Thanks for reporting this problem. I think maybe we're just not doing a very good job of handling auto-commit errors internally and they end up spilling into user logs. I added a JIRA to address this issue: https://issues.apache.org/jira/browse/KAFKA-2860. -Jason On Wed, Nov 18, 2015 at 10:16 AM, Guozhang Wangwrote: > Hello Martin, > > Could you paste the consumer config values in this thread as well? And is > the consumer co-located with the broker? > > Guozhang > > On Wed, Nov 18, 2015 at 7:40 AM, Martin Skøtt < > martin.sko...@falconsocial.com> wrote: > > > Hi, > > > > I'm tracking the 0.9.0.0 Git tag and have a Java consumer using the new > > API, but I'm seeing some strange issues. I run ZooKeeper and Kafka on my > > own machine using the settings files in config/ and no authentication. > > Build is done using Oracle JDK 8. I have 13 topics, each created with a > > replication factor of 1. > > > > The first time my consumer connects to the newly created topics (before > > anything has been written to the topic) it emits the following errors: > > > > [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : > > 0.9.0.0 > > [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId > : > > fe855f98263cb7b8 > > [main] ERROR > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto > > offset commit failed. > > org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException: The > > group coordinator is not available. > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Marking the coordinator 2147483647 dead. > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Attempt to join group xxx-yyy-reader failed due to obsolete coordinator > > information, retrying. > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Marking the coordinator 2147483647 dead. > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Attempt to join group xxx-yyy-reader failed due to obsolete coordinator > > information, retrying. > > > > If I restart my consumer then it starts without issue. > > > > When I start writing to the topics then my consumer reads data, but will > > often log messages like these: > > > > [main] ERROR > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto > > offset commit failed. > > org.apache.kafka.clients.consumer.internals.SendFailedException > > > > And occasionally these: > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Attempt to heart beat failed since member id is not valid, reset it and > > try to re-join group. > > [main] ERROR > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Error > > ILLEGAL_GENERATION occurred while committing offsets for group > > audience-indexer-reader > > [main] ERROR > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto > > offset commit failed. > > org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be > > completed due to group rebalance > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:497) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:438) > > at > > > > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:673) > > at > > > > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:646) > > at > > > > > org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167) > > at > > > > > org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) > > at > > > > > org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:350) > > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:288) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:303) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:197) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:187) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:157) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:352) > > at > > > > >
Re: Coordinator issues with 0.9 consumer
Hello Martin, Could you paste the consumer config values in this thread as well? And is the consumer co-located with the broker? Guozhang On Wed, Nov 18, 2015 at 7:40 AM, Martin Skøtt < martin.sko...@falconsocial.com> wrote: > Hi, > > I'm tracking the 0.9.0.0 Git tag and have a Java consumer using the new > API, but I'm seeing some strange issues. I run ZooKeeper and Kafka on my > own machine using the settings files in config/ and no authentication. > Build is done using Oracle JDK 8. I have 13 topics, each created with a > replication factor of 1. > > The first time my consumer connects to the newly created topics (before > anything has been written to the topic) it emits the following errors: > > [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : > 0.9.0.0 > [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : > fe855f98263cb7b8 > [main] ERROR > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto > offset commit failed. > org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException: The > group coordinator is not available. > [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator > - Marking the coordinator 2147483647 dead. > [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator > - Attempt to join group xxx-yyy-reader failed due to obsolete coordinator > information, retrying. > [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator > - Marking the coordinator 2147483647 dead. > [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator > - Attempt to join group xxx-yyy-reader failed due to obsolete coordinator > information, retrying. > > If I restart my consumer then it starts without issue. > > When I start writing to the topics then my consumer reads data, but will > often log messages like these: > > [main] ERROR > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto > offset commit failed. > org.apache.kafka.clients.consumer.internals.SendFailedException > > And occasionally these: > [main] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator > - Attempt to heart beat failed since member id is not valid, reset it and > try to re-join group. > [main] ERROR > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Error > ILLEGAL_GENERATION occurred while committing offsets for group > audience-indexer-reader > [main] ERROR > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto > offset commit failed. > org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be > completed due to group rebalance > at > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:497) > at > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:438) > at > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:673) > at > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:646) > at > > org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167) > at > > org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) > at > > org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) > at > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:350) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:288) > at > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:303) > at > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:197) > at > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:187) > at > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:157) > at > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:352) > at > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.maybeAutoCommitOffsetsSync(ConsumerCoordinator.java:384) > at > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:236) > at > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:209) > at > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:304) > at > > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:861) > at > >
Re: Coordinator issues with 0.9 consumer
Hi Guozhang, The consumer, broker, and zookeeper are all on the same machine - just testing out Kafka at the moment :) Here are the configuration values I set: properties.setProperty("bootstrap.servers", "localhost:9092"); properties.setProperty("group.id", "xxx-yyy-reader"); properties.setProperty("session.timeout.ms", "3"); properties.setProperty("enable.auto.commit", "true"); properties.setProperty("auto.commit.interval.ms", "1"); properties.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); properties.setProperty("value.deserializer", "xxxyyy.kafka.JsonDeserializer"); properties.setProperty("client.id", "xxx-yyy-consumer"); I guess they are fairly standard. Martin On 18 November 2015 at 19:16, Guozhang Wangwrote: > Hello Martin, > > Could you paste the consumer config values in this thread as well? And is > the consumer co-located with the broker? > > Guozhang > > On Wed, Nov 18, 2015 at 7:40 AM, Martin Skøtt < > martin.sko...@falconsocial.com> wrote: > > > Hi, > > > > I'm tracking the 0.9.0.0 Git tag and have a Java consumer using the new > > API, but I'm seeing some strange issues. I run ZooKeeper and Kafka on my > > own machine using the settings files in config/ and no authentication. > > Build is done using Oracle JDK 8. I have 13 topics, each created with a > > replication factor of 1. > > > > The first time my consumer connects to the newly created topics (before > > anything has been written to the topic) it emits the following errors: > > > > [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : > > 0.9.0.0 > > [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId > : > > fe855f98263cb7b8 > > [main] ERROR > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto > > offset commit failed. > > org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException: The > > group coordinator is not available. > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Marking the coordinator 2147483647 dead. > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Attempt to join group xxx-yyy-reader failed due to obsolete coordinator > > information, retrying. > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Marking the coordinator 2147483647 dead. > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Attempt to join group xxx-yyy-reader failed due to obsolete coordinator > > information, retrying. > > > > If I restart my consumer then it starts without issue. > > > > When I start writing to the topics then my consumer reads data, but will > > often log messages like these: > > > > [main] ERROR > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto > > offset commit failed. > > org.apache.kafka.clients.consumer.internals.SendFailedException > > > > And occasionally these: > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Attempt to heart beat failed since member id is not valid, reset it and > > try to re-join group. > > [main] ERROR > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Error > > ILLEGAL_GENERATION occurred while committing offsets for group > > audience-indexer-reader > > [main] ERROR > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto > > offset commit failed. > > org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be > > completed due to group rebalance > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:497) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:438) > > at > > > > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:673) > > at > > > > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:646) > > at > > > > > org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167) > > at > > > > > org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) > > at > > > > > org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:350) > > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:288) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:303) > > at > > > > >