Hi Martin, Thanks for reporting this problem. I think maybe we're just not doing a very good job of handling auto-commit errors internally and they end up spilling into user logs. I added a JIRA to address this issue: https://issues.apache.org/jira/browse/KAFKA-2860.
-Jason On Wed, Nov 18, 2015 at 10:16 AM, Guozhang Wang <wangg...@gmail.com> wrote: > Hello Martin, > > Could you paste the consumer config values in this thread as well? And is > the consumer co-located with the broker? > > Guozhang > > On Wed, Nov 18, 2015 at 7:40 AM, Martin Skøtt < > martin.sko...@falconsocial.com> wrote: > > > Hi, > > > > I'm tracking the 0.9.0.0 Git tag and have a Java consumer using the new > > API, but I'm seeing some strange issues. I run ZooKeeper and Kafka on my > > own machine using the settings files in config/ and no authentication. > > Build is done using Oracle JDK 8. I have 13 topics, each created with a > > replication factor of 1. > > > > The first time my consumer connects to the newly created topics (before > > anything has been written to the topic) it emits the following errors: > > > > [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : > > 0.9.0.0 > > [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId > : > > fe855f98263cb7b8 > > [main] ERROR > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto > > offset commit failed. > > org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException: The > > group coordinator is not available. > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Marking the coordinator 2147483647 dead. > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Attempt to join group xxx-yyy-reader failed due to obsolete coordinator > > information, retrying. > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Marking the coordinator 2147483647 dead. > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Attempt to join group xxx-yyy-reader failed due to obsolete coordinator > > information, retrying. > > > > If I restart my consumer then it starts without issue. > > > > When I start writing to the topics then my consumer reads data, but will > > often log messages like these: > > > > [main] ERROR > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto > > offset commit failed. > > org.apache.kafka.clients.consumer.internals.SendFailedException > > > > And occasionally these: > > [main] INFO > org.apache.kafka.clients.consumer.internals.AbstractCoordinator > > - Attempt to heart beat failed since member id is not valid, reset it and > > try to re-join group. > > [main] ERROR > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Error > > ILLEGAL_GENERATION occurred while committing offsets for group > > audience-indexer-reader > > [main] ERROR > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto > > offset commit failed. > > org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be > > completed due to group rebalance > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:497) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:438) > > at > > > > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:673) > > at > > > > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:646) > > at > > > > > org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167) > > at > > > > > org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) > > at > > > > > org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:350) > > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:288) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:303) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:197) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:187) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:157) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:352) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.maybeAutoCommitOffsetsSync(ConsumerCoordinator.java:384) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:236) > > at > > > > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:209) > > at > > > > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:304) > > at > > > > > org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:861) > > at > > > > > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:829) > > > > Does anyone have any input on what might be happening? > > > > -- > > Martin Skøtt > > > > > > -- > -- Guozhang >
