Hi All, This is only reproducible when I have 3 nodes in my cluster even in the start of the app, everything works fine on 2 nodes.
I have tried this again and faced the same error, this time I increased MAX_PARTITION_FETCH_BYTES_CONFIG to 10MB from default 1MB, still getting the same error. Any thoughts ? [2017-12-12 17:28:36,822] INFO [GroupCoordinator 90]: Preparing to rebalance group c-7-aq32 with old generation 25 (__consumer_offsets-39) (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:40,500] INFO [GroupCoordinator 90]: Stabilized group c-7-aq32 generation 26 (__consumer_offsets-39) (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:42,290] INFO [GroupCoordinator 90]: Assignment received from leader for group c-7-aq32 for generation 26 (kafka.coordinator.group.GroupCoordinator) *[2017-12-12 17:28:42,300] ERROR [GroupMetadataManager brokerId=90] Appending metadata message for group c-7-aq32 generation 26 failed due to org.apache.kafka.common.errors.RecordTooLargeException, returning UNKNOWN error code to the client (kafka.coordinator.group.GroupMetadataManager)* [2017-12-12 17:28:42,301] INFO [GroupCoordinator 90]: Preparing to rebalance group c-7-aq32 with old generation 26 (__consumer_offsets-39) (kafka.coordinator.group.GroupCoordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-1 3-consumer-e0cc0931-0619-4908-82c2-28f7bf9bace9 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-1 2-consumer-b3435055-3e95-47e3-b54d-1aff2b668a8e in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-8 -consumer-1385f1ce-cd8e-411c-acdf-6e51e0ae889e in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-2 -consumer-be3a159b-bb84-405a-954d-1f36ee22abe6 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-5 -consumer-f0b62435-a2db-48f0-aae9-a36c0a01ab41 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-4 -consumer-a36855f9-e015-444b-a5f9-684a59ada6c1 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-5 -consumer-6b8a4555-6ec4-409b-8437-98711af5ad69 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-2 -consumer-d62b0d68-9802-4d3c-9316-64ac5e4b755c in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-5 -consumer-c608b36e-6e83-49bd-9f75-7de86ca83bdd in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-1 0-consumer-9ba5e1e2-0478-4acd-b08d-ff37fb6f4e96 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-1 -consumer-10e57922-235c-454d-ba7d-d3d3737f2465 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-8 -consumer-6befe815-3f9e-4d40-a2b1-e6f7ccc28a39 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-1 4-consumer-58ae1271-5e3e-4ed6-a259-25c989faa9e0 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-9 -consumer-f5c39812-6843-4d31-9755-4ff39b4f6c7d in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread-3 -consumer-7a97d164-dc77-4071-8a88-711fee624233 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread-1 2-consumer-e24b7bc8-000e-4ce5-9e2e-603832ffbeed in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-1 4-consumer-f41cff13-5346-431b-b8f7-209d5df18651 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-4 -consumer-f590b670-c43a-423d-a3d7-b36765d9ed57 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread-1 -consumer-fc145893-a22e-4338-a4a2-7b46370060f6 in group c-7-aq32 has failed, removing it from the group (kafka.coordinator.group.Group Coordinator) On Wed, Dec 13, 2017 at 11:43 AM, Sameer Kumar <sam.kum.w...@gmail.com> wrote: > Hi All, > > Any pointers to the above issue that I can explore further. > > -Sameer. > > On Tue, Dec 12, 2017 at 5:40 PM, Sameer Kumar <sam.kum.w...@gmail.com> > wrote: > >> HI Ismael, >> >> This is what I see in the logs, I tried this twice and got the same >> exception. >> >> This is only reproducible when I have 3 nodes in my cluster even in the >> start of the app, everything works fine on 2 nodes. >> >> [2017-12-12 17:28:36,822] INFO [GroupCoordinator 90]: Preparing to >> rebalance group c-7-aq32 with old generation 25 (__consumer_offsets-39) >> (kafka.coordinator.group.GroupCoordinator) >> [2017-12-12 17:28:40,500] INFO [GroupCoordinator 90]: Stabilized group >> c-7-aq32 generation 26 (__consumer_offsets-39) >> (kafka.coordinator.group.GroupCoordinator) >> [2017-12-12 17:28:42,290] INFO [GroupCoordinator 90]: Assignment received >> from leader for group c-7-aq32 for generation 26 >> (kafka.coordinator.group.GroupCoordinator) >> *[2017-12-12 17:28:42,300] ERROR [GroupMetadataManager brokerId=90] >> Appending metadata message for group c-7-aq32 generation 26 failed due to >> org.apache.kafka.common.errors.RecordTooLargeException, returning UNKNOWN >> error code to the client (kafka.coordinator.group.GroupMetadataManager)* >> [2017-12-12 17:28:42,301] INFO [GroupCoordinator 90]: Preparing to >> rebalance group c-7-aq32 with old generation 26 (__consumer_offsets-39) >> (kafka.coordinator.group.GroupCoordinator) >> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread- >> 13-consumer-e0cc0931-0619-4908-82c2-28f7bf9bace9 in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread- >> 12-consumer-b3435055-3e95-47e3-b54d-1aff2b668a8e in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member >> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread- >> 8-consumer-1385f1ce-cd8e-411c-acdf-6e51e0ae889e in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread- >> 2-consumer-be3a159b-bb84-405a-954d-1f36ee22abe6 in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member >> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread- >> 5-consumer-f0b62435-a2db-48f0-aae9-a36c0a01ab41 in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread- >> 4-consumer-a36855f9-e015-444b-a5f9-684a59ada6c1 in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread- >> 5-consumer-6b8a4555-6ec4-409b-8437-98711af5ad69 in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread- >> 2-consumer-d62b0d68-9802-4d3c-9316-64ac5e4b755c in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread- >> 5-consumer-c608b36e-6e83-49bd-9f75-7de86ca83bdd in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread- >> 10-consumer-9ba5e1e2-0478-4acd-b08d-ff37fb6f4e96 in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,301] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread- >> 1-consumer-10e57922-235c-454d-ba7d-d3d3737f2465 in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread- >> 8-consumer-6befe815-3f9e-4d40-a2b1-e6f7ccc28a39 in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread- >> 14-consumer-58ae1271-5e3e-4ed6-a259-25c989faa9e0 in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread- >> 9-consumer-f5c39812-6843-4d31-9755-4ff39b4f6c7d in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6c5469ff-501c-4bae-aa3e-2a4b8fff9949-StreamThread- >> 3-consumer-7a97d164-dc77-4071-8a88-711fee624233 in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member >> c-7-aq32-6138ec53-4aff-4596-8f4b-44ae6f5d72da-StreamThread- >> 12-consumer-e24b7bc8-000e-4ce5-9e2e-603832ffbeed in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member >> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread- >> 14-consumer-f41cff13-5346-431b-b8f7-209d5df18651 in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member >> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread- >> 4-consumer-f590b670-c43a-423d-a3d7-b36765d9ed57 in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017-12-12 17:28:52,302] INFO [GroupCoordinator 90]: Member >> c-7-aq32-e78df498-16ea-4be0-9c39-7851cf90682b-StreamThread- >> 1-consumer-fc145893-a22e-4338-a4a2-7b46370060f6 in group c-7-aq32 has >> failed, removing it from the group (kafka.coordinator.group.Group >> Coordinator) >> [2017- >> >> On Tue, Dec 12, 2017 at 4:07 PM, Ismael Juma <ism...@juma.me.uk> wrote: >> >>> Can you please check the broker logs for errors? >>> >>> Ismael >>> >>> On Tue, Dec 12, 2017 at 12:10 PM, Sameer Kumar <sam.kum.w...@gmail.com> >>> wrote: >>> >>> > Hi All, >>> > >>> > Facing an strange exception while running Kafka Streams. I am reading >>> from >>> > a topic of 60 partitions. I am using exactly once in Kafka 1.0.0. >>> > >>> > Now, this error has started appearing recently. The application runs >>> fine >>> > on 2 nodes, but as soon as a 3rd node is added, it starts throwing >>> > exception. Please find the stacktrace attached. >>> > >>> > 2017-12-12 15:32:55 ERROR Kafka010Base:47 - Exception caught in thread >>> > c-7-aq32-5648256f-9142-49e2-98c0-da792e6da48e-StreamThread-5 >>> > org.apache.kafka.common.KafkaException: Unexpected error from >>> SyncGroup: >>> > The server experienced an unexpected error when processing the request >>> > at org.apache.kafka.clients.consumer.internals. >>> > AbstractCoordinator$SyncGroupResponseHandler.handle(Abstract >>> Coordinator. >>> > java:566) >>> > at org.apache.kafka.clients.consumer.internals. >>> > AbstractCoordinator$SyncGroupResponseHandler.handle(Abstract >>> Coordinator. >>> > java:539) >>> > at org.apache.kafka.clients.consumer.internals. >>> > AbstractCoordinator$CoordinatorResponseHandler. >>> > onSuccess(AbstractCoordinator.java:808) >>> > at org.apache.kafka.clients.consumer.internals. >>> > AbstractCoordinator$CoordinatorResponseHandler. >>> > onSuccess(AbstractCoordinator.java:788) >>> > at org.apache.kafka.clients.consumer.internals. >>> > RequestFuture$1.onSuccess(RequestFuture.java:204) >>> > at org.apache.kafka.clients.consumer.internals. >>> > RequestFuture.fireSuccess(RequestFuture.java:167) >>> > at org.apache.kafka.clients.consumer.internals. >>> > RequestFuture.complete(RequestFuture.java:127) >>> > at org.apache.kafka.clients.consumer.internals. >>> > ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion( >>> > ConsumerNetworkClient.java:506) >>> > at org.apache.kafka.clients.consumer.internals. >>> > ConsumerNetworkClient.firePendingCompletedRequests( >>> > ConsumerNetworkClient.java:353) >>> > at org.apache.kafka.clients.consumer.internals. >>> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:268) >>> > at org.apache.kafka.clients.consumer.internals. >>> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214) >>> > at org.apache.kafka.clients.consumer.internals. >>> > ConsumerNetworkClient.poll(ConsumerNetworkClient.java:174) >>> > at org.apache.kafka.clients.consumer.internals. >>> > AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:364) >>> > at org.apache.kafka.clients.consumer.internals. >>> > AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316) >>> > at org.apache.kafka.clients.consumer.internals. >>> > ConsumerCoordinator.poll(ConsumerCoordinator.java:295) >>> > at org.apache.kafka.clients.consumer.KafkaConsumer. >>> > pollOnce(KafkaConsumer.java:1138) >>> > at org.apache.kafka.clients.consumer.KafkaConsumer.poll( >>> > KafkaConsumer.java:1103) >>> > at org.apache.kafka.streams.processor.internals. >>> > StreamThread.pollRequests(StreamThread.java:851) >>> > at org.apache.kafka.streams.processor.internals. >>> > StreamThread.runOnce(StreamThread.java:808) >>> > at org.apache.kafka.streams.processor.internals. >>> > StreamThread.runLoop(StreamThread.java:774) >>> > at org.apache.kafka.streams.processor.internals. >>> > StreamThread.run(StreamThread.java:744) >>> > >>> > >>> > >>> > -Sameer. >>> > >>> >> >> >
