[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247164#comment-17247164 ] Levani Kokhreidze commented on KAFKA-10772: --- Thanks a lot [~cadonna] for the thorough investigation. I'll test 2.5+ brokers and see how it goes. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Assignee: Bruno Cadonna >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247142#comment-17247142 ] Bruno Cadonna commented on KAFKA-10772: --- [~lkokhreidze] [~thebearmayor] To sum up: - This bug is fixed in 2.5 onwards via KAFKA-9801. - You get the {{IllegalStateException}} instead of the error reported in KAFKA-9801 because the {{IllegalStateException}} was introduced in the fix for KAFKA-9801 and you use Kafka Streams 2.6. - You can fix this issue by upgrading your brokers to 2.5 or above. I am going to close this issue as a duplicate of KAFKA-9801. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Assignee: Bruno Cadonna >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246806#comment-17246806 ] Bruno Cadonna commented on KAFKA-10772: --- bq. It seems bad, and quite possibly a regression, for the client to actually fail to deserialize any broker response. Since the faulty assignment in this case is a byte buffer of size zero, the client cannot even fail to deserialize it because there is nothing to deserialize. However, the client could try to re-join in this case. But then it risks to start an infinite rejoin loop because there is not guarantee that next time it would get a non-empty assignment. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Assignee: Bruno Cadonna >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246797#comment-17246797 ] Bruno Cadonna commented on KAFKA-10772: --- I think this issue is fixed for brokers 2.5 and above via https://issues.apache.org/jira/browse/KAFKA-9801. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Assignee: Bruno Cadonna >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246741#comment-17246741 ] John Roesler commented on KAFKA-10772: -- Thanks, [~cadonna] ! The scenario you outline sounds vaguely like this one: [https://github.com/apache/kafka/pull/9270] https://issues.apache.org/jira/browse/KAFKA-10284 If that's right, then the earliest version it is fixed in is 2.5.2. The fix was to broadcast the updated member id synchronously. IIRC, we chose not to backport it more because the broker code is significantly different in 2.4, and we felt the risk of introducing a new, worse, bug was non-trivial. The root cause aside, I'm wondering if the client should handle this condition gracefully. It sounds like the manifestation here is different than what we observed in KAFKA-10284, which was just that the instance could fence itself. It seems bad, and quite possibly a regression, for the client to actually fail to deserialize _any_ broker response. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Assignee: Bruno Cadonna >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17246677#comment-17246677 ] Bruno Cadonna commented on KAFKA-10772: --- I think, I found the bug. If a static member joins the group it gets a member ID that is used to store the assignment for the static member broker side. A static member gets a new member ID each time it joins the group. That means, if a static member joins a group and gets a member ID that is used during the next assignment but before the assignment is broadcasted to the members in the SyncGroup response the same static member joins the group again (e.g. Streams client restart) and gets a new member ID, the group coordinator will not find the new member ID in the assignment and will store an empty assignment (i.e. an empty byte buffer) for the new member ID. Hence, the static member will get an empty assignment and throw the above {{IllegalStateException}}. For example, assume the following static members: * member A with group instance ID A and initial member ID 1 * member B with group instance ID B and initial member ID 2 The group leader will send the following assignment to the group coordinator: 1 -> some assigned partitions 2 -> some other assigned partitions Before the group coordinator gets this assignment and broadcast it, member A rejoins and get member ID 3. The group coordinator will store the following assignment: 2 -> some other assigned partitions 3 -> empty buffer Member A that now has member ID 3 will receive the empty buffer as its assignment. I could reproduce the issue with 2.4.1 brokers, but not with 2.6 brokers. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Assignee: Bruno Cadonna >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245927#comment-17245927 ] Bruno Cadonna commented on KAFKA-10772: --- The fix for KAFKA-10284 is a pure broker-side fix. So even if you use 2.6.1 you do not have the fix on your brokers. That means, [~ableegoldman] 's guess that it could be a symptom of KAFKA-10284 may still be correct. I am currently looking into this issue. If you could also provide some client and maybe also broker logs that show the bug, that would be awesome. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245885#comment-17245885 ] Bradley Peterson commented on KAFKA-10772: -- [~cadonna] no, our brokers are on 2.4.1. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245848#comment-17245848 ] Bruno Cadonna commented on KAFKA-10772: --- [~thebearmayor], do you run 2.6.1 built from [1cbc4da|https://github.com/apache/kafka/commit/1cbc4da0c9d19b25ffeb04cb2b52d827fbe38684] also broker side? > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245780#comment-17245780 ] Levani Kokhreidze commented on KAFKA-10772: --- Hi [~ableegoldman], I doubled checked the logs, there's no log statement that says "Assigned tasks to clients as" during the incident. That log statement appears only after we have restarted Kafka streams process and service started processing things successfully. Does this help? I'll provide the logs by end of the day. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245744#comment-17245744 ] Levani Kokhreidze commented on KAFKA-10772: --- Hi [~ableegoldman], Sorry, got a bit distracted. I'll see what I can find and try to add additional logs. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245587#comment-17245587 ] A. Sophie Blee-Goldman commented on KAFKA-10772: Specifically, I'm looking for the log message that starts with "Assigned tasks to clients as ..." that corresponds to the fatal rebalance. But the more logs the merrier :) > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245586#comment-17245586 ] A. Sophie Blee-Goldman commented on KAFKA-10772: Hey [~lkokhreidze] and/or [~thebearmayor] – would either of you be able to upload additional client-side logs the next time you see this? The logs that you attached earlier are pretty short, and unfortunately don't cover anything that can give us a hint as to what is going on. Probably the best place to start would be to get the logs from the client which was the group leader for the rebalance during which this exception was hit. That should show definitely whether the problem is in the assignor or the protocol itself, since the leader should log the full final assignment at INFO level > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245460#comment-17245460 ] Bill Bejeck commented on KAFKA-10772: - Hi [~vvcephei], Since this appears to be an outstanding issue with either static membership or something specific to 2.6 I'm going to say that it's not a blocker for 2.7. Additionally, it would seem that it's possible there's a workaround if it's possible to not use static-membership. I'll update the fix version accordingly. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Blocker > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245456#comment-17245456 ] John Roesler commented on KAFKA-10772: -- Hey [~bbejeck] , this might be a bit aggressive, but I don't think we've seen this before. Should it block 2.7.0? > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Blocker > Fix For: 2.7.0 > > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17244087#comment-17244087 ] Bradley Peterson commented on KAFKA-10772: -- Hi Sophie, just fyi we're seeing the same error, also with static group membership, but we are running a 2.6.1 built from [1cbc4da|https://github.com/apache/kafka/commit/1cbc4da0c9d19b25ffeb04cb2b52d827fbe38684]. Meaning we have the fix for the [KAFKA-10284]. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Major > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242025#comment-17242025 ] A. Sophie Blee-Goldman commented on KAFKA-10772: Hm, the leadership changes definitely rings a bell – maybe this is another symptom of KAFKA-10284 ? This should be fixed in 2.6.1 which is currently in the progress of being released. You could try out the RC for that release, or just build from source if that's an option. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Major > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241865#comment-17241865 ] Levani Kokhreidze commented on KAFKA-10772: --- Maybe this helps – we noticed that this problem happens mostly when there's leadership change in the cluster. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Major > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241863#comment-17241863 ] Levani Kokhreidze commented on KAFKA-10772: --- Hi [~ableegoldman], yes we are using static membership. I've attached logs for the incident.[^KAFKA-10772.log] > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Major > Attachments: KAFKA-10772.log > > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241814#comment-17241814 ] A. Sophie Blee-Goldman commented on KAFKA-10772: Hey [~lkokhreidze], thanks for the report. Definitely looks like a bug – based on the error message, presumably either in the assignment function or with static membership. Are you using static membership? Do you have any client logs from around the time of the exception? Broker logs would be useful as well, but probably only client logs can tell us if "the leader's assign function is buggy and did not return any assignment for this member" or not > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: Levani Kokhreidze >Priority: Major > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > Broker version 2.4.1, Client version 2.6.0 > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10772) java.lang.IllegalStateException: There are insufficient bytes available to read assignment from the sync-group response (actual byte size 0)
[ https://issues.apache.org/jira/browse/KAFKA-10772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17239528#comment-17239528 ] Levani Kokhreidze commented on KAFKA-10772: --- cc [~ableegoldman] [~guozhang] There have been some discussions around this exception in https://issues.apache.org/jira/browse/KAFKA-10134 but I couldn't find report around this bug. Not sure if 2.6.1 release addresses this problem, but we see this exception occurring that causes Kafka Streams to move to ERROR state. > java.lang.IllegalStateException: There are insufficient bytes available to > read assignment from the sync-group response (actual byte size 0) > > > Key: KAFKA-10772 > URL: https://issues.apache.org/jira/browse/KAFKA-10772 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Levani Kokhreidze >Priority: Major > > From time to time we encounter the following exception that results in Kafka > Streams threads dying. > {code:java} > Nov 27 00:59:53.681 streaming-app service: prod | streaming-app-2 | > stream-client [cluster1-profile-stats-pipeline-client-id] State transition > from REBALANCING to ERROR Nov 27 00:59:53.681 streaming-app service: prod | > streaming-app-2 | stream-client [cluster1-profile-stats-pipeline-client-id] > State transition from REBALANCING to ERROR Nov 27 00:59:53.682 streaming-app > service: prod | streaming-app-2 | 2020-11-27 00:59:53.681 ERROR 105 --- > [-StreamThread-1] .KafkaStreamsBasedStreamProcessingEngine : Stream > processing pipeline: [profile-stats] encountered unrecoverable exception. > Thread: [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is > completely dead. If all worker threads die, Kafka Streams will be moved to > permanent ERROR state. Nov 27 00:59:53.682 streaming-app service: prod | > streaming-app-2 | Stream processing pipeline: [profile-stats] encountered > unrecoverable exception. Thread: > [cluster1-profile-stats-pipeline-client-id-StreamThread-1] is completely > dead. If all worker threads die, Kafka Streams will be moved to permanent > ERROR state. java.lang.IllegalStateException: There are insufficient bytes > available to read assignment from the sync-group response (actual byte size > 0) , this is not expected; it is possible that the leader's assign function > is buggy and did not return any assignment for this member, or because static > member is configured and the protocol is buggy hence did not get the > assignment for this member at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:367) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:440) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:359) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:513) > at > org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1268) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:766) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:624) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)