[ https://issues.apache.org/jira/browse/KAFKA-9701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guozhang Wang resolved KAFKA-9701. ---------------------------------- Fix Version/s: 2.5.0 Resolution: Fixed > Consumer could catch InconsistentGroupProtocolException during rebalance > ------------------------------------------------------------------------ > > Key: KAFKA-9701 > URL: https://issues.apache.org/jira/browse/KAFKA-9701 > Project: Kafka > Issue Type: Bug > Affects Versions: 2.5.0 > Reporter: Boyang Chen > Assignee: Boyang Chen > Priority: Blocker > Fix For: 2.5.0 > > Attachments: cluster.log > > > The bug was due to an out-of-order handling of the SyncGroupRequest after the > LeaveGroupRequest. > The sequence of events are: > # The stream thread tries to rejoin the group during runOnce#poll > # The join group call was successful and group was waiting for sync group > result > # Outside the poll, task producer hits FencedException, triggering a > partition lost > # Stream thread unsubscribes and sends out an leave group, and gets the > local generation wipe out > # The sync group response was processed. Although it is legitimate, the > local protocol type becomes null in this case > # The sync group response hits the protocol type mismatch fatal exception > > [2020-03-20T*10:40:08-07:00*] > (streams-soak-trunk-eos_soak_i-01629239fa39901b4_streamslog) [2020-03-20 > 17:40:08,754] INFO > [stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1] > [Consumer > clientId=stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1-consumer, > groupId=stream-soak-test] (Re-)joining group > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [2020-03-20T*10:40:11-07:00*] > (streams-soak-trunk-eos_soak_i-01629239fa39901b4_streamslog) [2020-03-20 > 17:40:11,152] ERROR [kafka-producer-network-thread | > stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1-0_1-producer] > stream-thread > [stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1] task > [0_1] Error encountered sending record to topic network-id-repartition for > task 0_1 due to: > [2020-03-20T10:40:11-07:00] > (streams-soak-trunk-eos_soak_i-01629239fa39901b4_streamslog) > org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an > operation with an old epoch. Either there is a newer producer with the same > transactionalId, or the producer's transaction has been expired by the broker. > [2020-03-20T10:40:12-07:00] > (streams-soak-trunk-eos_soak_i-01629239fa39901b4_streamslog) [2020-03-20 > 17:40:12,048] INFO > [stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1] > stream-thread > [stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1] at > state RUNNING: partitions [logs.json.kafka-1, node-name-repartition-1, > logs.json.zookeeper-1, logs.kubernetes-1, windowed-node-counts-1, > logs.operator-1, logs.syslog-1] lost due to missed rebalance. > lost active tasks: [] > lost assigned standby tasks: [] > (org.apache.kafka.streams.processor.internals.StreamThread) > > [2020-03-20T*10:40:12-07:00*] > (streams-soak-trunk-eos_soak_i-01629239fa39901b4_streamslog) [2020-03-20 > 17:40:12,048] INFO > [stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1] > [Consumer > clientId=stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1-consumer, > groupId=stream-soak-test] Member > stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1-consumer-34c2198b-5bdd-470b-ae50-30a39873edab > sending LeaveGroup request to coordinator > ip-172-31-18-29.us-west-2.compute.internal:9092 (id: 2147482644 rack: null) > due to the consumer *unsubscribed from all topics* > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [2020-03-20T10:40:12-07:00] > (streams-soak-trunk-eos_soak_i-01629239fa39901b4_streamslog) [2020-03-20 > 17:40:12,048] INFO > [stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1] > [Consumer > clientId=stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1-consumer, > groupId=stream-soak-test] Unsubscribed all topics or patterns and assigned > partitions (org.apache.kafka.clients.consumer.KafkaConsumer) > [2020-03-20T10:40:17-07:00] > (streams-soak-trunk-eos_soak_i-01629239fa39901b4_streamslog) [2020-03-20 > 17:40:16,972] ERROR > [stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1] > [Consumer > clientId=stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1-consumer, > groupId=stream-soak-test] SyncGroup failed due to inconsistent Protocol > Name, received stream but expected null > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > [2020-03-20T10:40:17-07:00] > (streams-soak-trunk-eos_soak_i-01629239fa39901b4_streamslog) [2020-03-20 > 17:40:16,973] ERROR > [stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1] > stream-thread > [stream-soak-test-f7392d33-55d7-484f-8b72-578e22fead96-StreamThread-1] > Encountered the following exception during processing and the thread is going > to shut down: (org.apache.kafka.streams.processor.internals.StreamThread) > > ----------- Original Exception ------------- > INFO log shows that we accidentally hit an unexpected inconsistent group > protocol exception: > [2020-03-10T17:16:53-07:00] > (streams-soak-2-5-eos-broker-2-5_soak_i-00067445452c82fe8_streamslog) > [2020-03-11 *00:16:53,382*] INFO > [stream-soak-test-d3da8597-c371-450e-81d9-72aea6a26949-StreamThread-1] > stream-client [stream-soak-test-d3da8597-c371-450e-81d9-72aea6a26949] State > transition from REBALANCING to RUNNING (org.apache.kafka.streams.KafkaStreams) > > [2020-03-10T17:16:53-07:00] > (streams-soak-2-5-eos-broker-2-5_soak_i-00067445452c82fe8_streamslog) > [2020-03-11 *00:16:53,384*] WARN [kafka-producer-network-thread | > stream-soak-test-d3da8597-c371-450e-81d9-72aea6a26949-StreamThread-1-0_1-producer] > stream-thread > [stream-soak-test-d3da8597-c371-450e-81d9-72aea6a26949-StreamThread-1] task > [0_1] Error sending record to topic node-name-repartition due to Producer > attempted an operation with an old epoch. Either there is a newer producer > with the same transactionalId, or the producer's transaction has been expired > by the broker.; No more records will be sent and no more offsets will be > recorded for this task. > > > [2020-03-10T17:16:53-07:00] > (streams-soak-2-5-eos-broker-2-5_soak_i-00067445452c82fe8_streamslog) > [2020-03-11 *00:16:53,521*] INFO > [stream-soak-test-d3da8597-c371-450e-81d9-72aea6a26949-StreamThread-1] > [Consumer > clientId=stream-soak-test-d3da8597-c371-450e-81d9-72aea6a26949-StreamThread-1-consumer, > groupId=stream-soak-test] Member > stream-soak-test-d3da8597-c371-450e-81d9-72aea6a26949-StreamThread-1-consumer-d1c3c796-0bfb-4c1c-9fb4-5a807d8b53a2 > sending LeaveGroup request to coordinator > ip-172-31-20-215.us-west-2.compute.internal:9092 (id: 2147482646 rack: null) > due to the consumer unsubscribed from all topics > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) > > [2020-03-10T17:16:54-07:00] > (streams-soak-2-5-eos-broker-2-5_soak_i-00067445452c82fe8_streamslog) > [2020-03-11 *00:16:53,798*] ERROR > [stream-soak-test-d3da8597-c371-450e-81d9-72aea6a26949-StreamThread-1] > stream-thread > [stream-soak-test-d3da8597-c371-450e-81d9-72aea6a26949-StreamThread-1] > Encountered the following unexpected Kafka exception during processing, this > usually indicate Streams internal errors: > (org.apache.kafka.streams.processor.internals.StreamThread) > [2020-03-10T17:16:54-07:00] > (streams-soak-2-5-eos-broker-2-5_soak_i-00067445452c82fe8_streamslog) > org.apache.kafka.common.errors.InconsistentGroupProtocolException: The group > member's supported protocols are incompatible with those of existing members > or first group member tried to join with empty protocol type or empty > protocol list. > > Potentially needs further log to understand this. -- This message was sent by Atlassian Jira (v8.3.4#803005)