Steve Kecskes created KAFKA-9935:
------------------------------------
Summary: Kafka not releasing member from Consumer Group
Key: KAFKA-9935
URL: https://issues.apache.org/jira/browse/KAFKA-9935
Project: Kafka
Issue Type: Bug
Affects Versions: 2.4.1
Environment: Linux
Reporter: Steve Kecskes
Hello. I am experiencing an issue where Kafka is not releasing members from a
consumer group when the member crashes. The consumer group is then stuck in
rebalancing state indefinitely.
In this consumer group, there is only 1 client. This client has the following
related settings:
{code:java}
auto.commit.interval.ms = 5000
auto.offset.reset = earliest
bootstrap.servers = [austgkafka01.hk.eclipseoptions.com:9092]
check.crcs = true
client.dns.lookup = default
client.id = TraderAutomationViewServer_workaround_stuck_rebalance_20200427-0
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = TraderAutomationViewServer_workaround_stuck_rebalance_20200427
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 10000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class
org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 16777216
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
{code}
If the client crashes (not a graceful exit from group) the group remains in the
following state indefinitely.
{code}
Warning: Consumer group 'TraderAutomationViewServer_workaround_stuck_rebalance'
is rebalancing.
GROUP TOPIC
PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST
CLIENT-ID
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 10
6984061 7839599 855538 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 8
128459531 143736443 15276912 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 12
7216495 8106030 889535 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 6
122921729 137377358 14455629 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 14
5457618 6171142 713524 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 4
125647891 140542566 14894675 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 16
6780884 7557366 776482 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 2
130906388 146481942 15575554 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 18
6763721 7611284 847563 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 16
127231640 142439019 15207379 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 14
123966927 138503193 14536266 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 12
132218162 148075484 15857322 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 10
129898059 145320287 15422228 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 19
126839746 142013652 15173906 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 1
6472562 7320125 847563 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 3
6001010 6680685 679675 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 0
127150314 142450205 15299891 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 5
5584947 6222650 637703 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 7
6394613 7116260 721647 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 9
6416544 7256574 840030 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 9
124682226 139534207 14851981 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 11
5705737 6293935 588198 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 7
127549142 142739397 15190255 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 13
6741049 7517531 776482 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 5
127251434 142368626 15117192 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 15
6669643 7412866 743223 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 3
125716835 140706603 14989768 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 17
4572135 5096785 524650 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 17
133148873 148911674 15762801 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 19
4026841 4551491 524650 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 15
123523794 138414667 14890873 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 13
126244541 141143311 14898770 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 11
124746537 139592978 14846441 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance Dispersion 0
8444970 8444970 0 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance AutomationMonitor 0
1683809 1849581 165772 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 0
5453821 6062415 608594 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance Heartbeat 0
3754819072 3820413525 65594453 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 18
125249666 140089929 14840263 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 2
4365938 4890588 524650 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics 1
128224623 143317837 15093214 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 4
5457939 6150477 692538 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 6
6703558 7480040 776482 - -
-
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 8
8173671 9127344 953673 - -
-
{code}
The member is not released. Even the process is no longer running
{code}
Warning: Consumer group 'TraderAutomationViewServer_workaround_stuck_rebalance'
is rebalancing.
GROUP CONSUMER-ID
HOST
CLIENT-ID #PARTITIONS
TraderAutomationViewServer_workaround_stuck_rebalance
TraderAutomationViewServer_workaround_stuck_rebalance-0-f4e08bf8-e340-44f1-8872-9bcad9ed207a
/10.2.16.11 TraderAutomationViewServer_workaround_stuck_rebalance-0 0
{code}
The next option here is to create a new group or restart broker.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
