Abhi created KAFKA-7982:
---------------------------
Summary: ConcurrentModificationException and Continuous warnings
"Attempting to send response via channel for which there is no open connection"
Key: KAFKA-7982
URL: https://issues.apache.org/jira/browse/KAFKA-7982
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 2.1.1
Reporter: Abhi
Hi,
I am getting follow warnings in server.log continuosly and due to this client
consumer is not able to consumer messages.
[2019-02-20 10:26:30,312] WARN Attempting to send response via channel for
which there is no open connection, connection id
10.218.27.45:9092-10.219.25.239:35248-6259 (kafka.network.Processor)
[2019-02-20 10:26:56,760] WARN Attempting to send response via channel for
which there is no open connection, connection id
10.218.27.45:9092-10.219.25.239:35604-6261 (kafka.network.Processor)
I also noticed that before these warnings started to appear, following
concurrent modification exception for the same IP address:
[2019-02-20 09:01:11,175] INFO Initiating logout for
kafka/u-kafkatst-kafkadev-1.sd....@unix.com
(org.apache.kafka.common.security.kerberos.KerberosLogin)
[2019-02-20 09:01:11,176] WARN [SocketServer brokerId=1] Unexpected error from
/10.219.25.239; closing connection (org.apache.kafka.common.network.Selector)
java.util.ConcurrentModificationException
at
java.base/java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:970)
at java.base/java.util.LinkedList$ListItr.next(LinkedList.java:892)
at
java.base/javax.security.auth.Subject$SecureSet$1.next(Subject.java:1096)
at
java.base/javax.security.auth.Subject$ClassSet$1.run(Subject.java:1501)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at
java.base/javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1499)
at
java.base/javax.security.auth.Subject$ClassSet.<init>(Subject.java:1472)
at
java.base/javax.security.auth.Subject.getPrivateCredentials(Subject.java:764)
at java.security.jgss/sun.security.jgss.GSSUtil$1.run(GSSUtil.java:336)
at java.security.jgss/sun.security.jgss.GSSUtil$1.run(GSSUtil.java:328)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at
java.security.jgss/sun.security.jgss.GSSUtil.searchSubject(GSSUtil.java:328)
at
java.security.jgss/sun.security.jgss.wrapper.NativeGSSFactory.getCredFromSubject(NativeGSSFactory.java:53)
at
java.security.jgss/sun.security.jgss.wrapper.NativeGSSFactory.getCredentialElement(NativeGSSFactory.java:116)
at
java.security.jgss/sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:187)
at
java.security.jgss/sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:439)
at
java.security.jgss/sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:74)
at
java.security.jgss/sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:148)
at
jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:108)
at
jdk.security.jgss/com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85)
at
java.security.sasl/javax.security.sasl.Sasl.createSaslServer(Sasl.java:537)
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.lambda$createSaslKerberosServer$12(SaslServerAuthenticator.java:212)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.createSaslKerberosServer(SaslServerAuthenticator.java:211)
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.createSaslServer(SaslServerAuthenticator.java:164)
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.handleKafkaRequest(SaslServerAuthenticator.java:450)
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.authenticate(SaslServerAuthenticator.java:248)
at
org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:132)
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:532)
at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
at kafka.network.Processor.poll(SocketServer.scala:689)
at kafka.network.Processor.run(SocketServer.scala:594)
at java.base/java.lang.Thread.run(Thread.java:834)
[2019-02-22 00:18:29,439] INFO Initiating re-login for
kafka/u-kafkatst-kafkadev-1.sd.deshaw....@unix.deshaw.com
(org.apache.kafka.common.security.kerberos.KerberosLogin)
[2019-02-22 00:18:29,440] WARN [SocketServer brokerId=1] Unexpected error from
/10.219.25.239; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.KafkaException: Principal could not be determined from
Subject, this may be a transient failure due to Kerberos re-login
at
org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.firstPrincipal(SaslClientAuthenticator.java:435)
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.createSaslKerberosServer(SaslServerAuthenticator.java:177)
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.createSaslServer(SaslServerAuthenticator.java:164)
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.handleKafkaRequest(SaslServerAuthenticator.java:450)
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.authenticate(SaslServerAuthenticator.java:248)
at
org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:132)
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:532)
at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
at kafka.network.Processor.poll(SocketServer.scala:689)
at kafka.network.Processor.run(SocketServer.scala:594)
at java.base/java.lang.Thread.run(Thread.java:834)
[2019-02-22 00:18:29,440] WARN [SocketServer brokerId=1] Unexpected error from
/10.219.25.239; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.KafkaException: Principal could not be determined from
Subject, this may be a transient failure due to Kerberos re-login
at
org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.firstPrincipal(SaslClientAuthenticator.java:435)
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.createSaslKerberosServer(SaslServerAuthenticator.java:177)
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.createSaslServer(SaslServerAuthenticator.java:164)
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.handleKafkaRequest(SaslServerAuthenticator.java:450)
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.authenticate(SaslServerAuthenticator.java:248)
at
org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:132)
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:532)
at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
at kafka.network.Processor.poll(SocketServer.scala:689)
at kafka.network.Processor.run(SocketServer.scala:594)
at java.base/java.lang.Thread.run(Thread.java:834)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
