[
https://issues.apache.org/jira/browse/KAFKA-20766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095705#comment-18095705
]
dongfang commented on KAFKA-20766:
----------------------------------
I reproduced this issue again.
The blocking seems to happen during reverse DNS lookup triggered by
{{{}InetAddress.getCanonicalHostName(){}}}.
The connection establishment may be delayed for several seconds when PTR lookup
is slow or unavailable.
Is this behavior expected?
> SASL channel authentication may block under slow/broken reverse DNS
> resolution during connection establishment
> --------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20766
> URL: https://issues.apache.org/jira/browse/KAFKA-20766
> Project: Kafka
> Issue Type: Wish
> Components: clients
> Reporter: dongfang
> Priority: Minor
>
> h2. Description
> We observed a severe connection degradation issue in Kafka clusters when
> using certain DNS resolvers (e.g. 114.114.114.114) for reverse DNS lookup
> (PTR resolution).
> The issue manifests as:
> Massive {color:#FF0000}SYN_SENT{color} backlog on Kafka broker port 9092
> {color:#FF0000}listen queue overflow{color} and {color:#FF0000}SYN
> dropped{color} increasing rapidly
> Clients (Python, JVM) continuously retrying connection
> Kafka broker threads blocked in:
> {code:java}
> InetAddress.getHostByAddr → getHostName
> SaslServerAuthenticator.createSaslServer
> {code}
> h2. Root Cause Analysis
> From {color:#0000FF}jstack{color}:
> {code:java}
> java.net.InetAddress.getHostByAddr
> java.net.InetAddress.getHostName
> org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.createSaslServer
> {code}
> This indicates that reverse DNS lookup is triggered during SASL
> authentication path, causing blocking behavior when PTR resolution is slow or
> timeout.
> h2. DNS Dependency Issue
> We confirmed:
> Using {color:#008000}8.8.8.8{color} → normal behavior
> Using {color:#FF0000}114.114.114.114{color} → PTR lookup timeout (~10s delay)
> {code:java}
> getent hosts <ip>{code}
> becomes slow or blocked
> {code:java}
> host <ip>{code}
> also times out
> h2. Impact
> Broker acceptor threads blocked
> SYN backlog overflow:
> {code:java}
> SYNs to LISTEN sockets dropped
> listen queue overflowed
> {code}
> Kafka unstable under load
> Clients see intermittent connection failures
> h2. Related JIRA
> KAFKA-3352: Reverse DNS lookup performance issue
> KAFKA-8562: SASL authenticator hostname resolution improvement (partially
> addressed, but issue still observed in newer versions including 3.x and 4.x)
> h2. Suggestion
> Even if DNS server is external (114.114.114.114), it should NOT impact
> internal cluster stability.
> PTR resolution for internal IPs should ideally:
> * fallback immediately to IP
> * or use /etc/hosts only
> * or be disabled in broker authentication flow
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
