dongfang created KAFKA-20766:
--------------------------------
Summary: SASL channel authentication may block under slow/broken
reverse DNS resolution during connection establishment
Key: KAFKA-20766
URL: https://issues.apache.org/jira/browse/KAFKA-20766
Project: Kafka
Issue Type: Wish
Components: clients
Reporter: dongfang
h2. Description
We observed a severe connection degradation issue in Kafka clusters when using
certain DNS resolvers (e.g. 114.114.114.114) for reverse DNS lookup (PTR
resolution).
The issue manifests as:
Massive {color:#FF0000}SYN_SENT{color} backlog on Kafka broker port 9092
{color:#FF0000}listen queue overflow{color} and {color:#FF0000}SYN
dropped{color} increasing rapidly
Clients (Python, JVM) continuously retrying connection
Kafka broker threads blocked in:
{code:java}
InetAddress.getHostByAddr → getHostName
SaslServerAuthenticator.createSaslServer
{code}
h2. Root Cause Analysis
>From {color:#0000FF}jstack{color}:
{code:java}
java.net.InetAddress.getHostByAddr
java.net.InetAddress.getHostName
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.createSaslServer
{code}
This indicates that reverse DNS lookup is triggered during SASL authentication
path, causing blocking behavior when PTR resolution is slow or timeout.
h2. DNS Dependency Issue
We confirmed:
Using {color:#008000}8.8.8.8{color} → normal behavior
Using {color:#FF0000}114.114.114.114{color} → PTR lookup timeout (~10s delay)
{code:java}
getent hosts <ip>{code}
becomes slow or blocked
{code:java}
host <ip>{code}
also times out
h2. Impact
Broker acceptor threads blocked
SYN backlog overflow:
{code:java}
SYNs to LISTEN sockets dropped
listen queue overflowed
{code}
Kafka unstable under load
Clients see intermittent connection failures
h2. Related JIRA
KAFKA-3352: Reverse DNS lookup performance issue
KAFKA-8562: SASL authenticator hostname resolution improvement (partially
addressed, but issue still observed in newer versions including 3.x and 4.x)
h2. Suggestion
Even if DNS server is external (114.114.114.114), it should NOT impact internal
cluster stability.
PTR resolution for internal IPs should ideally:
* fallback immediately to IP
* or use /etc/hosts only
* or be disabled in broker authentication flow
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
