Ramachandran Krishnan created RANGER-5744:
---------------------------------------------
Summary: Harden plugins-docker-build CI against flaky Ozone
datanode Kerberos/KDC startup race
Key: RANGER-5744
URL: https://issues.apache.org/jira/browse/RANGER-5744
Project: Ranger
Issue Type: Improvement
Components: Ranger
Reporter: Ramachandran Krishnan
Assignee: Ramachandran Krishnan
h3. Summary
The {{plugins-docker-build}} GitHub Actions job intermittently fails when
{{ozone-datanode}} exits during Kerberos login to the Ranger KDC. Re-running
the failed job usually passes. This is an infrastructure timing issue in the
Docker Compose CI stack, not a regression in Ranger plugin code.
Example failure: [GitHub Actions run
31185432016|https://github.com/apache/ranger/actions/runs/31185432016/job/93444713551?pr=1142]
(PR #1142).
h3. Symptoms
java.lang.RuntimeException: Can't start the HDDS datanode plugin
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login:
for principal: dn/[email protected]
Caused by: java.net.PortUnreachableException
{{ozone-datanode}} fails within ~1 second of start; other containers are
unaffected. A job re-run typically succeeds.
h3. Root cause
Three conditions combine to create a race:
# Weak startup ordering — Ozone datanode/scm use {{{}depends_on: ranger-kdc:
service_started{}}}, so they start when the KDC container exists, not when
Kerberos is ready. {{ranger-zk}} already uses {{service_healthy}} for KDC.
# Weak KDC healthcheck — {{Dockerfile.ranger-kdc}} probes TCP port 88 with
{{{}nc{}}}, but Java Kerberos clients try UDP first by default.
# No Ozone startup wait — Unlike Hadoop ({{{}wait_for_keytab.sh{}}}) and OM
({{{}ranger-ozone-setup.sh{}}}), datanode/scm launch {{ozone datanode}} /
{{ozone scm}} immediately with no keytab or KDC readiness wait.
h3. Impact
* Intermittent {{plugins-docker-build}} CI failures on unrelated PRs
* False-negative signal; developers must re-run jobs manually
* Noise in PR reviews (e.g. RANGER-5736, PR #1142)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
