dongjoon-hyun commented on PR #58283: URL: https://github.com/apache/spark/pull/58283#issuecomment-5416834245
One more note on the motivation section, which no longer matches `master`. Since Spark 4.3 (SPARK-57351), `spark.kubernetes.executor.useDriverPodIP` is enabled by default, so executor pods connect to the driver RPC endpoint via the driver pod IP rather than the driver's K8s Service DNS name: https://github.com/apache/spark/blob/841f3a0332d/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L56-L61 So "each executor startup generates these spurious failed lookups" is not accurate for the default configuration on `master` anymore. What still resolves `spark.driver.host` is mainly the driver's advertised BlockManager address that executors use for block transfers: https://github.com/apache/spark/blob/841f3a0332d/core/src/main/scala/org/apache/spark/SparkEnv.scala#L718-L720 and those lookups are additionally subject to the JVM DNS cache. Could you update the description to describe the paths that still resolve the name on `master`, and share the measured effect on a current build (e.g. CoreDNS NXDOMAIN counts before/after)? Also, the test section mentions patching a jar manually. Which Spark version was that built from? If it predates 4.3, the observed NXDOMAIN volume would be from the driver service path that is no longer the default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
