dongjoon-hyun commented on PR #58283:
URL: https://github.com/apache/spark/pull/58283#issuecomment-5416812499
Thank you for the PR, and for the detailed description of the `ndots:5`
behavior.
I have a concern about the default value being a silent behavior change.
`spark.kubernetes.clusterDomain` defaults to `cluster.local`, and the driver
hostname is now always emitted as an absolute name with a trailing dot. On a
cluster configured with a non-default DNS domain (`kubelet
--cluster-domain=...`), the existing `<service>.<namespace>.svc` resolves
correctly through the pod's search list, but
`<service>.<namespace>.svc.cluster.local.` bypasses the search list entirely
and gets NXDOMAIN. Such clusters would break after upgrading, and there is no
way to restore the previous behavior because every value of the new config
produces an FQDN.
Note also that `DriverServiceFeatureStep` runs on the submission client,
which is often outside the cluster, so the correct domain cannot be
auto-detected there.
Could we make this opt-in instead? For example, `createOptional`, and keep
the current hostname when it is unset:
```scala
val driverHostname = kubernetesConf.get(KUBERNETES_CLUSTER_DOMAIN)
.map(d => s"$resolvedServiceName.${kubernetesConf.namespace}.svc.$d.")
.getOrElse(s"$resolvedServiceName.${kubernetesConf.namespace}.svc")
```
If we want to keep `cluster.local` as the default, we would at least need a
documented way to opt out and restore the legacy hostname.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
