Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/22704#discussion_r226127311 --- Diff: core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala --- @@ -49,8 +49,11 @@ private[deploy] class HadoopFSDelegationTokenProvider(fileSystems: Configuration val fetchCreds = fetchDelegationTokens(getTokenRenewer(hadoopConf), fsToGetTokens, creds) // Get the token renewal interval if it is not set. It will only be called once. - if (tokenRenewalInterval == null) { - tokenRenewalInterval = getTokenRenewalInterval(hadoopConf, sparkConf, fsToGetTokens) + // If running a Kerberos job on Kubernetes, you may specify that you wish to not + // obtain the tokenRenewal interval, as the renewal service may be external. --- End diff -- > And BTW, I know what you mean when you mention an external renewal service. But again, that does not exist, and until it does, you should not do things that assume its existence. Right, so in Spark there is no token renewal service ATM, but the existence of an external token service that places tokens into Secrets may exist within a company organization, no? Whether they leverage one, provided by Spark or not. So I personally thought that such a comment is sensible. But I'll remove it based on your reasoning. > If you're in this code, there are two options: Ah, very good point, checking for the presence of a keytab / principal as a flag, given that > It has always been, and always will be, the way to tell Spark that you want Spark to renew tokens itself makes sense > The current k8s backend is broken in that regard. The design was specifically for the popular use-case in which for cluster-mode we would not send keytabs around and instead read the DT from a secret. So true, it does break the traditional design because we are using a keytab and not enabling renewal. Contractually with your work, next steps would be to parallel the other resource-managers in allowing for the option to use the renewal code if the keytab and principal is already in the Driver. Just for interest, has there been any cases where the Driver is over-worked in running this renewal service and managing the DTs for many executors? In essence, could there be any convincing use-case to have the Driver use the keytab for login, but not want to do its own renewal as it might be the case that it can't handle the load?
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org