pan3793 commented on issue #5181: URL: https://github.com/apache/kyuubi/issues/5181#issuecomment-1685666307
Something extended to this issue, Kyuubi implemented DSv2 based Hive connector(a.k.a. KSHC). And in #4560 > ... make Kyuubi Spark Hive Connector(KSHC) support kerberized-HMS in cluster mode w/o keytab(which is the typical use case in Kyuubi) by implementing a `HadoopDelegationTokenProvider`. There are some notable tricks 1. `spark-sql` has some [inconsistent behaviors](https://github.com/apache/spark/pull/18648/files#diff-45c9b065d76b237bcfecda83b8ee08c1ff6592d6f85acca09c0fa01472e056afR123) on HiveClient initialization, which makes inconsistent behavior when you using `spark-sql` for testing. `spark-shell` and `beeline` + Kyuubi works well. 2. we must set different `hive.metastore.token.signature` for different HMS to distinguish the delegation tokens, otherwise the latter will overwrite the former. In #4560, we use the metastore uri as the signature if `hive.metastore.token.signature` is not set explicitly. So technically, to allow Iceberg to use different kerberized-HMS, you can register an additional KSHC catalog, and make sure they use the same metastore uri and signature, thus they can share the delegation tokens. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
