Thanks Jie for driving this discussion. I also prefer a pluggable delegation token provider. And I think users can use configurations to specify the hive conf path, similar to how users specify a hive catalog.
On Wed, Jan 13, 2021 at 4:51 PM Till Rohrmann <trohrm...@apache.org> wrote: > Hi Jie Wang, > > thanks for starting this discussion. To me the SPI approach sounds better > because it is not as brittle as using reflection. Concerning the > configuration, we could think about introducing some Hive specific > configuration options which allow us to specify these paths. How are other > projects which integrate with Hive are solving this problem? > > Cheers, > Till > > On Tue, Jan 12, 2021 at 4:13 PM 王 杰 <jackwan...@outlook.com> wrote: > >> Hi everyone, >> >> Currently, Hive delegation token is not obtained when Flink submits the >> application in Yarn mode using kinit way. The ticket is >> https://issues.apache.org/jira/browse/FLINK-20714. I'd like to start a >> discussion about how to support this feature. >> >> Maybe we have two options: >> 1. Using a reflection way to construct a Hive client to obtain the token, >> just same as the org.apache.flink.yarn.Utils.obtainTokenForHBase >> implementation. >> 2. Introduce a pluggable delegation provider via SPI. Delegation provider >> could be placed in connector related code, so reflection is not needed and >> is more extendable. >> >> >> >> Both options have to handle how to specify the HiveConf to use. In Hive >> connector, user could specify both hiveConfDir and hadoopConfDir when >> creating HiveCatalog. The hadoopConfDir may not the same as the Hadoop >> configuration in HadoopModule. >> >> Looking forward to your suggestions. >> >> -- >> Best regards! >> Jie Wang >> >> -- Best regards! Rui Li
