Hi Matyas, Thanks for your comments, answered inline.
G On Mon, Nov 7, 2022 at 2:58 PM Őrhidi Mátyás <matyas.orh...@gmail.com> wrote: > Hi Gabor, > > Thanks for driving this effort! A few thoughts on the topic: > - Could you please add a few examples of the delegation token providers we > expected to be added in the near future? Ideally these providers could be > configured independently from each other. However the configuration > defaults mentioned in the FLIP are derived from hadoop configuration. I > don't see the point here. > A clear plan is to add S3 now and Kafka possibly later on. S3 looks straightforward but that doesn't fit into the existing framework. On Kafka side I've added the Kafka provider to Spark so I can imagine similar solution w/ minor differences. Please see the Spark solution: https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#security Here the minor planned difference is that Spark handles Kafka token inside UGI which is not nice but works. I've just seen similar generalization effort on the Spark side too so the Kafka part may or may not change there. Not sure what you mean configs are derived from Hadoop. What I can think of is that "security.delegation.tokens.renewal.retry.backoff" is defaulting to "security.kerberos.tokens.renewal.retry.backoff". This is basically happening for backward compatibility purposes. The other thing what I can think of is that you miss the independent provider token obtain functionality. When I mean independent configuration I mean each provider has it's own set of keys which doesn't collide but the voting system when token obtain must happen remains and not touched in this FLIP. Under voting system I mean each provider may send back its end timestamp and the lowest wins (at that timestamp all tokens are going to be re-obtained). If that's what you mean we can think about solutions but it has nothing to do with framework generalization. > - Are we planning to support such scenarios where we need to read/write > from different authentication realms from the same application. Two Hadoop > clusters, Kafka clusters etc? This would need an authentication provider > per source/sink. > > It doesn't need 2 different provider per source/sink to do Kafka to Kafka. Such cases cross-realm trust can be configured w/ principal mapping. Please see the details here: https://gist.github.com/gaborgsomogyi/c636f352ccec7730ff41ac1d524cb87d Even if the gist was originally created for Spark I tend to do the same here in the future. Just to make an extract here Kafka principal mapping looks like this: sasl.kerberos.principal.to.local.rules=RULE:[1:$1@$0](.*@DT2HOST.COMPANY.COM )s/@.*//,DEFAULT Providing 2 users in Hadoop world is just impossible because UGI is basically a singleton. Of course everything can be hacked around (for ex. changing current user on-the-fly) but that would be such a headache what I think we must avoid. It would end-up in synchronization and performance hell. I've made some experiments in my Spark era and this would be the same here :) > Thanks, > Matyas > > > > On Mon, Nov 7, 2022 at 5:10 AM Gabor Somogyi <gabor.g.somo...@gmail.com> > wrote: > > > Hi team, > > > > Delegation token framework is going to be finished soon (added in > FLIP-211 > > < > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-211%3A+Kerberos+delegation+token+framework?src=contextnavpagetreemode > > > > > ). > > Previously there were concerns that the current implementation is bound > to > > Hadoop and Kerberos authentication. This is fair concern and as a result > > we've created a proposal to generalize the delegation token framework > > (practically making it authentication agnostic). > > > > This can open the path to add further non-hadoop and non-Kerberos based > > providers like S3 or many others. > > > > One can find the FLIP in: > > - Wiki: > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-272%3A+Generalized+delegation+token+support > > - document: > > > > > https://docs.google.com/document/d/12tFdx1AZVuW9BjwBht_pMNELgrqro8Z5-hzWeaRY4pc/edit?usp=sharing > > > > I would like to start a discussion to make the framework better. > > > > BR, > > G > > >
