looks better! no further concerns
Cheers,
Matyas
On Mon, Nov 7, 2022 at 9:21 AM Gabor Somogyi <gabor.g.somo...@gmail.com>
wrote:
> Oh gosh, copied wrong config keys so fixed my last mail with green.
>
> On Mon, Nov 7, 2022 at 6:07 PM Gabor Somogyi <gabor.g.somo...@gmail.com>
> wrote:
>
> > Hi Matyas,
> >
> > In the meantime I was thinking about the per provider re-obtain feature
> > and here are my thoughts related that:
> > * I think it's a good feature in general but as mentioned I would add it
> > in a separate FLIP
> > * In case of Hadoop providers it just wouldn't work (HBase doesn't have
> > end timestamp so actually HDFS is triggering the re-obtain) but in all
> > non-hadoop providers it's a good idea
> > * Adding "security.delegation.tokens.renewal.retry.backoff" and
> > "security.delegation.tokens.renewal.time-ratio" is needed but as you
> > mentioned fallback to kerberos configs just doesn't make sense
> > * In a later FLIP we can add per provider
> "security.delegation.token.provider.{providerName}.renewal.retry.backoff"
> > and/or
> "security.delegation.token.provider.{providerName}.renewal.time-ratio"
> > for non-hadoop providers
> > * This is an additional feature which justifies to separate Hadoop and
> > non-hadoop providers on the API level
> >
> > Waiting on your opinion.
> >
> > G
> >
> >
> > On Mon, Nov 7, 2022 at 4:17 PM Gabor Somogyi <gabor.g.somo...@gmail.com>
> > wrote:
> >
> >> Hi Matyas,
> >>
> >> Thanks for your comments, answered inline.
> >>
> >> G
> >>
> >>
> >> On Mon, Nov 7, 2022 at 2:58 PM Őrhidi Mátyás <matyas.orh...@gmail.com>
> >> wrote:
> >>
> >>> Hi Gabor,
> >>>
> >>> Thanks for driving this effort! A few thoughts on the topic:
> >>> - Could you please add a few examples of the delegation token providers
> >>> we
> >>> expected to be added in the near future? Ideally these providers could
> be
> >>> configured independently from each other. However the configuration
> >>> defaults mentioned in the FLIP are derived from hadoop configuration. I
> >>> don't see the point here.
> >>>
> >> A clear plan is to add S3 now and Kafka possibly later on.
> >>
> >> S3 looks straightforward but that doesn't fit into the existing
> framework.
> >> On Kafka side I've added the Kafka provider to Spark so I can imagine
> >> similar solution w/ minor differences.
> >> Please see the Spark solution:
> >>
> https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#security
> >> Here the minor planned difference is that Spark handles Kafka token
> >> inside UGI which is not nice but works.
> >> I've just seen similar generalization effort on the Spark side too so
> the
> >> Kafka part may or may not change there.
> >>
> >> Not sure what you mean configs are derived from Hadoop.
> >>
> >> What I can think of is that
> >> "security.delegation.tokens.renewal.retry.backoff" is defaulting to
> >> "security.kerberos.tokens.renewal.retry.backoff".
> >> This is basically happening for backward compatibility purposes.
> >>
> >> The other thing what I can think of is that you miss the independent
> >> provider token obtain functionality.
> >> When I mean independent configuration I mean each provider has it's own
> >> set of keys which doesn't collide
> >> but the voting system when token obtain must happen remains and not
> >> touched in this FLIP.
> >> Under voting system I mean each provider may send back its end timestamp
> >> and the lowest wins
> >> (at that timestamp all tokens are going to be re-obtained).
> >> If that's what you mean we can think about solutions but it has nothing
> >> to do with framework generalization.
> >>
> >>
> >>> - Are we planning to support such scenarios where we need to read/write
> >>> from different authentication realms from the same application. Two
> >>> Hadoop
> >>> clusters, Kafka clusters etc? This would need an authentication
> provider
> >>> per source/sink.
> >>>
> >>>
> >> It doesn't need 2 different provider per source/sink to do Kafka to
> >> Kafka. Such cases cross-realm trust can be configured w/ principal
> mapping.
> >> Please see the details here:
> >> https://gist.github.com/gaborgsomogyi/c636f352ccec7730ff41ac1d524cb87d
> >> Even if the gist was originally created for Spark I tend to do the same
> >> here in the future.
> >>
> >> Just to make an extract here Kafka principal mapping looks like this:
> >> sasl.kerberos.principal.to.local.rules=RULE:[1:$1@$0](.*@
> >> DT2HOST.COMPANY.COM)s/@.*//,DEFAULT
> >>
> >> Providing 2 users in Hadoop world is just impossible because UGI is
> >> basically a singleton.
> >> Of course everything can be hacked around (for ex. changing current user
> >> on-the-fly) but that would be such a
> >> headache what I think we must avoid. It would end-up in synchronization
> >> and performance hell.
> >> I've made some experiments in my Spark era and this would be the same
> >> here :)
> >>
> >>
> >>> Thanks,
> >>> Matyas
> >>>
> >>>
> >>>
> >>> On Mon, Nov 7, 2022 at 5:10 AM Gabor Somogyi <
> gabor.g.somo...@gmail.com>
> >>> wrote:
> >>>
> >>> > Hi team,
> >>> >
> >>> > Delegation token framework is going to be finished soon (added in
> >>> FLIP-211
> >>> > <
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-211%3A+Kerberos+delegation+token+framework?src=contextnavpagetreemode
> >>> > >
> >>> > ).
> >>> > Previously there were concerns that the current implementation is
> >>> bound to
> >>> > Hadoop and Kerberos authentication. This is fair concern and as a
> >>> result
> >>> > we've created a proposal to generalize the delegation token framework
> >>> > (practically making it authentication agnostic).
> >>> >
> >>> > This can open the path to add further non-hadoop and non-Kerberos
> based
> >>> > providers like S3 or many others.
> >>> >
> >>> > One can find the FLIP in:
> >>> > - Wiki:
> >>> >
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-272%3A+Generalized+delegation+token+support
> >>> > - document:
> >>> >
> >>> >
> >>>
> https://docs.google.com/document/d/12tFdx1AZVuW9BjwBht_pMNELgrqro8Z5-hzWeaRY4pc/edit?usp=sharing
> >>> >
> >>> > I would like to start a discussion to make the framework better.
> >>> >
> >>> > BR,
> >>> > G
> >>> >
> >>>
> >>
>
