Thanks Jay. Looking forward to your PR. ________________________________ From: Jay Sen <[email protected]> Sent: Tuesday, September 15, 2020 7:29 PM To: [email protected] <[email protected]> Subject: Re: Yarn token mgmt
got it, I was able to make some changes and collect tokens in Credential to write to file and it worked fine. will create PR that should be a breaking change. Thanks On Tue, Sep 15, 2020 at 10:12 AM Lei Sun <[email protected]> wrote: > Hi Jay, > > > So yes most of YarnAppLauncher is still relying on kerberos login. We are > using similar thing as OTHER_NAMENODES but it just happen to be the case > that Azkaban is taking over the heavy-lifting part. > > > Regards, > Lei > ________________________________ > From: Jay Sen <[email protected]> > Sent: Friday, September 11, 2020 12:23 AM > To: [email protected] <[email protected]> > Subject: Re: Yarn token mgmt > > Hi Lei, yes, that was helpful. > > the more i looked into it. I see all current method from yarnAppLauncher is > limited to local hadoop cluster with kerberos only. ( as it uses > getDelegationToken for the local hdfs) > Moreover, it does not honor multiple tokens like KMS-dt or possibly others > ( even though I sae other token fetching functions, its all for the local > cluster ), but not for the remote. > > The right way of doing this can be registrering the remote namenodes > dynamically at runtime from the CopySource as it appears, probably for the > long term solution. > For now, I was able to solve it via bunch of code changes with similar > functionality of OTHEER_NAMENODES. > > Thanks for the pointers. > -Jay > > On Tue, Sep 8, 2020 at 8:59 PM Lei Sun <[email protected]> wrote: > > > Hi Jay, > > > > > > Our workflow scheduler (Azkaban) did that for Gobblin to fetch remote > > tokens in the beginning of the job and add it to UGI. > > > > Hope it helps. > > > > > > Lei > > > > > > ________________________________ > > From: Jay Sen <[email protected]> > > Sent: Tuesday, September 8, 2020 6:46 PM > > To: [email protected] <[email protected]> > > Subject: Yarn token mgmt > > > > Hi Gobblin Dev team, > > > > I see the configs and functionality around creating and renewing the > token > > off of provided keytab file, but didn't ind any functionality that > creates > > token for remote system. > > > > so question is if we run Gobblin for hadoop to hadoop job ( source = > > CopySource ), how does it manages creating and renewing token for the > > remote hadoop cluster. > > > > Thanks > > jay > > >
