Hi Lei, yes, that was helpful. the more i looked into it. I see all current method from yarnAppLauncher is limited to local hadoop cluster with kerberos only. ( as it uses getDelegationToken for the local hdfs) Moreover, it does not honor multiple tokens like KMS-dt or possibly others ( even though I sae other token fetching functions, its all for the local cluster ), but not for the remote.
The right way of doing this can be registrering the remote namenodes dynamically at runtime from the CopySource as it appears, probably for the long term solution. For now, I was able to solve it via bunch of code changes with similar functionality of OTHEER_NAMENODES. Thanks for the pointers. -Jay On Tue, Sep 8, 2020 at 8:59 PM Lei Sun <[email protected]> wrote: > Hi Jay, > > > Our workflow scheduler (Azkaban) did that for Gobblin to fetch remote > tokens in the beginning of the job and add it to UGI. > > Hope it helps. > > > Lei > > > ________________________________ > From: Jay Sen <[email protected]> > Sent: Tuesday, September 8, 2020 6:46 PM > To: [email protected] <[email protected]> > Subject: Yarn token mgmt > > Hi Gobblin Dev team, > > I see the configs and functionality around creating and renewing the token > off of provided keytab file, but didn't ind any functionality that creates > token for remote system. > > so question is if we run Gobblin for hadoop to hadoop job ( source = > CopySource ), how does it manages creating and renewing token for the > remote hadoop cluster. > > Thanks > jay >
