[
https://issues.apache.org/jira/browse/GOBBLIN-1286?focusedWorklogId=500146&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-500146
]
ASF GitHub Bot logged work on GOBBLIN-1286:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 13/Oct/20 16:18
Start Date: 13/Oct/20 16:18
Worklog Time Spent: 10m
Work Description: sv2000 commented on a change in pull request #3125:
URL: https://github.com/apache/incubator-gobblin/pull/3125#discussion_r504081639
##########
File path:
gobblin-utility/src/main/java/org/apache/gobblin/util/hadoop/TokenUtils.java
##########
@@ -321,17 +321,15 @@ private static void getFsAndJtTokensImpl(final State
state, final Configuration
private static void getHdfsToken(Configuration conf, Credentials cred)
throws IOException {
FileSystem fs = FileSystem.get(conf);
LOG.info("Getting DFS token from " + fs.getUri());
- Token<?> fsToken = fs.getDelegationToken(getMRTokenRenewerInternal(new
JobConf()).toString());
- if (fsToken == null) {
- LOG.error("Failed to fetch DFS token for ");
- throw new IOException("Failed to fetch DFS token.");
+ String renewer = getMRTokenRenewerInternal(new JobConf()).toString();
+ Token<?>[] fsTokens = fs.addDelegationTokens(renewer, cred);
+ for(int i = 0; i< fsTokens.length; i++) {
Review comment:
Nit: Add whitespaces around "<".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 500146)
Time Spent: 0.5h (was: 20m)
> Support fetching multiple DFS tokens for HDFS Federation
> --------------------------------------------------------
>
> Key: GOBBLIN-1286
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1286
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: misc
> Reporter: Sandeep Devaiah
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently org.apache.gobblin.util.hadoop.TokenUtils#getHdfsToken adds the
> DistributedFileSystem's delegation token to the Credential's object. However,
> if the DFS is federated, then only one token for the primary name node is
> fetched and tokens for child namenodes are not fetched.
> Hence this method needs to be updated to use
> org.apache.hadoop.fs.FileSystem#addDelegationTokens instead of
> org.apache.hadoop.fs.FileSystem#getDelegationTokens which automatically
> fetches the delegation tokens for all child FS as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
