[
https://issues.apache.org/jira/browse/FALCON-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500243#comment-14500243
]
Mahak commented on FALCON-505:
------------------------------
For replication via hcat these hardcoded values can be replaced with variable
like falconSourceNamenode and falconTargetNamenode but unfortunately no such
variables are available for basic replication. If only these variables are made
available in the latter case, the issue can be resolved.
> Replication Job throws GSSException on secure cluster
> -------------------------------------------------------
>
> Key: FALCON-505
> URL: https://issues.apache.org/jira/browse/FALCON-505
> Project: Falcon
> Issue Type: Bug
> Components: replication
> Affects Versions: 0.5
> Environment: Hadoop2/YARN (both source and target clusters)
> Security enabled
> Reporter: Venkat R
>
> Replication job launched on target cluster by oozie a workflow throws
> GSSException when it tries to access the source cluster HDFS using webhdfs
> (as well as hftp).
> Both the source and target cluster oozie instances have the oozie-site.xml
> pointing to all the hadoop cluster configs they access (See first comment of
> JIRA: https://issues.apache.org/jira/browse/FALCON-389)
> It seems the Target cluster oozie coordinator instance was able to access the
> source clusters HDFS, but from the job running in the clutser node.
> But, it works if I add the following property to the
> oozie/conf/hadoop-conf-cluster-1/mapred-site.xml:
> <property>
> <name>mapreduce.job.hdfs-servers</name>
>
> <value>webhdfs://grid1nn01.grid.example.com,webhdfs://gird2nn01.grid.example.com</value>
> </property>
> this enabled grid1 to do webhdfs calls to grid2 and vice-versa. In the
> absence, it throws authentication errors.
> It seems Oozie needs to get tokens for both the clusters before it can kick
> off the Falcon job that does the distcp.
> It may be possible to add this property to the generated Oozie bundle by
> Falcon.
> Exception stacktrace:
> Failing Oozie Launcher, Main class
> [org.apache.falcon.latedata.LateDataHandler], main() threw exception,
> Authentication failed,
> url=http://gridnn01.grid.example.com:50070/webhdfs/v1/?op=GETDELEGATIONTOKEN&user.name=veramach
> java.io.IOException: Authentication failed,
> url=http://gridnn01.grid.example.com:50070/webhdfs/v1/?op=GETDELEGATIONTOKEN&user.name=veramach
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.init(WebHdfsFileSystem.java:490)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:531)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:953)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:143)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:227)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:381)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:402)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathRunner.getUrl(WebHdfsFileSystem.java:652)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.init(WebHdfsFileSystem.java:485)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:531)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:678)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:689)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:238)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1624)
> at org.apache.falcon.latedata.LateDataHandler.usage(LateDataHandler.java:269)
> at
> org.apache.falcon.latedata.LateDataHandler.getFileSystemUsageMetric(LateDataHandler.java:252)
> at
> org.apache.falcon.latedata.LateDataHandler.computeStorageMetric(LateDataHandler.java:224)
> at
> org.apache.falcon.latedata.LateDataHandler.computeMetrics(LateDataHandler.java:170)
> at org.apache.falcon.latedata.LateDataHandler.run(LateDataHandler.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.falcon.latedata.LateDataHandler.main(LateDataHandler.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:232)
> at
> org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:164)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.openHttpUrlConnection(WebHdfsFileSystem.java:475)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$200(WebHdfsFileSystem.java:431)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:457)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:454)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getHttpUrlConnection(WebHdfsFileSystem.java:453)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.init(WebHdfsFileSystem.java:487)
> ... 36 more
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
> ... 48 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
