[ https://issues.apache.org/jira/browse/YARN-2426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Varun Vasudev resolved YARN-2426. --------------------------------- Resolution: Fixed Fix Version/s: 2.6.0 Fixed with HDFS-6904 exposing an API to allow clients to set the service. > ResourceManger is not able renew WebHDFS token when application submitted by > Yarn WebService > -------------------------------------------------------------------------------------------- > > Key: YARN-2426 > URL: https://issues.apache.org/jira/browse/YARN-2426 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager, webapp > Affects Versions: 2.6.0 > Environment: Hadoop Keberos (Secure) cluster with > LinuxContainerExcutor is enabled > With SPNEGO on for Yarn new RM web services for application submission > So during application submission xml/json structure was pass webhdfs token > Reporter: Karam Singh > Assignee: Varun Vasudev > Fix For: 2.6.0 > > > Encountered this issue during using new YARN's RM WS for application > submission, on single node cluster while submitting Distributed Shell > application using RM WS(webservice). > For this we need pass custom script and AppMaster jar along with webhdfs > token. > Application was failing with ResouceManager was failing to renew token for > user (appOwner). So RM was Rejecting application with following exception > trace in RM log: > {code} > 2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer > (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to > add the application to the delegation token renewer. > java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation, > Service: <NNHOST>:<FSPORT>, Ident: (WEBHDFS delegation token 2222 for hrt_qa) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200, > op=RENEWDELEGATIONTOKEN, message=null > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318) > at > org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73) > at org.apache.hadoop.security.token.Token.renew(Token.java:377) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392) > ... 6 more > Caused by: java.io.IOException: The error stream is null. > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:329) > ... 24 more > 2014-08-19 03:12:54,735 DEBUG event.AsyncDispatcher > (AsyncDispatcher.java:dispatch(164)) - Dispatching the event > org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppRejectedEvent.EventType: > APP_REJECTED > {code} > From exception trace it is clear that RM is try contact to Namenode on FSPort > instead of Http port and failing to renew token > Looks like it is because WebHDFS token Namenodes IP and FSPort in delegation > token instead of http. Causing RM to contact WebHDFS on FSPort and failing to > renew token -- This message was sent by Atlassian JIRA (v6.3.4#6332)