[
https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084270#comment-16084270
]
Saisai Shao commented on SPARK-21376:
-------------------------------------
Hi [~tgraves], it is the local yarn launcher process which will launch Spark
application on yarn cluster. The problem here is that local launcher process
will always keep the initial token and not get renewed, so when application is
killed then local launcher process will try to delete the staging files, and
using this initial token will be failed in long running scenario.
> Token is not renewed in yarn client process in cluster mode
> -----------------------------------------------------------
>
> Key: SPARK-21376
> URL: https://issues.apache.org/jira/browse/SPARK-21376
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.1.1, 2.2.0
> Reporter: Yesha Vora
> Priority: Minor
>
> STR:
> * Set below config in spark-default.conf
> {code}
> spark.yarn.security.credentials.hbase.enabled true
> spark.hbase.connector.security.credentials.enabled false{code}
> * Set below config in hdfs-site.xml
> {code}
> 'dfs.namenode.delegation.token.max-lifetime':'43200000'
> 'dfs.namenode.delegation.token.renew-interval':'28800000' {code}
> * Run HDFSWordcount streaming app in yarn-cluster mode for 25 hrs.
> After 25 hours, noticing that HDFS Wordcount job is hitting
> HDFS_DELEGATION_TOKEN renewal issue.
> {code}
> 17/06/28 10:49:47 WARN Client: Exception encountered while connecting to the
> server :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 230 for hrt_qa) is expired
> 17/06/28 10:49:47 WARN Client: Failed to cleanup staging dir
> hdfs://mycluster0/user/hrt_qa/.sparkStaging/application_1498539861056_0015
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 230 for hrt_qa) is expired
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554)
> at org.apache.hadoop.ipc.Client.call(Client.java:1498){code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
