[
https://issues.apache.org/jira/browse/SPARK-37787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17468329#comment-17468329
]
huangzhir edited comment on SPARK-37787 at 1/4/22, 2:54 AM:
------------------------------------------------------------
This is because
HadoopFSDelegationTokenProvider.obtaindelegationtokens returns None when an
exception is encountered. HadoopDelegationTokenProvider that cannot be renewed
also returns None. If the obtaindelegationtokens of all
HadoopDelegationTokenProvider return None,HadoopDelegationTokenManager In
obtaindelegationtokens, call foldleft (long. Maxvalue) (math. Min) to return
long.MaxValue. Therefore, the log shows "scheduling renewal in 1921535501304.2
H"
At this time, HadoopFSDelegationToken cannot be renewed in time, resulting in
throw "hdfs_delete_token not found in cache" Exception
was (Author: JIRAUSER282843):
This is because
HadoopFSDelegationTokenProvider.obtaindelegationtokens returns None when an
exception is encountered. HadoopDelegationTokenProvider that cannot be renewed
also returns None. If the obtaindelegationtokens of all
HadoopDelegationTokenProvider return None,HadoopDelegationTokenManager In
obtaindelegationtokens, call foldleft (long. Maxvalue) (math. Min) to return
long.MaxValue. Therefore, the log shows "scheduling renewal in 1921535501304.2
H"
> Long running Spark Job(Spark ThriftServer) throw HDFS_DELEGATE_TOKEN not
> found in cache Exception
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-37787
> URL: https://issues.apache.org/jira/browse/SPARK-37787
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.0.0, 3.1.0, 3.2.0
> Environment: spark3 thrift server
>
> spark-default.conf
> spark.hadoop.fs.hdfs.impl.disable.cache=true
>
> Reporter: huangzhir
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> *HDFS_DELEGATE_TOKEN not found in cache exception* occurs when accessing
> spark thriftserver service. The specific exception is as follows:
> [Exception Log |
> https://raw.githubusercontent.com/huangzhir/Temp/main/image-3.png]
> !https://raw.githubusercontent.com/huangzhir/Temp/main/image-3.png!
> the HadoopDelegationTokenManager thow Exception when renewal
> DelegationToken,as follows:
>
> We are also find HadoopDelegationTokenManager log as follows:
> INFO [Credential Renewal Thread]
> org.apache.spark.deploy.security.HadoopDelegationTokenManager logInfo -
> *Scheduling renewal in 1921535501304.2 h.*
> [hdfs Exceptin log in HadoopDelegationTokenManager
> |https://raw.githubusercontent.com/huangzhir/Temp/main/spark%20thriftserver%20Exceptin.png]
> !https://raw.githubusercontent.com/huangzhir/Temp/main/spark%20thriftserver%20Exceptin.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
