[
https://issues.apache.org/jira/browse/SPARK-37787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17468329#comment-17468329
]
huangzhir commented on SPARK-37787:
-----------------------------------
This is because
HadoopFSDelegationTokenProvider.obtaindelegationtokens returns None when an
exception is encountered. HadoopDelegationTokenProvider that cannot be renewed
also returns None. If the obtaindelegationtokens of all
HadoopDelegationTokenProvider return None,HadoopDelegationTokenManager In
obtaindelegationtokens, call foldleft (long. Maxvalue) (math. Min) to return
long.MaxValue. Therefore, the log shows "scheduling renewal in 1921535501304.2
H"
> Long running Spark Job(Spark ThriftServer) throw HDFS_DELEGATE_TOKEN not
> found in cache Exception
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-37787
> URL: https://issues.apache.org/jira/browse/SPARK-37787
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.0.0, 3.1.0, 3.2.0
> Environment: spark3 thrift server
>
> spark-default.conf
> spark.hadoop.fs.hdfs.impl.disable.cache=true
>
> Reporter: huangzhir
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> *HDFS_DELEGATE_TOKEN not found in cache exception* occurs when accessing
> spark thriftserver service. The specific exception is as follows:
> [Exception Log |
> https://raw.githubusercontent.com/huangzhir/Temp/main/image-3.png]
> !https://raw.githubusercontent.com/huangzhir/Temp/main/image-3.png!
> the HadoopDelegationTokenManager thow Exception when renewal
> DelegationToken,as follows:
>
> We are also find HadoopDelegationTokenManager log as follows:
> INFO [Credential Renewal Thread]
> org.apache.spark.deploy.security.HadoopDelegationTokenManager logInfo -
> *Scheduling renewal in 1921535501304.2 h.*
> [hdfs Exceptin log in HadoopDelegationTokenManager
> |https://raw.githubusercontent.com/huangzhir/Temp/main/spark%20thriftserver%20Exceptin.png]
> !https://raw.githubusercontent.com/huangzhir/Temp/main/spark%20thriftserver%20Exceptin.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
