Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17335
I'm not sure if I understand your scenario correctly. In your case Spark
application is embeded into your own application, your application is still
worked after Spark is stopped. And because delegation tokens is expired
explicitly after yarn app is finished. Then your following hdfs operations
which honor delegation tokens will be failed, so you have to use tgt rather
than delegation tokens. Am I right?
I guess it related to this JIRA
(https://issues.apache.org/jira/browse/YARN-2964). It may already be fixed in
yarn side.
But with your fix, proxy user is not worked. And I think to handle your
scenario, we could deliberately remove all the tokens in current UGI after
application is finished. So that your following hdfs operations could honor tgt
to get new tokens.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
