This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e584ed4ad96 [SPARK-44581][YARN] Fix the bug that ShutdownHookManager
gets wrong UGI from SecurityManager of ApplicationMaster
e584ed4ad96 is described below
commit e584ed4ad96a0f0573455511d7be0e9b2afbeb96
Author: 余良 <yul...@chinaunicom.cn>
AuthorDate: Wed Aug 9 13:46:57 2023 +0800
[SPARK-44581][YARN] Fix the bug that ShutdownHookManager gets wrong UGI
from SecurityManager of ApplicationMaster
### What changes were proposed in this pull request?
I make the SecurityManager instance a lazy value
### Why are the changes needed?
fix the bug in issue
[SPARK-44581](https://issues.apache.org/jira/browse/SPARK-44581)
**Bug:**
In spark3.2 it throws the
org.apache.hadoop.security.AccessControlException, but in spark2.4 this hook
does not throw exception.
I rebuild the hadoop-client-api.jar, and add some debug log before the
hadoop shutdown hook is created, and rebuild the spark-yarn.jar to add some
debug log when creating the spark shutdown hook manager, here is the screenshot
of the log:
We can see from the screenshot, the ShutdownHookManager is initialized
before the ApplicationManager create a new ugi.
**Reason**
The main cause is that ShutdownHook thread is created before we create the
ugi in ApplicationMaster.
When we set the config key "hadoop.security.credential.provider.path", the
ApplicationMaster will try to get a filesystem when generating SSLOptions, and
when initialize the filesystem during which it will generate a new thread whose
ugi is inherited from the current process (yarn).
After this, it will generate a new ugi (SPARK_USER) in ApplicationMaster
and execute the doAs() function.
Here is the chain of the call:
ApplicationMaster.(ApplicationMaster.scala:83) ->
org.apache.spark.SecurityManager.(SecurityManager.scala:98) ->
org.apache.spark.SSLOptions$.parse(SSLOptions.scala:188) ->
org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2353) ->
org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2434)
->
org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:82)
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
I didn't add new UnitTest for this, but I rebuild the package, and runs a
program in my cluster, and turns out that the user when I delete the staging
file turns to be the same with the SPARK_USER.
Closes #42405 from liangyu-1/SPARK-44581.
Authored-by: 余良 <yul...@chinaunicom.cn>
Signed-off-by: Kent Yao <y...@apache.org>
---
.../src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
index 0149a3f6217..4fa7b66c9e5 100644
---
a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
+++
b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
@@ -79,7 +79,7 @@ private[spark] class ApplicationMaster(
private val isClusterMode = args.userClass != null
- private val securityMgr = new SecurityManager(sparkConf)
+ private lazy val securityMgr = new SecurityManager(sparkConf)
private var metricsSystem: Option[MetricsSystem] = None
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
