cxzl25 commented on PR #43579:
URL: https://github.com/apache/spark/pull/43579#issuecomment-2112662447
I thought about it, and `spark.yarn.user.classpath.first` doesn't quite have
the same effect as `spark.driver.userClassPathFirst` or
`spark.executor.userClassPathFirst`.
`spark.yarn.user.classpath.first` is similar to
`mapreduce.job.user.classpath.first`, declaring that the user's jar is placed
at the front of the classpath, so that the YARN startup container will load it
first, and it is the same classloader as the jars of spark.
And `spark.{driver,executor}.userClassPathFirst` does not modify the
classpath, but uses `ChildFirstURLClassLoader` to load the jar, which is not
the same classloader as the spark jars, and the behavior of the two is
different.
So the modification of this PR may be invalid. Not sure why this parameter
is deprecated.
https://github.com/apache/spark/blob/d0385c4a99c172fa3e1ba2d72a65c8632b5c72a9/core/src/main/scala/org/apache/spark/SparkConf.scala#L605-L606
https://github.com/apache/spark/blob/d0385c4a99c172fa3e1ba2d72a65c8632b5c72a9/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1527-L1537
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
