cxzl25 commented on PR #43579: URL: https://github.com/apache/spark/pull/43579#issuecomment-2112662447
I thought about it, and `spark.yarn.user.classpath.first` doesn't quite have the same effect as `spark.driver.userClassPathFirst` or `spark.executor.userClassPathFirst`. `spark.yarn.user.classpath.first` is similar to `mapreduce.job.user.classpath.first`, declaring that the user's jar is placed at the front of the classpath, so that the YARN startup container will load it first, and it is the same classloader as the jars of spark. And `spark.{driver,executor}.userClassPathFirst` does not modify the classpath, but uses `ChildFirstURLClassLoader` to load the jar, which is not the same classloader as the spark jars, and the behavior of the two is different. So the modification of this PR may be invalid. Not sure why this parameter is deprecated. https://github.com/apache/spark/blob/d0385c4a99c172fa3e1ba2d72a65c8632b5c72a9/core/src/main/scala/org/apache/spark/SparkConf.scala#L605-L606 https://github.com/apache/spark/blob/d0385c4a99c172fa3e1ba2d72a65c8632b5c72a9/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1527-L1537 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org