Hi,
Just noticed that [1] (and also [3]) is very cautious with $ and $$ to
expand environment variables.
javaOpts += "-Djava.io.tmpdir=" +
new Path(
YarnSparkHadoopUtil.expandEnvironment(Environment.PWD), // <-- here
YarnConfiguration.DEFAULT_CONTAINER_TEMP_DIR
)
Few lines below in the same method [2] the code doesn't seem to bother
to call $ directly :
Client.buildPath(Environment.PWD.$(), uri.getPath())
Why? Any particular reason for the difference in the two lines? #curious
[1]
https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala#L156
[2]
https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala#L203
[3]
https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala#L210
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
