Hi, Will it be a problem if the staging directory is already deleted? Because even the directory doesn't exist, fs.delete(stagingDirPath, true) won't cause failure but just return false.
Rostyslav Sotnychenko wrote > Hi all! > > I am a bit confused why Spark AM and Client are both trying to delete > Staging Directory. > > https://github.com/apache/spark/blob/branch-2.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1110 > https://github.com/apache/spark/blob/branch-2.1/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L233 > > As you can see, in case if a job was running on YARN in Cluster deployment > mode, both AM and Client will try to delete Staging directory if job > succeeded and eventually one of them will fail to do this, because the > other one already deleted the directory. > > Shouldn't we add some check to Client? > > > Thanks, > Rostyslav ----- Liang-Chi Hsieh | @viirya Spark Technology Center http://www.spark.tc/ -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Both-Spark-AM-and-Client-are-trying-to-delete-Staging-Directory-tp20588p20600.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
