Thanks all! I was using another DFS instead of HDFS, which was logging an error when fs.delete got called on non-existing path. In Spark 2.0.1 which I was using previously, everything was working fine because existence of an additional check that was made prior to deleting. However that check got removed in 2.1 (SPARK-16736 <https://issues.apache.org/jira/browse/SPARK-16736>, commit <https://github.com/apache/spark/pull/14371/files#diff-b050df3f55b82065803d6e83453b9706>), so I started seeing an error from my DFS.
Its not a problem in any way (i.e. it does not affect Spark job in any way), so everything is fine. I just wanted to make sure its not a Spark issue. Thanks, Rostyslav On Sun, Jan 15, 2017 at 3:19 PM, Liang-Chi Hsieh <vii...@gmail.com> wrote: > > Hi, > > Will it be a problem if the staging directory is already deleted? Because > even the directory doesn't exist, fs.delete(stagingDirPath, true) won't > cause failure but just return false. > > > Rostyslav Sotnychenko wrote > > Hi all! > > > > I am a bit confused why Spark AM and Client are both trying to delete > > Staging Directory. > > > > https://github.com/apache/spark/blob/branch-2.1/yarn/ > src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L1110 > > https://github.com/apache/spark/blob/branch-2.1/yarn/ > src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L233 > > > > As you can see, in case if a job was running on YARN in Cluster > deployment > > mode, both AM and Client will try to delete Staging directory if job > > succeeded and eventually one of them will fail to do this, because the > > other one already deleted the directory. > > > > Shouldn't we add some check to Client? > > > > > > Thanks, > > Rostyslav > > > > > > ----- > Liang-Chi Hsieh | @viirya > Spark Technology Center > http://www.spark.tc/ > -- > View this message in context: http://apache-spark- > developers-list.1001551.n3.nabble.com/Both-Spark-AM-and- > Client-are-trying-to-delete-Staging-Directory-tp20588p20600.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
