Actually this has been merged to the master branch https://github.com/apache/spark/pull/947
-- Nan Zhu On Thursday, June 12, 2014 at 2:39 PM, Daniel Siegmann wrote: > The old behavior (A) was dangerous, so it's good that (B) is now the default. > But in some cases I really do want to replace the old data, as per (C). For > example, I may rerun a previous computation (perhaps the input data was > corrupt and I'm rerunning with good input). > > Currently I have to write separate code to remove the files before calling > Spark. It would be very convenient if Spark could do this for me. Has anyone > created a JIRA issue to support (C)? > > > On Mon, Jun 9, 2014 at 3:02 AM, Aaron Davidson <ilike...@gmail.com > (mailto:ilike...@gmail.com)> wrote: > > It is not a very good idea to save the results in the exact same place as > > the data. Any failures during the job could lead to corrupted data, because > > recomputing the lost partitions would involve reading the original > > (now-nonexistent) data. > > > > As such, the only "safe" way to do this would be to do as you said, and > > only delete the input data once the entire output has been successfully > > created. > > > > > > On Sun, Jun 8, 2014 at 10:32 PM, innowireless TaeYun Kim > > <taeyun....@innowireless.co.kr (mailto:taeyun....@innowireless.co.kr)> > > wrote: > > > Without (C), what is the best practice to implement the following > > > scenario? > > > > > > 1. rdd = sc.textFile(FileA) > > > 2. rdd = rdd.map(...) // actually modifying the rdd > > > 3. rdd.saveAsTextFile(FileA) > > > > > > Since the rdd transformation is 'lazy', rdd will not materialize until > > > saveAsTextFile(), so FileA must still exist, but it must be deleted before > > > saveAsTextFile(). > > > > > > What I can think is: > > > > > > 3. rdd.saveAsTextFile(TempFile) > > > 4. delete FileA > > > 5. rename TempFile to FileA > > > > > > This is not very convenient... > > > > > > Thanks. > > > > > > -----Original Message----- > > > From: Patrick Wendell [mailto:pwend...@gmail.com] > > > Sent: Tuesday, June 03, 2014 11:40 AM > > > To: user@spark.apache.org (mailto:user@spark.apache.org) > > > Subject: Re: How can I make Spark 1.0 saveAsTextFile to overwrite existing > > > file > > > > > > (A) Semantics in Spark 0.9 and earlier: Spark will ignore Hadoo's output > > > format check and overwrite files in the destination directory. > > > But it won't clobber the directory entirely. I.e. if the directory already > > > had "part1" "part2" "part3" "part4" and you write a new job outputing only > > > two files ("part1", "part2") then it would leave the other two files > > > intact, > > > confusingly. > > > > > > (B) Semantics in Spark 1.0 and earlier: Runs Hadoop OutputFormat check > > > which > > > means the directory must not exist already or an excpetion is thrown. > > > > > > (C) Semantics proposed by Nicholas Chammas in this thread (AFAIK): > > > Spark will delete/clobber an existing destination directory if it exists, > > > then fully over-write it with new data. > > > > > > I'm fine to add a flag that allows (B) for backwards-compatibility > > > reasons, > > > but my point was I'd prefer not to have (C) even though I see some cases > > > where it would be useful. > > > > > > - Patrick > > > > > > On Mon, Jun 2, 2014 at 4:25 PM, Sean Owen <so...@cloudera.com > > > (mailto:so...@cloudera.com)> wrote: > > > > Is there a third way? Unless I miss something. Hadoop's OutputFormat > > > > wants the target dir to not exist no matter what, so it's just a > > > > question of whether Spark deletes it for you or errors. > > > > > > > > On Tue, Jun 3, 2014 at 12:22 AM, Patrick Wendell <pwend...@gmail.com > > > > (mailto:pwend...@gmail.com)> > > > wrote: > > > >> We can just add back a flag to make it backwards compatible - it was > > > >> just missed during the original PR. > > > >> > > > >> Adding a *third* set of "clobber" semantics, I'm slightly -1 on that > > > >> for the following reasons: > > > >> > > > >> 1. It's scary to have Spark recursively deleting user files, could > > > >> easily lead to users deleting data by mistake if they don't > > > >> understand the exact semantics. > > > >> 2. It would introduce a third set of semantics here for saveAsXX... > > > >> 3. It's trivial for users to implement this with two lines of code > > > >> (if output dir exists, delete it) before calling saveAsHadoopFile. > > > >> > > > >> - Patrick > > > >> > > > > > > > > > -- > Daniel Siegmann, Software Developer > Velos > Accelerating Machine Learning > > 440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001 > E: daniel.siegm...@velos.io (mailto:daniel.siegm...@velos.io) W: www.velos.io > (http://www.velos.io)
