I'm just circling back to this now. Is the commit protocol an acceptable way of making this configureable? I could make the temp path (currently "_temporary") configureable if that is what you are referring to.
Michael Armbrust wrote > We didn't go this way initially because it doesn't work on storage systems > that have weaker guarantees than HDFS with respect to rename. That said, > I'm happy to look at other options if we want to make this configurable. > >> After hesitating for a while, I wrote a custom commit protocol to solve >> the problem. It combines HadoopMapReduceCommitProtocol's behavior of >> writing to a temp file first, with ManifestFileCommitProtocol. From what >> I can tell ManifestFileCommitProtocol is required for the normal >> Structured >> Streaming behavior of being able to resume streams from a known point. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
