When writing a dataframe, a _SUCCESS file is created to mark that the entire dataframe is written. However, the existence of this _SUCCESS does not seem to be validated by default on reads. This would allow in some cases for partially written dataframes to be read back. Is this behavior configurable? Is lack of validation intentional?
Thanks! Here is an example from spark 2.1.0 shell. I would expect the read step to fail because I've manually removed the _SUCCESS file: scala> spark.range(10).write.save("/tmp/test") $ rm /tmp/test/_SUCCESS scala> spark.read.parquet("/tmp/test").show() +---+ | id| +---+ | 8| | 9| | 3| | 4| | 5| | 0| | 6| | 7| | 2| | 1| +---+ -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SUCCESS-file-validation-on-read-tp28564.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org