After repartitioning a DataFrame in Spark 1.3.0 I get a .parquet exception
when saving toAmazon's S3. The data that I try to write is 10G.
logsForDate
.repartition(10)
.saveAsParquetFile(destination) // -- Exception here
The exception I receive is:
java.io.IOException: The file being
After repartitioning a *DataFrame* in *Spark 1.3.0* I get a *.parquet*
exception
when saving to*Amazon's S3*. The data that I try to write is 10G.
logsForDate
.repartition(10)
.saveAsParquetFile(destination) // -- Exception here
The exception I receive is:
java.io.IOException: The file
How can one disable *Partition discovery* in *Spark 1.3.0 *when using
*sqlContext.parquetFile*?
Alternatively, is there a way to load *.parquet* files without *Partition
discovery*?
Cosmin
