GitHub user dilipbiswal opened a pull request: https://github.com/apache/spark/pull/20525
SPARK-23271 Parquet output contains only _SUCCESS file after writing an empty dataframe ## What changes were proposed in this pull request? Below are the two cases. ``` SQL case 1 scala> List.empty[String].toDF().rdd.partitions.length res18: Int = 1 ``` When we write the above data frame as parquet, we create a parquet file containing just the schema of the data frame. Case 2 ``` SQL scala> val anySchema = StructType(StructField("anyName", StringType, nullable = false) :: Nil) anySchema: org.apache.spark.sql.types.StructType = StructType(StructField(anyName,StringType,false)) scala> spark.read.schema(anySchema).csv("/tmp/empty_folder").rdd.partitions.length res22: Int = 0 ``` For the 2nd case, since number of partitions = 0, we don't call the write task (the task has logic to create the empty metadata only parquet file) The fix attempts to repartition the empty rdd to size 1 before we proceed to setup the write job. ## How was this patch tested? A new test is added to DataframeReaderWriterSuite. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dilipbiswal/spark spark-23271 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20525.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20525 ---- commit 2764b1c0aa43a104393da909f388861209220d4f Author: Dilip Biswal <dbiswal@...> Date: 2018-02-07T07:45:32Z SPARK-23271 Parquet output contains only _SUCCESS file after writing an empty dataframe ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org