Re: Spark 2.3.0 DataFrame.write.parquet() behavior change from 2.2.0

Yea What’s the scenario you want the empty partitions configurable? Do you still need empty files? > 在 2018年5月8日，03:35，Victor Tso-Guillen > 写道： > > Found it: SPARK-21435 > > On Mon, May 7, 2018 at 2:18 PM Victor Tso-Guillen

Re: Spark 2.3.0 DataFrame.write.parquet() behavior change from 2.2.0

Found it: SPARK-21435 On Mon, May 7, 2018 at 2:18 PM Victor Tso-Guillen wrote: > It appears that between 2.2.0 and 2.3.0 DataFrame.write.parquet() skips > writing empty parquet files for empty partitions. Is this configurable? Is > there a Jira that tracks this change? > >

Spark 2.3.0 DataFrame.write.parquet() behavior change from 2.2.0

It appears that between 2.2.0 and 2.3.0 DataFrame.write.parquet() skips writing empty parquet files for empty partitions. Is this configurable? Is there a Jira that tracks this change? Thanks, Victor