Jungtaek Lim created SPARK-29999: ------------------------------------ Summary: Writing empty partition via DSv2 implementation of FileStreamSink throws FileNotFoundException Key: SPARK-29999 URL: https://issues.apache.org/jira/browse/SPARK-29999 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.0.0 Reporter: Jungtaek Lim
I've discovered this issue while playing with Spark 3.0.0-preview. Unlike DSv1 version of file sink, DSv2 version of file sink lazily creates output file which leads empty partition to not actually create output file. The behavior itself is not a problem, but while committing task in ManifestFileCommitProtocol, `fs.getFileStatus` throws FileNotFoundException since the file is not created. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org