dongjoon-hyun commented on PR #59797: URL: https://github.com/apache/spark/pull/59797#issuecomment-6429903403
Thanks for raising this. I'd like to keep the change as it is and not add an `ss-migration-guide.md` entry. The new check makes the streaming file sink consistent with batch file writes. Batch writes have rejected empty and nested empty schemas since Spark 2.4, and `sql-migration-guide.md` already documents that. The PR description also calls out that existing path-sink queries writing empty or nested empty schemas fail to restart, and that `spark.sql.legacy.allowEmptySchemaWrite` restores the previous behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
