Sid4030 commented on PR #59759: URL: https://github.com/apache/spark/pull/59759#issuecomment-6385854190
I agree with @spark-test that documenting this file-sink restart behavior in `docs/streaming/ss-migration-guide.md` is helpful for users upgrading streaming checkpoints. Here is the proposed entry under `## Upgrading from Structured Streaming 4.3 to 4.4`: - Since Spark 4.4, Structured Streaming validates partition columns when starting a query with a file stream sink (`DataStreamWriter.start` or `toTable`), matching batch file writes (`DataFrameWriter`). Streaming queries with invalid partition column types (e.g., `ARRAY`, `MAP`, `STRUCT`, `VARIANT`, or strings with non-binary collations like `UTF8_LCASE`), duplicate partition columns, or partitioning by all columns are now rejected at query start with `AnalysisException`. Existing streaming queries with such partition columns will fail to restart from their checkpoints; in such cases, users should update their partition specification and start with a new checkpoint location and output path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
