dongjoon-hyun commented on PR #59894: URL: https://github.com/apache/spark/pull/59894#issuecomment-6485698400
Thanks for the review. The inline threads cover the three findings, all addressed in 450d0bd8529. Here are the replies to the review-level concern and the decision challenges. The PR description is updated accordingly. **Restart policy.** A restarted query no longer switches the layout of a sink log that already has committed files. It fails to start with `STREAMING_FILE_SINK_PARTITIONING_MISMATCH` before it writes, so the committed rows stay readable. The in-place changes you describe, such as dropping `partitionBy` or setting it to the partitioning of the table, now fail at the start instead of mixing layouts in `_spark_metadata`. Sink logs that already mix layouts and path-based `start(path)` writers are not touched. **Explicit empty `partitionBy()`.** It is now handled as not given (`partitioningColumns.filter(_.nonEmpty)`), so classic Scala, Java, and PySpark behave like Spark Connect, and a query with `partitionBy()` against a partitioned table writes with the partitioning of the table. The test for a partitioned table covers it. I did not add a Connect test because the server code is unchanged. **`V2TableWithV1Fallback.v1Table`.** In this tree the only implementations are test fixtures, and this branch already treats `v1Table` as authoritative for the view check, the provider, and the location, so I kept it for the partition check and the sink layout as well. I cannot verify implementations outside this tree. If one of them does not keep its partition columns in `v1Table`, limiting the check to `V1Table` is straightforward. **Legacy flag, migration guide, and target release.** This PR does not add a legacy configuration or a migration-guide entry, and the target branch is left to the committers. The error message names the recovery: a new table location with a new checkpoint location, or rewriting the committed files with the partition columns of the table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
