peter-toth commented on code in PR #59894:
URL: https://github.com/apache/spark/pull/59894#discussion_r4413872584
##########
sql/api/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala:
##########
@@ -215,8 +215,11 @@ abstract class DataStreamWriter[T] extends
WriteConfigMethods[DataStreamWriter[T
* [[org.apache.spark.sql.streaming.StreamingQuery]] object can be used to
interact with the
* stream.
*
- * For v1 table, partitioning columns provided by `partitionBy` will be
respected no matter the
- * table exists or not. A new table will be created if the table not exists.
+ * For v1 table, a new table will be created with the partitioning columns
provided by
Review Comment:
**Finding 1.** This Scaladoc and the PySpark docstring are the only places
that tell users about the new start-time failures. A user who upgrades a
running query learns about them from the exception on restart.
`docs/streaming/ss-migration-guide.md` has an entry of this kind for
SPARK-60520's partition-column validation. Something like this, in the section
of the release this ships in:
```markdown
- Since Spark 4.4, `DataStreamWriter.toTable` writes to an existing V1 table
with the partitioning of the table. A query whose `partitionBy` does not match
the partitioning of the table, including the order of the columns, fails to
start. A query that already committed files to the table in another layout, for
example without `partitionBy` to a partitioned table, fails to start with
`STREAMING_FILE_SINK_PARTITIONING_MISMATCH`. To recover, write to a new table
location with a new checkpoint location, or rewrite the committed files with
the partition columns of the table.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
