yyanyy commented on code in PR #57865:
URL: https://github.com/apache/spark/pull/57865#discussion_r3817415907
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSessionCatalogSuite.scala:
##########
@@ -98,6 +98,25 @@ class DataSourceV2DataFrameSessionCatalogSuite
verifyTable("t", df)
}
}
+
+ test("SPARK-58389: time travel options are ignored for V1 table writes") {
+ withTable("t") {
+ sql("CREATE TABLE t(c BIGINT) USING csv")
+ val df = spark.range(1).toDF("c")
+
+ df.write
+ .format(v2Format)
+ .option("versionAsOf", "1")
+ .insertInto("t")
+ df.write
Review Comment:
I tried the suggested replacement, but it does not exercise the V1 fallback
we want to protect. With `USE_V1_SOURCE_LIST` empty, the CSV table is loaded
through the V2 path. I confirmed this by temporarily throwing in `case Some(_:
V1Table)`; the suite still passed, meaning that branch was not reached.
The regression covered here is specifically a V2 writer entry point
resolving the existing table as a `V1Table` and returning to
`saveAsV1TableCommand` before rejecting `versionAsOf`:
https://github.com/apache/spark/blob/5009aff212877af88930b6c8f88c79b0e907bb1b/sql/core/src/main/scala/org/apache/spark/sql/classic/DataFrameWriter.scala#L494-L508
Therefore, I kept `v2Format` with the existing CSV V1 table.
`SaveMode.Ignore` is intentional: `insertInto` separately verifies an actual
write, while this second call verifies that `saveAsTable` reaches the same V1
fallback without throwing or mutating the existing table. I added a comment and
separated the assertions to make this clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
