spark-test commented on code in PR #59797:
URL: https://github.com/apache/spark/pull/59797#discussion_r4383563722
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala:
##########
@@ -293,6 +293,52 @@ abstract class FileStreamSinkSuite extends StreamTest {
}
}
+ test("SPARK-60563: Check INTERVAL and empty struct columns when starting a
query") {
+ val df = MemoryStream[Int].toDF()
+ val intervalDf = df.selectExpr("value AS i", "make_interval(0, 0, 0,
value) AS d")
Review Comment:
**Nit (P3):** This test pins only the rejecting side of the new interval
check, and only through `start(path)`: `intervalDf` uses `make_interval` (a
CalendarInterval), and the only `toTable` case below streams `emptyStructDf`.
Two behaviors the PR description declares have no assertion that would fail on
regression:
- ANSI intervals stay allowed. `DataSource.createSink` passes
`forbidAnsiIntervals = false`, but nothing here starts a file-sink query with a
`make_ym_interval` or `make_dt_interval` column, and I found no other streaming
file-sink test that does. If that argument were flipped, such queries would
fail at `start()` for every format and this test would still pass.
- `toTable` rejects INTERVAL columns. That holds only because the
`disallowWritingIntervals` call sits outside the `if (catalogTable.isEmpty)`
block. `toTable` into an existing V1 file-source table does not compare the
stream schema with the table schema before `createSink`, so moving the call
inside that block would undetectably bring back the late failure at the first
micro-batch.
The production code behaves correctly today; this is only about the new
test's regression sensitivity.
**Verification:**
- **Behavior:** Starting a file stream sink query with ANSI year-month and
day-time interval columns succeeds for each covered built-in file format, and
the test would fail if the start-time check began rejecting ANSI intervals.
- **Regression:** Writing a stream with a CalendarInterval column to an
existing file-source table through toTable fails at query start with
UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE, and the test would fail if the interval
check were moved under the path-only empty-schema condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
