This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 4a6081d5c8c7 [SPARK-57564][SQL][TEST] Add unit-test coverage for TIME
to match DATE/TIMESTAMP
4a6081d5c8c7 is described below
commit 4a6081d5c8c70f61bb19109d4ab7e0e27dacdf42
Author: Vaibhav Garg <[email protected]>
AuthorDate: Thu Jun 25 10:55:29 2026 +0200
[SPARK-57564][SQL][TEST] Add unit-test coverage for TIME to match
DATE/TIMESTAMP
### What changes were proposed in this pull request?
This PR adds unit-test coverage for the TIME data type (`TimeType`,
`java.time.LocalTime`) to three existing catalyst test suites, bringing TIME to
parity with the sibling DATE/TIMESTAMP test cases already present in each:
- **ExpressionEncoderSuite** — Added `LocalTime` encoder round-trip tests:
a plain time value, midnight boundary, max-microseconds boundary
(23:59:59.999999), an array of times, and `Option`/`Map` variants. Mirrors the
existing date/timestamp `encodeDecodeTest` cases.
- **DDLParserSuite** — Added a test covering `TIME` and `TIME(p)` precision
columns in `CREATE TABLE`, `ALTER TABLE ADD COLUMNS`, `ALTER COLUMN ... TYPE`,
a column with a TIME typed-literal `DEFAULT`, and `PARTITIONED BY` a TIME
column. Mirrors the existing SPARK-57164 nanosecond-timestamp DDL test.
- **DataTypeWriteCompatibilitySuite** — Added a dedicated TIME
write-compatibility test in the shared base suite so it executes under both the
strict (`canUpCast`) and ANSI (`canANSIStoreAssign`) store-assignment policies.
It checks TIME→TIME across precisions and TIME↔DATE/TIMESTAMP/TIMESTAMP_NTZ in
both directions, deriving the allowed/rejected expectation from the
policy-bound cast rule. Mirrors the existing SPARK-37707 datetime-compatibility
test.
No production code is changed. This is test-only.
### Why are the changes needed?
The TIME data type shipped in Spark 4.1.0 (SPIP SPARK-51162), but catalyst
unit-test coverage for TIME lags behind DATE and TIMESTAMP. This PR closes that
gap as part of the SPARK-57550 umbrella (extend TIME type support across the
codebase). Adequate test coverage is needed to catch regressions as further
TIME integration work lands.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
All three suites were run individually and pass locally:
- `build/sbt "catalyst/testOnly *ExpressionEncoderSuite"` — 399 tests pass
- `build/sbt "catalyst/testOnly
org.apache.spark.sql.catalyst.parser.DDLParserSuite"` — 149 tests pass
- `build/sbt "catalyst/testOnly *StrictDataTypeWriteCompatibilitySuite
*ANSIDataTypeWriteCompatibilitySuite"` — 60 tests pass (the new TIME test
executes under both policies)
No existing tests were modified.
### Was this patch authored or co-authored using generative AI tooling?
Yes. Generated using Kiro (Claude Opus 4.8).
Closes #56764 from vboo123/SPARK-57564.
Lead-authored-by: Vaibhav Garg <[email protected]>
Co-authored-by: Vaibhav Garg <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit 00e01d465167d32186abb9a01511b29986113701)
Signed-off-by: Max Gekk <[email protected]>
---
.../catalyst/encoders/ExpressionEncoderSuite.scala | 9 +++++
.../spark/sql/catalyst/parser/DDLParserSuite.scala | 33 +++++++++++++++-
.../types/DataTypeWriteCompatibilitySuite.scala | 44 ++++++++++++++++++++++
3 files changed, 85 insertions(+), 1 deletion(-)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala
index 287b99d10d65..ddc85b9b3ef4 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.catalyst.encoders
import java.math.BigInteger
import java.sql.{Date, Timestamp}
+import java.time.LocalTime
import java.util.Arrays
import scala.collection.mutable
@@ -210,6 +211,10 @@ class ExpressionEncoderSuite extends
CodegenInterpretedPlanTest with AnalysisTes
encodeDecodeTest(Date.valueOf("2012-12-23"), "date")
encodeDecodeTest(Timestamp.valueOf("2016-01-29 10:00:00"), "timestamp")
encodeDecodeTest(Array(Timestamp.valueOf("2016-01-29 10:00:00")), "array of
timestamp")
+ encodeDecodeTest(LocalTime.of(12, 34, 56), "SPARK-57564: time")
+ encodeDecodeTest(LocalTime.MIDNIGHT, "SPARK-57564: midnight time")
+ encodeDecodeTest(LocalTime.of(23, 59, 59, 999999000), "SPARK-57564: max
micros time")
+ encodeDecodeTest(Array(LocalTime.of(12, 34, 56)), "SPARK-57564: array of
time")
encodeDecodeTest(Array[Byte](13, 21, -23), "binary")
encodeDecodeTest(Seq(31, -123, 4), "seq of int")
@@ -454,6 +459,10 @@ class ExpressionEncoderSuite extends
CodegenInterpretedPlanTest with AnalysisTes
"SPARK-45896: seq of option of date")
encodeDecodeTest(Map(0 -> Some(Date.valueOf("2023-01-01"))),
"SPARK-45896: map of option of date")
+ encodeDecodeTest(Seq(Some(LocalTime.of(12, 34, 56))),
+ "SPARK-57564: seq of option of time")
+ encodeDecodeTest(Map(0 -> Some(LocalTime.of(12, 34, 56))),
+ "SPARK-57564: map of option of time")
encodeDecodeTest(Seq(Some(BigDecimal(200))), "SPARK-45896: seq of option of
bigdecimal")
encodeDecodeTest(Map(0 -> Some(BigDecimal(200))), "SPARK-45896: map of
option of bigdecimal")
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
index 2e8132926dcd..4edeb3176798 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
@@ -30,7 +30,7 @@ import
org.apache.spark.sql.connector.catalog.TableChange.ColumnPosition.{after,
import org.apache.spark.sql.connector.expressions.{ApplyTransform,
BucketTransform, ClusterByTransform, DaysTransform, FieldReference,
HoursTransform, IdentityTransform, LiteralValue, MonthsTransform, Transform,
YearsTransform}
import org.apache.spark.sql.connector.expressions.LogicalExpressions.bucket
import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.{DataType, Decimal, IntegerType, LongType,
StringType, StructType, TimestampLTZNanosType, TimestampNTZNanosType,
TimestampType}
+import org.apache.spark.sql.types.{DataType, Decimal, IntegerType, LongType,
StringType, StructType, TimestampLTZNanosType, TimestampNTZNanosType,
TimestampType, TimeType}
import org.apache.spark.storage.StorageLevelMapper
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
@@ -133,6 +133,37 @@ class DDLParserSuite extends AnalysisTest {
}
}
+ test("SPARK-57564: TIME type in CREATE TABLE / ALTER TABLE columns") {
+ Seq(
+ "TIME" -> TimeType(),
+ "TIME(0)" -> TimeType(0),
+ "TIME(3)" -> TimeType(3),
+ "TIME(6)" -> TimeType(6)).foreach {
+ case (spelling, expected) =>
+ // CREATE TABLE column.
+ val created = parsePlan(s"CREATE TABLE t (c $spelling) USING parquet")
+ assert(created.asInstanceOf[CreateTable].columns.head.dataType ===
expected)
+ // ALTER TABLE ... ADD COLUMNS.
+ val added = parsePlan(s"ALTER TABLE t ADD COLUMNS (c $spelling)")
+ assert(added.asInstanceOf[AddColumns].columnsToAdd.head.dataType ===
expected)
+ // ALTER TABLE ... ALTER COLUMN ... TYPE.
+ val altered = parsePlan(s"ALTER TABLE t ALTER COLUMN c TYPE $spelling")
+ assert(altered.asInstanceOf[AlterColumns].specs.head.newDataType ===
Some(expected))
+ }
+ // A column DEFAULT declared with a TIME type and a TIME typed-literal
default.
+ val withDefault = parsePlan(
+ "CREATE TABLE t (c TIME DEFAULT TIME '12:34:56') USING parquet")
+ val colDef = withDefault.asInstanceOf[CreateTable].columns.head
+ assert(colDef.dataType === TimeType())
+ assert(colDef.defaultValue.isDefined)
+ // PARTITIONED BY a TIME column.
+ val partitioned = parsePlan(
+ "CREATE TABLE t (id INT, c TIME) USING parquet PARTITIONED BY (c)")
+ val createTable = partitioned.asInstanceOf[CreateTable]
+ assert(createTable.columns.exists(col => col.name == "c" && col.dataType
=== TimeType()))
+ assert(createTable.partitioning ===
Seq(IdentityTransform(FieldReference("c"))))
+ }
+
test("create/replace table - with IF NOT EXISTS") {
val sql = "CREATE TABLE IF NOT EXISTS my_tab(a INT, b STRING) USING
parquet"
testCreateOrReplaceDdl(
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeWriteCompatibilitySuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeWriteCompatibilitySuite.scala
index ba3eaf46a559..131eab34762f 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeWriteCompatibilitySuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeWriteCompatibilitySuite.scala
@@ -320,6 +320,50 @@ abstract class DataTypeWriteCompatibilityBaseSuite extends
SparkFunSuite {
}
}
+ test("SPARK-57564: Check TIME type write compatibility") {
+ // Expectations are derived from the policy-bound `canCast` (canUpCast for
the strict suite,
+ // canANSIStoreAssign for the ANSI suite), so this single test stays
correct under both
+ // subclasses: strict allows only TIME -> identical TIME, while ANSI
additionally allows
+ // writes across datetime types and across TIME precisions.
+ def checkPair(write: DataType, read: DataType): Unit = {
+ if (canCast(write, read)) {
+ assertAllowed(write, read, "t",
+ s"Should allow writing $write to $read because cast is safe")
+ } else {
+ val errs = new mutable.ArrayBuffer[String]()
+ checkError(
+ exception = intercept[AnalysisException] (
+ DataTypeUtils.canWrite("", write, read, true,
analysis.caseSensitiveResolution,
+ "t", storeAssignmentPolicy, errMsg => errs += errMsg)
+ ),
+ condition = "INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_SAFELY_CAST",
+ parameters = Map(
+ "tableName" -> "``",
+ "colName" -> "`t`",
+ "srcType" -> toSQLType(write),
+ "targetType" -> toSQLType(read)
+ )
+ )
+ }
+ }
+
+ val timeTypes = Seq(TimeType(0), TimeType(3), TimeType(6))
+ // TIME -> TIME across all precision combinations (both directions via
full cross product).
+ timeTypes.foreach { w =>
+ timeTypes.foreach { r =>
+ checkPair(w, r)
+ }
+ }
+ // TIME <-> other datetime types, both directions.
+ val otherDateTimeTypes = Seq(DateType, TimestampType, TimestampNTZType)
+ timeTypes.foreach { t =>
+ otherDateTimeTypes.foreach { o =>
+ checkPair(t, o)
+ checkPair(o, t)
+ }
+ }
+ }
+
test("Check struct types: missing required field") {
val missingRequiredField = StructType(Seq(StructField("x", FloatType,
nullable = false)))
val errs = new mutable.ArrayBuffer[String]()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
