This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 51cea42a5860 [SPARK-57587][SQL][TEST] Generate TIME values with the
declared precision in RandomDataGenerator
51cea42a5860 is described below
commit 51cea42a58605ebdabbd9c0364c36d8d4c9ceced
Author: Maxim Gekk <[email protected]>
AuthorDate: Thu Jun 25 22:20:30 2026 +0200
[SPARK-57587][SQL][TEST] Generate TIME values with the declared precision
in RandomDataGenerator
### What changes were proposed in this pull request?
Make TIME test-data generation honor the declared `TimeType` precision:
- `RandomDataGenerator`: in the `case t: TimeType =>` branch, truncate the
special/interesting TIME values to `t.precision`. Previously only the uniform
random draw was truncated, while the special values (`00:00:00`,
`23:59:59.999999`, `23:59:59.999999999`) were emitted unmodified, so the
interesting-value path produced non-conforming values such as
`23:59:59.999999999` even for low precisions like `TimeType(0)`.
- `DataTypeTestUtils.timeTypes`: add an intermediate `TimeType(3)` between
`MIN_PRECISION` (0) and `MAX_PRECISION` (9), so the suites looping over
`ordered` / `atomicTypes` / `propertyCheckSupported` actually exercise a
non-endpoint precision.
`LiteralGenerator.timeLiteralGen` is already precision-aware (landed with
SPARK-57551), so no change is needed there.
This is the precision-conformance follow-up to SPARK-51403 (TIME as
ordered/atomic type) and SPARK-51669 (random TIME values in tests). SPARK-57551
raised `TimeType.MAX_PRECISION` from 6 to 9, widening the gap the generators
must cover.
### Why are the changes needed?
The precision dimension of `DataTypeTestUtils.timeTypes` was effectively
not exercised: the interesting-value path produced TIME values that do not
conform to the declared precision, and only the two endpoints (0 and 9) were
covered. Suites that loop over `ordered` / `atomicTypes` /
`propertyCheckSupported` (e.g. `PredicateSuite`, `ConditionalExpressionSuite`,
`ArithmeticExpressionSuite`, `OrderingSuite`, `SortSuite`, `CastSuite`,
`RandomDataGeneratorSuite`) therefore silently tested [...]
### Does this PR introduce _any_ user-facing change?
No. This only fixes test data generation; there is no production code or
versioning change.
### How was this patch tested?
By running the affected TIME-covering suites and confirming they pass:
`RandomDataGeneratorSuite`, `PredicateSuite`, `ConditionalExpressionSuite`,
`ArithmeticExpressionSuite`, `OrderingSuite`, `CastSuite`,
`CastWithAnsiOnSuite`, `UnsafeRowSuite`, and `SortSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
Closes #56779 from MaxGekk/time-fix-generators.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit bf0ce64f2ad697345b05c200235a514e09d5fc10)
Signed-off-by: Max Gekk <[email protected]>
---
.../test/scala/org/apache/spark/sql/RandomDataGenerator.scala | 10 ++++++++--
.../scala/org/apache/spark/sql/types/DataTypeTestUtils.scala | 1 +
2 files changed, 9 insertions(+), 2 deletions(-)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala
index 1390155fc7b6..1d618bb112fb 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/RandomDataGenerator.scala
@@ -319,11 +319,17 @@ object RandomDataGenerator {
.map(i => Instant.ofEpochSecond(i.getEpochSecond,
truncate(i.getNano).toLong))
)
case t: TimeType =>
+ // Honor the declared precision: both the uniform random draw and the
special values are
+ // truncated to `t.precision` so the generated TIME(p) values carry at
most p
+ // fractional-second digits (mirrors the nanosecond-timestamp branches
above).
val specialTimes = Seq(
"00:00:00",
"23:59:59.999999",
"23:59:59.999999999"
- )
+ ).map(LocalTime.parse)
+ .map(lt => DateTimeUtils.nanosToLocalTime(
+ DateTimeUtils.truncateTimeToPrecision(
+ DateTimeUtils.localTimeToNanos(lt), t.precision)))
randomNumeric[LocalTime](
rand,
(rand: Random) => {
@@ -332,7 +338,7 @@ object RandomDataGenerator {
rand.between(0L, 24 * 60 * 60 * 1000 * 1000 * 1000L),
t.precision)
DateTimeUtils.nanosToLocalTime(nanos)
},
- specialTimes.map(LocalTime.parse)
+ specialTimes
)
case CalendarIntervalType => Some(() => {
val months = rand.nextInt(1000)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeTestUtils.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeTestUtils.scala
index a33b6010f4b6..ceb435bd2552 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeTestUtils.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeTestUtils.scala
@@ -71,6 +71,7 @@ object DataTypeTestUtils {
val timeTypes: Seq[TimeType] = Seq(
TimeType(TimeType.MIN_PRECISION),
+ TimeType(3),
TimeType(TimeType.MAX_PRECISION))
val timestampNanosTypes: Seq[DatetimeType] = Seq(
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
