cloud-fan commented on code in PR #57680:
URL: https://github.com/apache/spark/pull/57680#discussion_r3702113321
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala:
##########
@@ -302,6 +302,35 @@ abstract class TypeCoercionHelper {
}
}
+ /** Whether `dt` is on the LTZ/NTZ timestamp fractional-precision axis (a
micro or nanos type). */
+ private def isTimestampFamily(dt: DataType): Boolean =
+ TimestampFamily.fractionalPrecision(dt).isDefined
+
+ /**
+ * Common operand type for [[SubtractTimestamps]] over two differing
timestamp-family operands.
+ * The operands are widened to the larger of the two fractional-second
precisions (the micro types
+ * count as 6, the nanos types carry their own precision `p` in [7, 9]) and
unified in one
+ * time-zone family:
+ * - a cross-family pair unifies in the no-time-zone (NTZ) family,
mirroring the microsecond
+ * precedent where TIMESTAMP - TIMESTAMP_NTZ coerces both operands to
TIMESTAMP_NTZ;
+ * - a same-family pair keeps that family, so a TIMESTAMP -
TIMESTAMP_LTZ(p) style pair still
+ * subtracts in the session time zone (DST-aware) exactly as a pure LTZ
pair does.
+ * The subtraction reads only each operand's epochMicros and always yields a
microsecond-grid
Review Comment:
Legacy mode returns `CalendarIntervalType`, so this is not always a
`DayTimeIntervalType`. Please describe it as a microsecond-grid interval or
qualify the concrete type by configuration.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala:
##########
@@ -2407,6 +2407,91 @@ class DateExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
}
}
+ test("SPARK-57832: subtract nanosecond-precision timestamps") {
+ // The difference between two nanosecond timestamps is reported on the
microsecond grid: only
+ // each operand's epochMicros participates, so the sub-microsecond
remainder is truncated. The
+ // two operands below share the same wall clock down to the microsecond
and differ only within
Review Comment:
These operands are on different days and use `.123456` versus `.000000`, so
they do not match through the microsecond. Please describe this case as
dropping only the sub-microsecond digits; the later pair is the one that should
explain the zero result.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala:
##########
@@ -4523,7 +4523,13 @@ case class SubtractTimestamps(
def this(endTimestamp: Expression, startTimestamp: Expression) =
this(endTimestamp, startTimestamp, SQLConf.get.legacyIntervalEnabled)
- override def inputTypes: Seq[AbstractDataType] = Seq(AnyTimestampType,
AnyTimestampType)
+ // Nanosecond-precision timestamps are accepted alongside the microsecond
types. The difference is
+ // always reported on the microsecond grid (a DayTimeIntervalType has
microsecond resolution), so
Review Comment:
The microsecond-grid behavior is common to both modes, but
`DayTimeIntervalType` is only the non-legacy result. Please qualify the
parenthetical because legacy mode returns `CalendarIntervalType`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
