stevomitric commented on code in PR #57681:
URL: https://github.com/apache/spark/pull/57681#discussion_r3753476711
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/types/ops/TimestampNanosOrcOps.scala:
##########
@@ -69,6 +79,19 @@ case class TimestampLTZNanosOrcOps(t: TimestampLTZNanosType)
extends OrcTypeOps
val instant = ts.toInstant
set(ordinal, DateTimeUtils.instantToTimestampNanos(instant, t.precision))
}
+
+ // The physical ORC category is a timestamp, so the search argument uses the
TIMESTAMP leaf type.
+ override def predicateLeafType: Option[PredicateLeaf.Type] =
Some(PredicateLeaf.Type.TIMESTAMP)
+
+ // The filter literal is an external java.time.Instant (see
CatalystTypeConverters). ORC evaluates
+ // a TIMESTAMP_INSTANT predicate against the stored UTC value shifted into
the JVM default zone,
+ // so the literal must be a java.sql.Timestamp whose local wall clock equals
the instant's UTC
+ // wall clock; Timestamp.valueOf(LocalDateTime at UTC) produces exactly
that. Any non-Instant
+ // value is passed through unchanged.
+ override def castFilterLiteral(value: Any): Any = value match {
+ case i: Instant => Timestamp.valueOf(LocalDateTime.ofInstant(i,
ZoneOffset.UTC))
Review Comment:
swapped in the proposed new `Timestamp(i.toEpochMilli)+setNanos` and re-ran
the e2e test under America/Los_Angeles — it fails `checkAnswer `.
With useUTCTimestamp=false, ORC compares against
`TimestampStatisticsImpl.getMinimum()`, which is
`convertFromUtc(TimeZone.getDefault(), stored)`, the local wall-clock frame,
not the raw UTC epoch. So the literal must be in that frame.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
