Raviston Thanasekar created SPARK-61463:
-------------------------------------------
Summary: ORC filter pushdown skips matching rows for TIMESTAMP
values in the repeated hour at the end of daylight saving time
Key: SPARK-61463
URL: https://issues.apache.org/jira/browse/SPARK-61463
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Raviston Thanasekar
With spark.sql.orc.filterPushdown=true (the default), a predicate on a
TIMESTAMP column can skip an ORC file whose rows match, when a value is in the
first occurrence of the hour that repeats at the end of daylight saving time in
the JVM time zone:
{code:sql}
-- JVM and session time zone America/Los_Angeles
CREATE TABLE t USING orc AS SELECT TIMESTAMP'2020-11-01 01:30:00-07:00' AS t;
SELECT count(*) FROM t WHERE t = TIMESTAMP'2020-11-01 01:30:00-07:00'; -- 0,
expected 1
SELECT count(*) FROM t WHERE t <= TIMESTAMP'2020-11-01 01:30:00-07:00'; -- 0,
expected 1
{code}
The same value with -08:00 (the second 01:30) is found. In Europe/Berlin,
TIMESTAMP'2024-10-27 02:30:00+02:00' returns 0 for = in the same way. With
spark.sql.orc.filterPushdown=false, all of them return 1.
The same happens for values within a few hours of 1900-01-01 00:00 in time
zones whose offset in 1900 differs from their current standard offset. With the
JVM in Europe/Madrid, on files of 1900-01-01 00:00, 00:30 and 1900-01-02 00:00,
t = TIMESTAMP'1900-01-01 00:00:00' returns 0 instead of 1, and t >=
TIMESTAMP'1900-01-01 00:00:00' returns 1 instead of 3. Other affected time
zones include CET, MET, Europe/Brussels, Europe/Amsterdam, Africa/Ceuta,
Antarctica/Casey and Pacific/Kiritimati.
Cause: the ORC writer stores a timestamp statistic as its wall clock, millis +
offset (SerializationUtils.convertToUtc, from TimestampTreeWriter). When the
reader evaluates a predicate (useUTCTimestamp=false),
TimestampStatisticsImpl.getMinimum()/getMaximum() convert it back with
SerializationUtils.convertFromUtc:
{code:java}
public static long convertFromUtc(TimeZone local, long time) {
int offset = local.getOffset(time - local.getRawOffset());
return time - offset;
}
{code}
It looks up the offset at time - rawOffset, which is not the value's instant
when the offset at that instant differs from the current raw offset:
* In the repeated hour, the wall clock maps to the second occurrence, so the
statistic of a value in the first one comes back one hour later than the value,
and the pushed down literal falls outside [min, max].
* Before 1900, java.util.TimeZone returns the current raw offset instead of the
historical one, so a wall clock near 1900-01-01 00:00 is converted with the
wrong offset.
The bug is in ORC (convertFromUtc is the same in apache/orc main), and ORC-451
looks related. On the Spark side, the predicate is correct, but ORC skips the
files.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]