Raviston Thanasekar created ORC-2236:
----------------------------------------
Summary: Predicate push down skips row groups with TIMESTAMP
values in the repeated hour at the end of daylight saving time
Key: ORC-2236
URL: https://issues.apache.org/jira/browse/ORC-2236
Project: ORC
Issue Type: Bug
Components: Java
Affects Versions: 2.3.1
Reporter: Raviston Thanasekar
Attachments: OrcDstRepro.java
With useUTCTimestamp=false (the default), a SearchArgument on a TIMESTAMP
column skips a row group whose values match, when a value is in the first
occurrence of the hour that repeats at the end of daylight saving time in the
JVM default time zone. With orc-core 2.3.1, the attached OrcDstRepro.java
writes one value per file with the Java API and reads it back with and without
{{t = value}}:
{noformat}
JVM time zone America/Los_Angeles
2020-11-01T01:30:00-07:00: value 2020-11-01 01:30:00.0 (1604219400000), file
min 2020-11-01 01:30:00.0 (1604223000000), rows 1, rows with t = value: 0 <--
row group skipped
2020-11-01T01:30:00-08:00: value 2020-11-01 01:30:00.0 (1604223000000), file
min 2020-11-01 01:30:00.0 (1604223000000), rows 1, rows with t = value: 1
2020-07-01T12:00:00-07:00: value 2020-07-01 12:00:00.0 (1593630000000), file
min 2020-07-01 12:00:00.0 (1593630000000), rows 1, rows with t = value: 1
JVM time zone Europe/Berlin
2024-10-27T02:30:00+02:00: value 2024-10-27 02:30:00.0 (1729989000000), file
min 2024-10-27 02:30:00.0 (1729992600000), rows 1, rows with t = value: 0 <--
row group skipped
2024-10-27T02:30:00+01:00: value 2024-10-27 02:30:00.0 (1729992600000), file
min 2024-10-27 02:30:00.0 (1729992600000), rows 1, rows with t = value: 1
JVM time zone Europe/Madrid
1900-01-01T00:00:00-00:14:44: value 1900-01-01 00:00:00.0 (-2208987916000),
file min 1900-01-01 00:00:00.0 (-2208992400000), rows 1, rows with t = value: 0
<-- row group skipped
1900-01-02T00:00:00-00:14:44: value 1900-01-02 00:00:00.0 (-2208901516000),
file min 1900-01-02 00:00:00.0 (-2208901516000), rows 1, rows with t = value: 1
{noformat}
The writer stores the statistic as the wall clock of the value, millis + offset
({{SerializationUtils.convertToUtc}} in {{TimestampTreeWriter}}).
{{TimestampStatisticsImpl.getMinimum()}} and {{getMaximum()}} convert it back
with {{SerializationUtils.convertFromUtc}}, which looks up the offset at {{time
- rawOffset}}:
{code:java}
public static long convertFromUtc(TimeZone local, long time) {
int offset = local.getOffset(time - local.getRawOffset());
return time - offset;
}
{code}
That isn't the offset of the value when the offset at that instant differs from
the current raw offset:
* A wall clock in the repeated hour belongs to two instants. {{convertFromUtc}}
always returns the second one, so the minimum and maximum of a row group with a
value in the first one are one hour later than the value.
* Before 1900, {{java.util.TimeZone}} returns the current raw offset instead of
the historical one. A wall clock near 1900-01-01 00:00 is then converted with a
different offset than it was written with, in time zones such as Europe/Madrid,
CET, Europe/Brussels, Africa/Ceuta, Antarctica/Casey and Pacific/Kiritimati.
As the wall clock alone can't tell the two instants apart, a fix probably needs
the minimum to use the earliest instant with that wall clock and the maximum
the latest one, so that the range still covers the values.
This was found through Spark, where it makes queries on ORC tables return fewer
rows with filter push down on: SPARK-61463. ORC-451 looks related.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
