Raviston Thanasekar created SPARK-61366:
-------------------------------------------
Summary: ORC filter pushdown drops rows for TIMESTAMP_LTZ(p) and
TIMESTAMP_NTZ(p) values before 1582-10-15
Key: SPARK-61366
URL: https://issues.apache.org/jira/browse/SPARK-61366
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Raviston Thanasekar
With spark.sql.orc.filterPushdown=true (the default), a predicate on a
TIMESTAMP_LTZ(p) or TIMESTAMP_NTZ(p) column
(spark.sql.timestampNanosTypes.enabled) can skip an ORC file whose rows match,
when the value is before 1582-10-15. With spark.sql.orc.filterPushdown=false
the same queries return the row.
{code:sql}
SET spark.sql.timestampNanosTypes.enabled=true;
CREATE TABLE o USING orc AS SELECT CAST('0001-01-02 00:00:00' AS
TIMESTAMP_LTZ(9)) AS t;
SELECT count(*) FROM o WHERE t = CAST('0001-01-02 00:00:00' AS
TIMESTAMP_LTZ(9)); -- 0, expected 1
SELECT count(*) FROM o WHERE t <= CAST('0001-01-02 00:00:00' AS
TIMESTAMP_LTZ(9)); -- 0, expected 1
{code}
Measured on master (e540d564a21), one file per value, comparing count() with
pushdown on and off, for the predicates =, <=, >=, <, > with the stored value.
The JVM and session time zones were both UTC, and then both
America/Los_Angeles; the results were the same.
||Type||Value||Wrong with pushdown on||
|TIMESTAMP_LTZ(9)|0001-01-02 00:00:00|=, <= (0 instead of 1)|
|TIMESTAMP_LTZ(9)|1000-01-01 00:00:00|=, >=|
|TIMESTAMP_LTZ(9)|1500-01-01 00:00:00|=, >=|
|TIMESTAMP_LTZ(9)|1582-10-04 20:00:00|=, >=|
|TIMESTAMP_NTZ(9)|the same four values|the same predicates|
|both|1582-10-16 00:00:00, 2020-05-25 10:11:12|none|
This looks like the same ORC behavior as SPARK-60152, which fixes it for
TimestampType in https://github.com/apache/spark/pull/60293. ORC's
RecordReaderImpl.evaluatePredicateProto deserializes the row group statistics
with ColumnStatisticsImpl.deserialize(..., writerUsedProlepticGregorian, true).
For a file written in the hybrid Julian/Gregorian calendar,
DateUtils.convertTimeToProleptic then moves the statistics whose wall clock is
before 1582-10-15 by up to 10 days. The literals that
TimestampNanosOrcOps.castFilterLiteral builds (java.sql.Timestamp.valueOf of a
LocalDateTime) are not converted the same way. I have not traced the exact
representation that TimestampNanosOrcOps writes, so the right fix (not pushing
such predicates down, as in SPARK-60152, or converting the literal) needs
checking.
The nanosecond timestamp types are behind the internal flag
spark.sql.timestampNanosTypes.enabled, which is false by default outside of
tests, and their ORC filter pushdown was added in SPARK-57823.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]