Raviston Thanasekar created SPARK-61366:
-------------------------------------------

             Summary: ORC filter pushdown drops rows for TIMESTAMP_LTZ(p) and 
TIMESTAMP_NTZ(p) values before 1582-10-15
                 Key: SPARK-61366
                 URL: https://issues.apache.org/jira/browse/SPARK-61366
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Raviston Thanasekar


With spark.sql.orc.filterPushdown=true (the default), a predicate on a 
TIMESTAMP_LTZ(p) or TIMESTAMP_NTZ(p) column 
(spark.sql.timestampNanosTypes.enabled) can skip an ORC file whose rows match, 
when the value is before 1582-10-15. With spark.sql.orc.filterPushdown=false 
the same queries return the row.

{code:sql}
SET spark.sql.timestampNanosTypes.enabled=true;
CREATE TABLE o USING orc AS SELECT CAST('0001-01-02 00:00:00' AS 
TIMESTAMP_LTZ(9)) AS t;
SELECT count(*) FROM o WHERE t = CAST('0001-01-02 00:00:00' AS 
TIMESTAMP_LTZ(9));   -- 0, expected 1
SELECT count(*) FROM o WHERE t <= CAST('0001-01-02 00:00:00' AS 
TIMESTAMP_LTZ(9));  -- 0, expected 1
{code}

Measured on master (e540d564a21), one file per value, comparing count() with 
pushdown on and off, for the predicates =, <=, >=, <, > with the stored value. 
The JVM and session time zones were both UTC, and then both 
America/Los_Angeles; the results were the same.

||Type||Value||Wrong with pushdown on||
|TIMESTAMP_LTZ(9)|0001-01-02 00:00:00|=, <= (0 instead of 1)|
|TIMESTAMP_LTZ(9)|1000-01-01 00:00:00|=, >=|
|TIMESTAMP_LTZ(9)|1500-01-01 00:00:00|=, >=|
|TIMESTAMP_LTZ(9)|1582-10-04 20:00:00|=, >=|
|TIMESTAMP_NTZ(9)|the same four values|the same predicates|
|both|1582-10-16 00:00:00, 2020-05-25 10:11:12|none|

This looks like the same ORC behavior as SPARK-60152, which fixes it for 
TimestampType in https://github.com/apache/spark/pull/60293. ORC's 
RecordReaderImpl.evaluatePredicateProto deserializes the row group statistics 
with ColumnStatisticsImpl.deserialize(..., writerUsedProlepticGregorian, true). 
For a file written in the hybrid Julian/Gregorian calendar, 
DateUtils.convertTimeToProleptic then moves the statistics whose wall clock is 
before 1582-10-15 by up to 10 days. The literals that 
TimestampNanosOrcOps.castFilterLiteral builds (java.sql.Timestamp.valueOf of a 
LocalDateTime) are not converted the same way. I have not traced the exact 
representation that TimestampNanosOrcOps writes, so the right fix (not pushing 
such predicates down, as in SPARK-60152, or converting the literal) needs 
checking.

The nanosecond timestamp types are behind the internal flag 
spark.sql.timestampNanosTypes.enabled, which is false by default outside of 
tests, and their ORC filter pushdown was added in SPARK-57823.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to