Metin Hatipogullari created SPARK-61738:
-------------------------------------------

             Summary: Parquet filter pushdown on TIMESTAMP_MILLIS columns 
truncates sub-millisecond literals and skips matching rows
                 Key: SPARK-61738
                 URL: https://issues.apache.org/jira/browse/SPARK-61738
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.5.9, 4.0.5, 4.1.4, 4.2.1, 5.0.0
            Reporter: Metin Hatipogullari


ParquetFilters.timestampToMillis floors the literal to whole milliseconds. That 
is safe for =, >, >=, <=, because Spark re-evaluates the filter after the scan, 
but not for < and <> (and NOT IN, which becomes notEq). Row groups whose min, 
or whose only value, equals the floored literal are skipped. Default 
configuration.

{code:python}
spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MILLIS")
for t in ["1900-01-01 00:00:00", "2020-01-01 00:00:00"]:   # one file each
    spark.sql(f"SELECT TIMESTAMP'{t}' AS 
t").coalesce(1).write.mode("append").parquet("/tmp/z1")
df = spark.read.parquet("/tmp/z1")
df.where("t < TIMESTAMP'1900-01-01 00:00:00.000001'").count()    # 0, expected 1
df.where("t <> TIMESTAMP'2020-01-01 00:00:00.000001'").count()   # 1, expected 2
{code}

Correct with spark.sql.parquet.filterPushdown=false, with TIMESTAMP_MICROS 
files, and with literals on the millisecond grid. Same result with the V2 
source and with the vectorized reader off.

Reproduced on 3.5.9, 4.0.5, 4.1.4, 4.2.1 and master (b2fc1bef1b).

The ORC equivalent was SPARK-60326 / SPARK-60436. For Parquet, 
valueCanMakeFilterOn already refuses sub-microsecond bounds; refusing literals 
that are not on the millisecond grid for ParquetTimestampMillisType would fix 
it (or round lt up and skip notEq for them).

Found while comparing the same files with pushdown on and off.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to