[
https://issues.apache.org/jira/browse/SPARK-61738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61738:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> Parquet filter pushdown on TIMESTAMP_MILLIS columns truncates sub-millisecond
> literals and skips matching rows
> --------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61738
> URL: https://issues.apache.org/jira/browse/SPARK-61738
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.9, 5.0.0, 4.2.1, 4.0.5, 4.1.4
> Reporter: Metin Hatipogullari
> Priority: Major
> Labels: correctness, pull-request-available
>
> ParquetFilters.timestampToMillis floors the literal to whole milliseconds.
> That is safe for =, >, >=, <=, because Spark re-evaluates the filter after
> the scan, but not for < and <> (and NOT IN, which becomes notEq). Row groups
> whose min, or whose only value, equals the floored literal are skipped.
> Default configuration.
> {code:python}
> spark.conf.set("spark.sql.session.timeZone", "UTC")
> spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MILLIS")
> for t in ["1900-01-01 00:00:00", "2020-01-01 00:00:00"]: # one file each
> spark.sql(f"SELECT TIMESTAMP'{t}' AS
> t").coalesce(1).write.mode("append").parquet("/tmp/z1")
> df = spark.read.parquet("/tmp/z1")
> df.where("t < TIMESTAMP'1900-01-01 00:00:00.000001'").count() # 0,
> expected 1
> df.where("t <> TIMESTAMP'2020-01-01 00:00:00.000001'").count() # 1,
> expected 2
> {code}
> Correct with spark.sql.parquet.filterPushdown=false, with TIMESTAMP_MICROS
> files, and with literals on the millisecond grid. Same result with the V2
> source and with the vectorized reader off.
> Reproduced on 3.5.9, 4.0.5, 4.1.4, 4.2.1 and master (b2fc1bef1b).
> The ORC equivalent was SPARK-60326 / SPARK-60436. For Parquet,
> valueCanMakeFilterOn already refuses sub-microsecond bounds; refusing
> literals that are not on the millisecond grid for ParquetTimestampMillisType
> would fix it (or round lt up and skip notEq for them).
> Found while comparing the same files with pushdown on and off.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]