Metin Hatipogullari created SPARK-61738:
-------------------------------------------
Summary: Parquet filter pushdown on TIMESTAMP_MILLIS columns
truncates sub-millisecond literals and skips matching rows
Key: SPARK-61738
URL: https://issues.apache.org/jira/browse/SPARK-61738
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.5.9, 4.0.5, 4.1.4, 4.2.1, 5.0.0
Reporter: Metin Hatipogullari
ParquetFilters.timestampToMillis floors the literal to whole milliseconds. That
is safe for =, >, >=, <=, because Spark re-evaluates the filter after the scan,
but not for < and <> (and NOT IN, which becomes notEq). Row groups whose min,
or whose only value, equals the floored literal are skipped. Default
configuration.
{code:python}
spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MILLIS")
for t in ["1900-01-01 00:00:00", "2020-01-01 00:00:00"]: # one file each
spark.sql(f"SELECT TIMESTAMP'{t}' AS
t").coalesce(1).write.mode("append").parquet("/tmp/z1")
df = spark.read.parquet("/tmp/z1")
df.where("t < TIMESTAMP'1900-01-01 00:00:00.000001'").count() # 0, expected 1
df.where("t <> TIMESTAMP'2020-01-01 00:00:00.000001'").count() # 1, expected 2
{code}
Correct with spark.sql.parquet.filterPushdown=false, with TIMESTAMP_MICROS
files, and with literals on the millisecond grid. Same result with the V2
source and with the vectorized reader off.
Reproduced on 3.5.9, 4.0.5, 4.1.4, 4.2.1 and master (b2fc1bef1b).
The ORC equivalent was SPARK-60326 / SPARK-60436. For Parquet,
valueCanMakeFilterOn already refuses sub-microsecond bounds; refusing literals
that are not on the millisecond grid for ParquetTimestampMillisType would fix
it (or round lt up and skip notEq for them).
Found while comparing the same files with pushdown on and off.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]