Metin Hatipogullari created SPARK-61739:
-------------------------------------------

             Summary: Filter pushdown with a DATE or TIMESTAMP literal between 
1582-10-05 and 1582-10-14 skips matching rows
                 Key: SPARK-61739
                 URL: https://issues.apache.org/jira/browse/SPARK-61739
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.5.9, 4.0.5, 4.1.4, 4.2.1, 5.0.0
            Reporter: Metin Hatipogullari


Those days exist in Spark's proleptic Gregorian calendar but not in 
java.sql.Date's hybrid calendar. DataSourceStrategy converts the literal with 
convertToScala, so the pushed filter carries 1582-10-15 (DATE) or a time ten 
days later (TIMESTAMP), and row groups are skipped. Default configuration.

{code:python}
import datetime
for d in ["1582-10-04", "1582-10-15", "1582-10-16"]:   # one file each
    spark.createDataFrame([(datetime.date.fromisoformat(d),)], "d 
DATE").coalesce(1).write.mode("append").parquet("/tmp/gap")
df = spark.read.parquet("/tmp/gap")
df.where("d > DATE'1582-10-10'").count()    # 1, expected 2
df.where("d <> DATE'1582-10-10'").count()   # 2, expected 3
{code}

PushedFilters shows GreaterThan(d,1582-10-15). TIMESTAMP_MICROS Parquet columns 
are affected as well. ORC DATE columns behave the same, also with 
spark.sql.datetime.java8API.enabled=true (OrcFilters converts LocalDate back to 
java.sql.Date). For ORC TIMESTAMP, SPARK-60152 (apache/spark#60293) already 
stops pushing these literals, but only inside OrcFilters, so the Parquet path 
and ORC DATE are still affected.

When the file itself holds a gap value (any writer using the proleptic 
calendar: Spark in CORRECTED mode, pyarrow, delta-rs), equality fails too:
{code:python}
import datetime, pyarrow as pa, pyarrow.parquet as pq
pq.write_table(pa.table({"d": pa.array([datetime.date(1582, 10, 10)], 
pa.date32())}), "/tmp/gap2/a.parquet")
spark.read.parquet("/tmp/gap2").where("d = DATE'1582-10-10'").count()   # 0, 
expected 1
{code}
The same happens through Delta tables (spark.read.format("delta")), because the 
scan uses the same Parquet pushdown; it does not depend on Delta's data 
skipping.

Reproduced on 3.5.9, 4.0.5, 4.1.4, 4.2.1 and master. I have a patch that skips 
pushdown for literals in the gap, with tests, and can open a PR.

Found while comparing the same files with pushdown on and off.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to