[
https://issues.apache.org/jira/browse/SPARK-61543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61543:
-----------------------------------
Labels: pull-request-available (was: )
> JDBC pushdown renders java.time.Instant filter literals in the session time
> zone, so PostgreSQL returns wrong rows when spark.sql.session.timeZone
> differs from the JVM time zone
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61543
> URL: https://issues.apache.org/jira/browse/SPARK-61543
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Aleksandar Trajković
> Priority: Major
> Labels: pull-request-available
>
> JDBC pushdown renders java.time.Instant filter literals in the session time
> zone, so PostgreSQL returns wrong rows when spark.sql.session.timeZone
> differs from the JVM time zone
> When spark.sql.datetime.java8API.enabled=true (the Thrift server always
> enables it), TIMESTAMP literals in pushed-down filters reach
> JdbcDialect.compileValue as java.time.Instant. compileValue formats them as a
> zoneless string in spark.sql.session.timeZone. With java8API=false the
> literal is a java.sql.Timestamp, formatted by Timestamp.toString in the JVM
> default time zone.
> PostgreSQL reads a zoneless literal compared with a timestamptz column in its
> session TimeZone, which pgjdbc sets to the JVM default time zone at connect.
> When the Spark session time zone differs from the JVM time zone, the pushed
> bounds shift by the offset between the two zones. The query returns different
> rows than the same query with pushdown disabled.
> Repro (JVM zone UTC): a timestamptz table with 8 rows across the UTC/Chicago
> day boundary. Then SET spark.sql.datetime.java8API.enabled=true; SET TIME
> ZONE 'America/Chicago'; SELECT id FROM pg.public.events WHERE ts >= DATE
> '2026-06-26' AND ts < DATE '2026-06-27'.
> Pushed: ("ts" >= '2026-06-26 00:00:00') AND ("ts" < '2026-06-27 00:00:00').
> Actual: ids 1–4 (the UTC day). Expected: ids 3–7 (the Chicago day), which is
> what Spark returns with pushDownPredicate=false or java8API=false.
> Other dialects that use the base compileValue for Instant may be affected as
> well.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]