Aleksandar Trajković created SPARK-61543:
--------------------------------------------
Summary: JDBC pushdown renders java.time.Instant filter literals
in the session time zone, so PostgreSQL returns wrong rows when
spark.sql.session.timeZone differs from the JVM time zone
Key: SPARK-61543
URL: https://issues.apache.org/jira/browse/SPARK-61543
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.4.0
Reporter: Aleksandar Trajković
JDBC pushdown renders java.time.Instant filter literals in the session time
zone, so PostgreSQL returns wrong rows when spark.sql.session.timeZone differs
from the JVM time zone
When spark.sql.datetime.java8API.enabled=true (the Thrift server always enables
it), TIMESTAMP literals in pushed-down filters reach JdbcDialect.compileValue
as java.time.Instant. compileValue formats them as a zoneless string in
spark.sql.session.timeZone. With java8API=false the literal is a
java.sql.Timestamp, formatted by Timestamp.toString in the JVM default time
zone.
PostgreSQL reads a zoneless literal compared with a timestamptz column in its
session TimeZone, which pgjdbc sets to the JVM default time zone at connect.
When the Spark session time zone differs from the JVM time zone, the pushed
bounds shift by the offset between the two zones. The query returns different
rows than the same query with pushdown disabled.
Repro (JVM zone UTC): a timestamptz table with 8 rows across the UTC/Chicago
day boundary. Then SET spark.sql.datetime.java8API.enabled=true; SET TIME ZONE
'America/Chicago'; SELECT id FROM pg.public.events WHERE ts >= DATE
'2026-06-26' AND ts < DATE '2026-06-27'.
Pushed: ("ts" >= '2026-06-26 00:00:00') AND ("ts" < '2026-06-27 00:00:00').
Actual: ids 1–4 (the UTC day). Expected: ids 3–7 (the Chicago day), which is
what Spark returns with pushDownPredicate=false or java8API=false.
Other dialects that use the base compileValue for Instant may be affected as
well.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]