Hyukjin Kwon created SPARK-61340:
------------------------------------
Summary: Send datetime literals as TIMESTAMP_NTZ when the session
prefers it, in the Python client as PySpark does
Key: SPARK-61340
URL: https://issues.apache.org/jira/browse/SPARK-61340
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Under spark.sql.timestampType=TIMESTAMP_NTZ, the Python drop-in sent every
datetime literal (lit, Column comparisons, isin, when, the elements of
lit([...]) and the values of spark.sql args) as a TIMESTAMP of its instant,
while PySpark 4.3.0's LiteralExpression._infer_type sends a TIMESTAMP_NTZ of
its wall clock when the active session prefers it (is_timestamp_ntz_preferred,
read for each datetime literal). With a client zone other than the session's,
the values differ: with the conf set, a client in Asia/Seoul and a UTC session,
df.filter(col("t") == datetime(2020, 6, 1, 12)) on a TIMESTAMP_NTZ column
matched 0 rows (PySpark: 1), and spark.sql("SELECT CAST(:t AS STRING)",
args={"t": datetime(2020, 6, 1, 12)}) gave 2020-06-01 03:00:00 (PySpark:
12:00:00). The drop-in's createDataFrame already followed the conf.
The fix sends a TIMESTAMP_NTZ of the wall clock (calendar.timegm of the time
tuple, as TimestampNTZType.toInternal, also for an aware datetime) when the
session prefers it, else a TIMESTAMP as before.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]