Hyukjin Kwon created SPARK-61340:
------------------------------------

             Summary: Send datetime literals as TIMESTAMP_NTZ when the session 
prefers it, in the Python client as PySpark does
                 Key: SPARK-61340
                 URL: https://issues.apache.org/jira/browse/SPARK-61340
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Under spark.sql.timestampType=TIMESTAMP_NTZ, the Python drop-in sent every 
datetime literal (lit, Column comparisons, isin, when, the elements of 
lit([...]) and the values of spark.sql args) as a TIMESTAMP of its instant, 
while PySpark 4.3.0's LiteralExpression._infer_type sends a TIMESTAMP_NTZ of 
its wall clock when the active session prefers it (is_timestamp_ntz_preferred, 
read for each datetime literal). With a client zone other than the session's, 
the values differ: with the conf set, a client in Asia/Seoul and a UTC session, 
df.filter(col("t") == datetime(2020, 6, 1, 12)) on a TIMESTAMP_NTZ column 
matched 0 rows (PySpark: 1), and spark.sql("SELECT CAST(:t AS STRING)", 
args={"t": datetime(2020, 6, 1, 12)}) gave 2020-06-01 03:00:00 (PySpark: 
12:00:00). The drop-in's createDataFrame already followed the conf.
The fix sends a TIMESTAMP_NTZ of the wall clock (calendar.timegm of the time 
tuple, as TimestampNTZType.toInternal, also for an aware datetime) when the 
session prefers it, else a TIMESTAMP as before.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to