Tian Gao created SPARK-54890:
--------------------------------
Summary: Allow user to enforce timezone match in conversion
Key: SPARK-54890
URL: https://issues.apache.org/jira/browse/SPARK-54890
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.2.0
Reporter: Tian Gao
Our timestamp conversion is messed up and it's impossible to fix it without
seriously breaking backward compatibility. The fundamental issue is that we mix
timezone-aware and timezone-naive timestamps together. Python has implicit
rules for naive timestamps and we have a session local timezone config. It's
not possible to even define a rule that all code can follow.
The only way to resolve this once and for good is to enforce the conversion.
Only allow aware timestamps to convert from/to TimestampType and naive
timestamps to convert from/to TimestampNTZType.
We can't make this default, not for now, but at least we can give user a mode
that can be reasoned.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
