[
https://issues.apache.org/jira/browse/SPARK-54890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-54890:
-----------------------------------
Labels: pull-request-available (was: )
> Allow user to enforce timezone match in conversion
> --------------------------------------------------
>
> Key: SPARK-54890
> URL: https://issues.apache.org/jira/browse/SPARK-54890
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.2.0
> Reporter: Tian Gao
> Priority: Major
> Labels: pull-request-available
>
> Our timestamp conversion is messed up and it's impossible to fix it without
> seriously breaking backward compatibility. The fundamental issue is that we
> mix timezone-aware and timezone-naive timestamps together. Python has
> implicit rules for naive timestamps and we have a session local timezone
> config. It's not possible to even define a rule that all code can follow.
> The only way to resolve this once and for good is to enforce the conversion.
> Only allow aware timestamps to convert from/to TimestampType and naive
> timestamps to convert from/to TimestampNTZType.
> We can't make this default, not for now, but at least we can give user a mode
> that can be reasoned.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
