Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/18664
It looks like "SESSION_LOCAL_TIMEZONE" is not respected in most of the
pyspark functionality.
I think `df.collect()` and `df.toPandas` can be fixed to respect
SESSION_LOCAL_TIMEZONE.
`TimestampType().toInternal(dt)` is tricky because it doesn't have a
reference to the SQLConf object, we should maybe deprecate this method?
I think we can also create separate Jira to address these, this PR can fix
just the Arrow path. @BryanCutler and @ueshin do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
