[
https://issues.apache.org/jira/browse/SPARK-60081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18145024#comment-18145024
]
Yaoxuan Wu commented on SPARK-60081:
------------------------------------
pandas UDFs are also affected, through a different code path and in the
opposite direction: the *first* occurrence of the repeated hour comes back one
hour late.
{code:python}
import os, time
os.environ["TZ"] = "America/Los_Angeles"
time.tzset()
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.functions import pandas_udf
from pyspark.sql.types import TimestampType
spark = SparkSession.builder.config("spark.sql.session.timeZone",
"America/Los_Angeles").getOrCreate()
identity = pandas_udf(lambda s: s, TimestampType())
# 2021-11-07 08:30 UTC = 01:30 PDT, the first 01:30 of the night the clocks go
back (epoch 1636273800)
df = spark.sql("SELECT timestamp'2021-11-07 08:30:00Z' AS t")
print(df.select(F.unix_timestamp(identity("t"))).first()[0])
# expected: 1636273800
# actual: 1636277400 (one hour late: the second 01:30, PST)
# 2021-11-07 09:30 UTC = 01:30 PST, the second 01:30 (epoch 1636277400)
df = spark.sql("SELECT timestamp'2021-11-07 09:30:00Z' AS t")
print(df.select(F.unix_timestamp(identity("t"))).first()[0])
# 1636277400 (correct)
{code}
Same on 4.1.1 and 4.2.0. pandas UDF input is converted to tz-naive wall times
in the session time zone, and the output is converted back with
{{_check_series_convert_timestamps_internal}}
({{pyspark/sql/pandas/types.py}}), which calls {{s.dt.tz_localize(tz,
ambiguous=False)}}: a wall time that occurs twice always becomes the
standard-time (later) instant. The comment there documents this choice; the
result is still that an identity pandas UDF changes the value. The same applies
to the LMT -> standard time switch: with session time zone America/New_York,
{{TIMESTAMP'1883-11-18 12:00:00'}} comes back 238 s (3 min 58 s) later.
> Python UDFs get the wrong value for TIMESTAMPs in the repeated DST hour
> -----------------------------------------------------------------------
>
> Key: SPARK-60081
> URL: https://issues.apache.org/jira/browse/SPARK-60081
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 4.2.0, 4.1.1, 5.0.0
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: correctness, pull-request-available
>
> When clocks go back for DST, one local hour happens twice. For a TIMESTAMP in
> the second occurrence of that hour, a Python UDF gets a value that is one
> hour too early. Even an identity UDF {{lambda x: x}} returns a different
> value.
>
> {code:java}
> import os, time
> os.environ["TZ"] = "America/Los_Angeles"
> time.tzset()
> from pyspark.sql import SparkSession, functions as F
> from pyspark.sql.types import LongType, TimestampType
> spark = SparkSession.builder.config("spark.sql.session.timeZone",
> "America/Los_Angeles").getOrCreate()
> # 2021-11-07 09:30 UTC = 01:30 PST, the second 01:30 of the night the clocks
> go back.
> # Its epoch second is 1636277400.
> df = spark.sql("SELECT timestamp'2021-11-07 09:30:00Z' AS t")# 1) The value a
> Python UDF receives
> udf_input = F.udf(lambda x: int(x.timestamp()), LongType())
> df.select(udf_input("t")).first()[0]
> # expected: 1636277400
> # Spark 4.2.0: 1636273800 (one hour early)# 2) An identity UDF should return
> its input unchanged
> identity = F.udf(lambda x: x, TimestampType())
> df.select(F.unix_timestamp(identity("t"))).first()[0]
> # expected: 1636277400
> # Spark 4.2.0: 1636273800 (one hour early)
> {code}
>
> The results above use 4.2.0's default, an Arrow-optimized UDF. Both paths are
> wrong on 4.1.1, 4.2.0 and master:
> || ||What the UDF receives||What an identity UDF returns||
> |Arrow UDF ({{{}useArrow=True{}}}, default in 4.2)|1 hour early|1 hour early|
> |Pickled UDF ({{{}useArrow=False{}}}, default in 4.1)|correct|1 hour early|
>
> The first 01:30 (PDT, {{{}08:30Z{}}}) and all other times are correct.
> *Cause*
> Python marks the second occurrence with {{{}datetime.fold = 1{}}}, and
> PySpark drops {{fold}} in two places:
> # Arrow input conversion ({{{}pyspark/sql/conversion.py{}}}):
> {{value.astimezone().replace(tzinfo=None)}} always gives {{{}fold=0{}}}. A
> fix: {{{}datetime.datetime.fromtimestamp(value.timestamp()){}}}.
> # {{TimestampType.toInternal}} ({{{}pyspark/sql/types.py{}}}, used for UDF
> results on the pickled path and by {{{}createDataFrame{}}}):
> {{time.mktime(dt.timetuple())}} ignores {{{}fold{}}}. A fix:
> {{{}dt.timestamp(){}}}.
> Since pyspark-4.2 makes Arrow UDFs the default, every Python UDF now reads
> these timestamps wrongly by default, while the pickled path reads them
> correctly.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]