Tim Meehan created SPARK-61648:
----------------------------------
Summary: Arrow Python UDF fails on naive TimestampType results on
9999-12-31 and 0001-01-01
Key: SPARK-61648
URL: https://issues.apache.org/jira/browse/SPARK-61648
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0, 4.0.0
Reporter: Tim Meehan
h3. Symptom
An Arrow-optimized Python UDF ({{useArrow=True}}, the default for {{@udf}}
since 4.2.0) with a {{TIMESTAMP}} return type fails its task when it returns a
naive {{datetime}} on 9999-12-31, at any time of day, or on 0001-01-01:
{noformat}
ValueError: year 10000 is out of range
ValueError: year 0 is out of range
{noformat}
9999-12-30 23:59:59 converts. The pickled path ({{useArrow=False}}) accepts the
same values.
The failing code is {{convert_timestamp}} in
{{LocalDataToArrowConversion._create_converter}}
({{python/pyspark/sql/conversion.py}}), which returns
{{value.astimezone(datetime.timezone.utc)}} for {{TimestampType}} (and for
{{TimestampLTZNanosType}}, new in 4.4.0). The same converter also converts
local rows in Spark Connect {{createDataFrame}}, the rows of Arrow-optimized
Python UDTFs and the rows of Python data source readers, so those fail on the
same values.
h3. Cause
For a naive value, {{datetime.astimezone}} first works out the local time zone.
CPython's algorithm ({{Lib/_pydatetime.py}}: {{astimezone}} ->
{{_local_timezone}} -> {{_mktime}}; {{Modules/_datetimemodule.c}} does the
same) probes the local offset one day away ({{u1 +/- 24*3600}}) to detect folds
and gaps, and turns each probe back into a {{datetime}}. On the first and the
last day that {{datetime}} supports, a probe falls outside its range and raises.
h3. Reproduce
No JVM is needed:
{code:python}
import datetime
from pyspark.sql.conversion import LocalDataToArrowConversion
from pyspark.sql.types import TimestampType
convert = LocalDataToArrowConversion._create_converter(
TimestampType(), none_on_identity=True,
int_to_decimal_coercion_enabled=False
)
for value in (
datetime.datetime(9999, 12, 31, 23, 59, 59, 999999),
datetime.datetime(9999, 12, 31),
datetime.datetime(1, 1, 1),
datetime.datetime(9999, 12, 30, 23, 59, 59),
):
print(repr(value), TimestampType().toInternal(value))
try:
print(" arrow:", repr(convert(value)))
except ValueError as error:
print(" arrow: ValueError:", error)
{code}
On master (c094224fb6c), with Python 3.12.3 and PyArrow 26.0.1 on Linux:
{noformat}
TZ=UTC
datetime.datetime(9999, 12, 31, 23, 59, 59, 999999) 253402300799999999
arrow: ValueError: year 10000 is out of range
datetime.datetime(9999, 12, 31, 0, 0) 253402214400000000
arrow: ValueError: year 10000 is out of range
datetime.datetime(1, 1, 1, 0, 0) -62135596800000000
arrow: ValueError: year 0 is out of range
datetime.datetime(9999, 12, 30, 23, 59, 59) 253402214399000000
arrow: datetime.datetime(9999, 12, 30, 23, 59, 59,
tzinfo=datetime.timezone.utc)
TZ=America/Los_Angeles
datetime.datetime(9999, 12, 31, 23, 59, 59, 999999) 253402329599999999
arrow: ValueError: year 10000 is out of range
datetime.datetime(9999, 12, 31, 0, 0) 253402243200000000
arrow: ValueError: year 10000 is out of range
datetime.datetime(1, 1, 1, 0, 0) -62135568422000000
arrow: ValueError: year 0 is out of range
datetime.datetime(9999, 12, 30, 23, 59, 59) 253402243199000000
arrow: datetime.datetime(9999, 12, 31, 7, 59, 59,
tzinfo=datetime.timezone.utc)
TZ=Asia/Tokyo
datetime.datetime(9999, 12, 31, 23, 59, 59, 999999) 253402268399999999
arrow: ValueError: year 10000 is out of range
datetime.datetime(9999, 12, 31, 0, 0) 253402182000000000
arrow: ValueError: year 10000 is out of range
datetime.datetime(1, 1, 1, 0, 0) -62135630339000000
arrow: ValueError: year 0 is out of range
datetime.datetime(9999, 12, 30, 23, 59, 59) 253402181999000000
arrow: datetime.datetime(9999, 12, 30, 14, 59, 59,
tzinfo=datetime.timezone.utc)
{noformat}
The number on each value's line is what the pickled path returns for it.
h3. The pickled path
A pickled UDF applies {{TimestampType.toInternal}} to each result. For a naive
value that is {{time.mktime(dt.timetuple())}}, the local-time interpretation,
and the JVM takes the integer as it is ({{EvaluatePython.makeFromJava}}). So
with {{useArrow=False}} these results succeed, with the values above.
h3. West of UTC
Computing the instant as {{toInternal}} does and returning a UTC-aware
{{datetime}} would not be enough. West of UTC, the instant of a late 9999-12-31
local time is in year 10000, past {{datetime.max}}: under America/Los_Angeles,
{{toInternal}} returns 253402329599999999 for 9999-12-31 23:59:59.999999, which
is 10000-01-01 07:59:59.999999 UTC. East of UTC the same happens at the other
end: the instant of 0001-01-01 00:00 under Asia/Tokyo (local mean time,
UTC+09:18:59) is in year 0. No UTC {{datetime}} can hold these instants. The
converter's output only goes to PyArrow as {{timestamp[us, tz=UTC]}}, which can
hold them, and PyArrow converts an aware {{datetime}} by its own UTC offset.
h3. Versions
Read in {{python/pyspark/sql/conversion.py}} at each tag: the same
{{astimezone}} line is in v4.0.0, v4.0.5, v4.1.0, v4.1.4, v4.2.0, v4.2.1,
v4.3.0 and master. Arrow-optimized UDF results go through it from 4.1.0 when
{{useArrow=True}}, and by default from 4.2.0
({{spark.sql.execution.pythonUDF.arrow.enabled}} defaults to true). In 4.0.x,
Arrow UDF results do not use this converter; there, Connect {{createDataFrame}}
and Python data source reads do.
SPARK-54421 reports the same line raising {{OSError}} on Windows over a much
wider range of dates.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]