Tim Meehan created SPARK-61648:
----------------------------------

             Summary: Arrow Python UDF fails on naive TimestampType results on 
9999-12-31 and 0001-01-01
                 Key: SPARK-61648
                 URL: https://issues.apache.org/jira/browse/SPARK-61648
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0, 4.0.0
            Reporter: Tim Meehan


h3. Symptom

An Arrow-optimized Python UDF ({{useArrow=True}}, the default for {{@udf}} 
since 4.2.0) with a {{TIMESTAMP}} return type fails its task when it returns a 
naive {{datetime}} on 9999-12-31, at any time of day, or on 0001-01-01:

{noformat}
ValueError: year 10000 is out of range
ValueError: year 0 is out of range
{noformat}

9999-12-30 23:59:59 converts. The pickled path ({{useArrow=False}}) accepts the 
same values.

The failing code is {{convert_timestamp}} in 
{{LocalDataToArrowConversion._create_converter}} 
({{python/pyspark/sql/conversion.py}}), which returns 
{{value.astimezone(datetime.timezone.utc)}} for {{TimestampType}} (and for 
{{TimestampLTZNanosType}}, new in 4.4.0). The same converter also converts 
local rows in Spark Connect {{createDataFrame}}, the rows of Arrow-optimized 
Python UDTFs and the rows of Python data source readers, so those fail on the 
same values.

h3. Cause

For a naive value, {{datetime.astimezone}} first works out the local time zone. 
CPython's algorithm ({{Lib/_pydatetime.py}}: {{astimezone}} -> 
{{_local_timezone}} -> {{_mktime}}; {{Modules/_datetimemodule.c}} does the 
same) probes the local offset one day away ({{u1 +/- 24*3600}}) to detect folds 
and gaps, and turns each probe back into a {{datetime}}. On the first and the 
last day that {{datetime}} supports, a probe falls outside its range and raises.

h3. Reproduce

No JVM is needed:

{code:python}
import datetime
from pyspark.sql.conversion import LocalDataToArrowConversion
from pyspark.sql.types import TimestampType

convert = LocalDataToArrowConversion._create_converter(
    TimestampType(), none_on_identity=True, 
int_to_decimal_coercion_enabled=False
)
for value in (
    datetime.datetime(9999, 12, 31, 23, 59, 59, 999999),
    datetime.datetime(9999, 12, 31),
    datetime.datetime(1, 1, 1),
    datetime.datetime(9999, 12, 30, 23, 59, 59),
):
    print(repr(value), TimestampType().toInternal(value))
    try:
        print("  arrow:", repr(convert(value)))
    except ValueError as error:
        print("  arrow: ValueError:", error)
{code}

On master (c094224fb6c), with Python 3.12.3 and PyArrow 26.0.1 on Linux:

{noformat}
TZ=UTC
datetime.datetime(9999, 12, 31, 23, 59, 59, 999999) 253402300799999999
  arrow: ValueError: year 10000 is out of range
datetime.datetime(9999, 12, 31, 0, 0) 253402214400000000
  arrow: ValueError: year 10000 is out of range
datetime.datetime(1, 1, 1, 0, 0) -62135596800000000
  arrow: ValueError: year 0 is out of range
datetime.datetime(9999, 12, 30, 23, 59, 59) 253402214399000000
  arrow: datetime.datetime(9999, 12, 30, 23, 59, 59, 
tzinfo=datetime.timezone.utc)
TZ=America/Los_Angeles
datetime.datetime(9999, 12, 31, 23, 59, 59, 999999) 253402329599999999
  arrow: ValueError: year 10000 is out of range
datetime.datetime(9999, 12, 31, 0, 0) 253402243200000000
  arrow: ValueError: year 10000 is out of range
datetime.datetime(1, 1, 1, 0, 0) -62135568422000000
  arrow: ValueError: year 0 is out of range
datetime.datetime(9999, 12, 30, 23, 59, 59) 253402243199000000
  arrow: datetime.datetime(9999, 12, 31, 7, 59, 59, 
tzinfo=datetime.timezone.utc)
TZ=Asia/Tokyo
datetime.datetime(9999, 12, 31, 23, 59, 59, 999999) 253402268399999999
  arrow: ValueError: year 10000 is out of range
datetime.datetime(9999, 12, 31, 0, 0) 253402182000000000
  arrow: ValueError: year 10000 is out of range
datetime.datetime(1, 1, 1, 0, 0) -62135630339000000
  arrow: ValueError: year 0 is out of range
datetime.datetime(9999, 12, 30, 23, 59, 59) 253402181999000000
  arrow: datetime.datetime(9999, 12, 30, 14, 59, 59, 
tzinfo=datetime.timezone.utc)
{noformat}

The number on each value's line is what the pickled path returns for it.

h3. The pickled path

A pickled UDF applies {{TimestampType.toInternal}} to each result. For a naive 
value that is {{time.mktime(dt.timetuple())}}, the local-time interpretation, 
and the JVM takes the integer as it is ({{EvaluatePython.makeFromJava}}). So 
with {{useArrow=False}} these results succeed, with the values above.

h3. West of UTC

Computing the instant as {{toInternal}} does and returning a UTC-aware 
{{datetime}} would not be enough. West of UTC, the instant of a late 9999-12-31 
local time is in year 10000, past {{datetime.max}}: under America/Los_Angeles, 
{{toInternal}} returns 253402329599999999 for 9999-12-31 23:59:59.999999, which 
is 10000-01-01 07:59:59.999999 UTC. East of UTC the same happens at the other 
end: the instant of 0001-01-01 00:00 under Asia/Tokyo (local mean time, 
UTC+09:18:59) is in year 0. No UTC {{datetime}} can hold these instants. The 
converter's output only goes to PyArrow as {{timestamp[us, tz=UTC]}}, which can 
hold them, and PyArrow converts an aware {{datetime}} by its own UTC offset.

h3. Versions

Read in {{python/pyspark/sql/conversion.py}} at each tag: the same 
{{astimezone}} line is in v4.0.0, v4.0.5, v4.1.0, v4.1.4, v4.2.0, v4.2.1, 
v4.3.0 and master. Arrow-optimized UDF results go through it from 4.1.0 when 
{{useArrow=True}}, and by default from 4.2.0 
({{spark.sql.execution.pythonUDF.arrow.enabled}} defaults to true). In 4.0.x, 
Arrow UDF results do not use this converter; there, Connect {{createDataFrame}} 
and Python data source reads do.

SPARK-54421 reports the same line raising {{OSError}} on Windows over a much 
wider range of dates.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to