[
https://issues.apache.org/jira/browse/SPARK-61648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61648:
-----------------------------------
Labels: pull-request-available (was: )
> Arrow Python UDF fails on naive TimestampType results on 9999-12-31 and
> 0001-01-01
> ----------------------------------------------------------------------------------
>
> Key: SPARK-61648
> URL: https://issues.apache.org/jira/browse/SPARK-61648
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 4.1.0, 4.0.0, 4.2.0, 4.3.0, 5.0.0
> Reporter: Tim Meehan
> Priority: Major
> Labels: pull-request-available
>
> h3. Symptom
> An Arrow-optimized Python UDF ({{useArrow=True}}, the default for {{@udf}}
> since 4.2.0) with a {{TIMESTAMP}} return type fails its task when it returns
> a naive {{datetime}} on 9999-12-31, at any time of day, or on 0001-01-01:
> {noformat}
> ValueError: year 10000 is out of range
> ValueError: year 0 is out of range
> {noformat}
> 9999-12-30 23:59:59 converts. The pickled path ({{useArrow=False}}) accepts
> the same values.
> The failing code is {{convert_timestamp}} in
> {{LocalDataToArrowConversion._create_converter}}
> ({{python/pyspark/sql/conversion.py}}), which returns
> {{value.astimezone(datetime.timezone.utc)}} for {{TimestampType}} (and for
> {{TimestampLTZNanosType}}, new in 4.4.0). The same converter also converts
> local rows in Spark Connect {{createDataFrame}}, the rows of Arrow-optimized
> Python UDTFs and the rows of Python data source readers, so those fail on the
> same values.
> h3. Cause
> For a naive value, {{datetime.astimezone}} first works out the local time
> zone. CPython's algorithm ({{Lib/_pydatetime.py}}: {{astimezone}} ->
> {{_local_timezone}} -> {{_mktime}}; {{Modules/_datetimemodule.c}} does the
> same) probes the local offset one day away ({{u1 +/- 24*3600}}) to detect
> folds and gaps, and turns each probe back into a {{datetime}}. On the first
> and the last day that {{datetime}} supports, a probe falls outside its range
> and raises.
> h3. Reproduce
> No JVM is needed:
> {code:python}
> import datetime
> from pyspark.sql.conversion import LocalDataToArrowConversion
> from pyspark.sql.types import TimestampType
> convert = LocalDataToArrowConversion._create_converter(
> TimestampType(), none_on_identity=True,
> int_to_decimal_coercion_enabled=False
> )
> for value in (
> datetime.datetime(9999, 12, 31, 23, 59, 59, 999999),
> datetime.datetime(9999, 12, 31),
> datetime.datetime(1, 1, 1),
> datetime.datetime(9999, 12, 30, 23, 59, 59),
> ):
> print(repr(value), TimestampType().toInternal(value))
> try:
> print(" arrow:", repr(convert(value)))
> except ValueError as error:
> print(" arrow: ValueError:", error)
> {code}
> On master (c094224fb6c), with Python 3.12.3 and PyArrow 26.0.1 on Linux:
> {noformat}
> TZ=UTC
> datetime.datetime(9999, 12, 31, 23, 59, 59, 999999) 253402300799999999
> arrow: ValueError: year 10000 is out of range
> datetime.datetime(9999, 12, 31, 0, 0) 253402214400000000
> arrow: ValueError: year 10000 is out of range
> datetime.datetime(1, 1, 1, 0, 0) -62135596800000000
> arrow: ValueError: year 0 is out of range
> datetime.datetime(9999, 12, 30, 23, 59, 59) 253402214399000000
> arrow: datetime.datetime(9999, 12, 30, 23, 59, 59,
> tzinfo=datetime.timezone.utc)
> TZ=America/Los_Angeles
> datetime.datetime(9999, 12, 31, 23, 59, 59, 999999) 253402329599999999
> arrow: ValueError: year 10000 is out of range
> datetime.datetime(9999, 12, 31, 0, 0) 253402243200000000
> arrow: ValueError: year 10000 is out of range
> datetime.datetime(1, 1, 1, 0, 0) -62135568422000000
> arrow: ValueError: year 0 is out of range
> datetime.datetime(9999, 12, 30, 23, 59, 59) 253402243199000000
> arrow: datetime.datetime(9999, 12, 31, 7, 59, 59,
> tzinfo=datetime.timezone.utc)
> TZ=Asia/Tokyo
> datetime.datetime(9999, 12, 31, 23, 59, 59, 999999) 253402268399999999
> arrow: ValueError: year 10000 is out of range
> datetime.datetime(9999, 12, 31, 0, 0) 253402182000000000
> arrow: ValueError: year 10000 is out of range
> datetime.datetime(1, 1, 1, 0, 0) -62135630339000000
> arrow: ValueError: year 0 is out of range
> datetime.datetime(9999, 12, 30, 23, 59, 59) 253402181999000000
> arrow: datetime.datetime(9999, 12, 30, 14, 59, 59,
> tzinfo=datetime.timezone.utc)
> {noformat}
> The number on each value's line is what the pickled path returns for it.
> h3. The pickled path
> A pickled UDF applies {{TimestampType.toInternal}} to each result. For a
> naive value that is {{time.mktime(dt.timetuple())}}, the local-time
> interpretation, and the JVM takes the integer as it is
> ({{EvaluatePython.makeFromJava}}). So with {{useArrow=False}} these results
> succeed, with the values above.
> h3. West of UTC
> Computing the instant as {{toInternal}} does and returning a UTC-aware
> {{datetime}} would not be enough. West of UTC, the instant of a late
> 9999-12-31 local time is in year 10000, past {{datetime.max}}: under
> America/Los_Angeles, {{toInternal}} returns 253402329599999999 for 9999-12-31
> 23:59:59.999999, which is 10000-01-01 07:59:59.999999 UTC. East of UTC the
> same happens at the other end: the instant of 0001-01-01 00:00 under
> Asia/Tokyo (local mean time, UTC+09:18:59) is in year 0. No UTC {{datetime}}
> can hold these instants. The converter's output only goes to PyArrow as
> {{timestamp[us, tz=UTC]}}, which can hold them, and PyArrow converts an aware
> {{datetime}} by its own UTC offset.
> h3. Versions
> Read in {{python/pyspark/sql/conversion.py}} at each tag: the same
> {{astimezone}} line is in v4.0.0, v4.0.5, v4.1.0, v4.1.4, v4.2.0, v4.2.1,
> v4.3.0 and master. Arrow-optimized UDF results go through it from 4.1.0 when
> {{useArrow=True}}, and by default from 4.2.0
> ({{spark.sql.execution.pythonUDF.arrow.enabled}} defaults to true). In 4.0.x,
> Arrow UDF results do not use this converter; there, Connect
> {{createDataFrame}} and Python data source reads do.
> SPARK-54421 reports the same line raising {{OSError}} on Windows over a much
> wider range of dates.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]