Yaoxuan Wu created SPARK-60827:
----------------------------------
Summary: DAY TO SECOND intervals beyond +-106751 days silently
wrap around when passed to a pandas UDF or mapInPandas
Key: SPARK-60827
URL: https://issues.apache.org/jira/browse/SPARK-60827
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 4.2.0
Reporter: Yaoxuan Wu
{code:python}
import pandas as pd
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.functions import pandas_udf
spark = SparkSession.builder.getOrCreate()
df = spark.sql("SELECT make_dt_interval(d) AS iv FROM VALUES (106751),
(106752), (200000), (-200000) AS t(d)")
@pandas_udf("long")
def days(s: pd.Series) -> pd.Series:
return s.dt.days
df.select(F.expr("extract(day FROM iv)").alias("native"),
days("iv").alias("pandas")).show()
# native: 106751, 106752, 200000, -200000
# pandas: 106751, -106752, -13504, 13503
def py_days(it):
for pdf in it:
yield pd.DataFrame({"d": [x.days for x in pdf["iv"]]})
df.mapInPandas(py_days, "d long").show() # 106751, -106752, -13504, 13503
{code}
{{INTERVAL DAY TO SECOND}} holds up to +-106751991 days (int64 microseconds).
When such a value is passed to pandas (pandas UDF input, {{mapInPandas}}),
anything beyond +-106751 days, the range of {{timedelta64[ns]}}, silently wraps
around. There is no error. A TIMESTAMP outside the {{datetime64[ns]}} range
raises an error on the same path instead.
Cause: the Arrow {{duration[us]}} column is converted with
{{to_pandas(coerce_temporal_nanoseconds=True)}}. In 4.2.0 this is
{{ArrowArrayToPandasConversion.convert_legacy}} in
{{python/pyspark/sql/conversion.py}}; {{DayTimeIntervalType}} is not in
{{_prefer_convert_numpy}}, so it takes the legacy path. In 4.0 / 4.1 it is
{{ArrowStreamPandasSerializer.arrow_to_pandas}} in
{{python/pyspark/sql/pandas/serializers.py}}. pyarrow checks the timestamp cast
to ns for overflow, but not the duration cast:
{code:python}
import pyarrow as pa
from datetime import timedelta, datetime
pa.array([timedelta(days=200000)],
pa.duration("us")).to_pandas(coerce_temporal_nanoseconds=True)
# Timedelta('-13504 days +00:25:26.290448384')
pa.array([datetime(3000, 1, 1)],
pa.timestamp("us")).to_pandas(coerce_temporal_nanoseconds=True)
# ArrowInvalid: Casting from timestamp[us] to timestamp[ns] would result in out
of bounds timestamp
{code}
SPARK-60499 mentions the same interval overflow for Arrow {{toPandas()}} on
pandas 3. Its suggested fix drops the flag only there, and only for pandas 3.
The UDF / {{mapInPandas}} input path above wraps as well, with pandas 2.2.3 and
3.0.2.
Reproduces on 4.0.0, 4.1.1 and 4.2.0 (pyarrow 24.0.0, pandas 2.2.3 and 3.0.2).
The same {{coerce_temporal_nanoseconds=True}} line is on master.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
