[ 
https://issues.apache.org/jira/browse/SPARK-60827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-60827:
-----------------------------------
    Labels: correctness pull-request-available  (was: correctness)

> DAY TO SECOND intervals beyond +-106751 days silently wrap around when passed 
> to a pandas UDF or mapInPandas
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-60827
>                 URL: https://issues.apache.org/jira/browse/SPARK-60827
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 4.2.0
>            Reporter: Yaoxuan Wu
>            Priority: Major
>              Labels: correctness, pull-request-available
>
> {code:python}
> import pandas as pd
> from pyspark.sql import SparkSession, functions as F
> from pyspark.sql.functions import pandas_udf
> spark = SparkSession.builder.getOrCreate()
> df = spark.sql("SELECT make_dt_interval(d) AS iv FROM VALUES (106751), 
> (106752), (200000), (-200000) AS t(d)")
> @pandas_udf("long")
> def days(s: pd.Series) -> pd.Series:
>     return s.dt.days
> df.select(F.expr("extract(day FROM iv)").alias("native"), 
> days("iv").alias("pandas")).show()
> # native: 106751, 106752, 200000, -200000
> # pandas: 106751, -106752, -13504, 13503
> def py_days(it):
>     for pdf in it:
>         yield pd.DataFrame({"d": [x.days for x in pdf["iv"]]})
> df.mapInPandas(py_days, "d long").show()   # 106751, -106752, -13504, 13503
> {code}
> {{INTERVAL DAY TO SECOND}} holds up to +-106751991 days (int64 microseconds). 
> When such a value is passed to pandas (pandas UDF input, {{mapInPandas}}), 
> anything beyond +-106751 days, the range of {{timedelta64[ns]}}, silently 
> wraps around. There is no error. A TIMESTAMP outside the {{datetime64[ns]}} 
> range raises an error on the same path instead.
> Cause: the Arrow {{duration[us]}} column is converted with 
> {{to_pandas(coerce_temporal_nanoseconds=True)}}. In 4.2.0 this is 
> {{ArrowArrayToPandasConversion.convert_legacy}} in 
> {{python/pyspark/sql/conversion.py}}; {{DayTimeIntervalType}} is not in 
> {{_prefer_convert_numpy}}, so it takes the legacy path. In 4.0 / 4.1 it is 
> {{ArrowStreamPandasSerializer.arrow_to_pandas}} in 
> {{python/pyspark/sql/pandas/serializers.py}}. pyarrow checks the timestamp 
> cast to ns for overflow, but not the duration cast:
> {code:python}
> import pyarrow as pa
> from datetime import timedelta, datetime
> pa.array([timedelta(days=200000)], 
> pa.duration("us")).to_pandas(coerce_temporal_nanoseconds=True)
> # Timedelta('-13504 days +00:25:26.290448384')
> pa.array([datetime(3000, 1, 1)], 
> pa.timestamp("us")).to_pandas(coerce_temporal_nanoseconds=True)
> # ArrowInvalid: Casting from timestamp[us] to timestamp[ns] would result in 
> out of bounds timestamp
> {code}
> SPARK-60499 mentions the same interval overflow for Arrow {{toPandas()}} on 
> pandas 3. Its suggested fix drops the flag only there, and only for pandas 3. 
> The UDF / {{mapInPandas}} input path above wraps as well, with pandas 2.2.3 
> and 3.0.2.
> Reproduces on 4.0.0, 4.1.1 and 4.2.0 (pyarrow 24.0.0, pandas 2.2.3 and 
> 3.0.2). The same {{coerce_temporal_nanoseconds=True}} line is on master.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to