[
https://issues.apache.org/jira/browse/SPARK-60827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60827:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> DAY TO SECOND intervals beyond +-106751 days silently wrap around when passed
> to a pandas UDF or mapInPandas
> ------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60827
> URL: https://issues.apache.org/jira/browse/SPARK-60827
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 4.2.0
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: correctness, pull-request-available
>
> {code:python}
> import pandas as pd
> from pyspark.sql import SparkSession, functions as F
> from pyspark.sql.functions import pandas_udf
> spark = SparkSession.builder.getOrCreate()
> df = spark.sql("SELECT make_dt_interval(d) AS iv FROM VALUES (106751),
> (106752), (200000), (-200000) AS t(d)")
> @pandas_udf("long")
> def days(s: pd.Series) -> pd.Series:
> return s.dt.days
> df.select(F.expr("extract(day FROM iv)").alias("native"),
> days("iv").alias("pandas")).show()
> # native: 106751, 106752, 200000, -200000
> # pandas: 106751, -106752, -13504, 13503
> def py_days(it):
> for pdf in it:
> yield pd.DataFrame({"d": [x.days for x in pdf["iv"]]})
> df.mapInPandas(py_days, "d long").show() # 106751, -106752, -13504, 13503
> {code}
> {{INTERVAL DAY TO SECOND}} holds up to +-106751991 days (int64 microseconds).
> When such a value is passed to pandas (pandas UDF input, {{mapInPandas}}),
> anything beyond +-106751 days, the range of {{timedelta64[ns]}}, silently
> wraps around. There is no error. A TIMESTAMP outside the {{datetime64[ns]}}
> range raises an error on the same path instead.
> Cause: the Arrow {{duration[us]}} column is converted with
> {{to_pandas(coerce_temporal_nanoseconds=True)}}. In 4.2.0 this is
> {{ArrowArrayToPandasConversion.convert_legacy}} in
> {{python/pyspark/sql/conversion.py}}; {{DayTimeIntervalType}} is not in
> {{_prefer_convert_numpy}}, so it takes the legacy path. In 4.0 / 4.1 it is
> {{ArrowStreamPandasSerializer.arrow_to_pandas}} in
> {{python/pyspark/sql/pandas/serializers.py}}. pyarrow checks the timestamp
> cast to ns for overflow, but not the duration cast:
> {code:python}
> import pyarrow as pa
> from datetime import timedelta, datetime
> pa.array([timedelta(days=200000)],
> pa.duration("us")).to_pandas(coerce_temporal_nanoseconds=True)
> # Timedelta('-13504 days +00:25:26.290448384')
> pa.array([datetime(3000, 1, 1)],
> pa.timestamp("us")).to_pandas(coerce_temporal_nanoseconds=True)
> # ArrowInvalid: Casting from timestamp[us] to timestamp[ns] would result in
> out of bounds timestamp
> {code}
> SPARK-60499 mentions the same interval overflow for Arrow {{toPandas()}} on
> pandas 3. Its suggested fix drops the flag only there, and only for pandas 3.
> The UDF / {{mapInPandas}} input path above wraps as well, with pandas 2.2.3
> and 3.0.2.
> Reproduces on 4.0.0, 4.1.1 and 4.2.0 (pyarrow 24.0.0, pandas 2.2.3 and
> 3.0.2). The same {{coerce_temporal_nanoseconds=True}} line is on master.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]