[ 
https://issues.apache.org/jira/browse/SPARK-61360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61360:
-----------------------------------
    Labels: pull-request-available  (was: )

> Fix DateOffset imports with pandas 3.1
> --------------------------------------
>
>                 Key: SPARK-61360
>                 URL: https://issues.apache.org/jira/browse/SPARK-61360
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Pandas API on Spark
>    Affects Versions: 5.0.0
>            Reporter: Ruifeng Zheng
>            Priority: Major
>              Labels: pull-request-available
>
> The scheduled pandas 3 CI job fails before running pandas-on-Spark tests with 
> pandas 3.1.1:
> https://github.com/apache/spark/actions/runs/42413930652/job/128532657680
> ImportError: cannot import name 'DateOffset' from 'pandas.tseries.frequencies'
> Both python/pyspark/pandas/frame.py and python/pyspark/pandas/series.py 
> import DateOffset from pandas.tseries.frequencies. Import it from the public 
> pandas.tseries.offsets namespace instead, while keeping to_offset imported 
> from pandas.tseries.frequencies.
> The public import exists in the minimum-supported pandas 2.2.0. Package 
> import checks passed with pandas 2.3.3 and 3.0.2, and the replacement 
> DateOffset import and date-offset arithmetic passed with pandas 3.1.1. No 
> version-dependent import is needed.
> This fixes the DateOffset import failure only. A full pandas-on-Spark import 
> with pandas 3.1.1 then encounters the separate infer_dtype_from_object 
> compatibility issue.
> Draft fix: https://github.com/apache/spark/pull/60238



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to