[
https://issues.apache.org/jira/browse/SPARK-60523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60523:
-----------------------------------
Labels: pull-request-available (was: )
> pandas-on-Spark np.gcd with a left-hand scalar raises AttributeError
> --------------------------------------------------------------------
>
> Key: SPARK-60523
> URL: https://issues.apache.org/jira/browse/SPARK-60523
> Project: Spark
> Issue Type: Bug
> Components: Pandas API on Spark, PySpark
> Affects Versions: 4.2.0
> Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
> Python: 3.11.17
> pyspark 4.2.0
> findspark 2.0.1
> pandas 3.0.6
> numpy 2.4.6
> Reporter: asddfl
> Priority: Critical
> Labels: pull-request-available
>
> When using the pandas API on Spark (pyspark.pandas), calling np.gcd with a
> Python integer as its left operand and an integer Series as its right operand
> raises AttributeError. The equivalent Pandas/NumPy call succeeds for the same
> data.
> The example contains one integer value and no missing values. The failure
> occurs in NumPy ufunc dispatch before a result can be produced.
> {code:python}
> # Minimized from err_bug1.py: A left-hand scalar is incorrectly treated as a
> distributed Series.
> import pandas as pd
> import numpy as np
> from pyspark.sql import SparkSession
> import pyspark.pandas as ps
> t0 = pd.DataFrame({'c0': [1]})
> print("Pandas:", flush=True)
> result = np.gcd(2, t0['c0'])
> print(result, flush=True)
> spark = (
> SparkSession.builder
> .master("local[2]")
> .config("spark.sql.ansi.enabled", "false")
> .config("spark.sql.shuffle.partitions", "2")
> .getOrCreate()
> )
> t0 = ps.DataFrame({'c0': [1]})
> print("PySpark Pandas:", flush=True)
> result = np.gcd(2, t0['c0'])
> print(result, flush=True)
> spark.stop()
> {code}
> {code:bash}
> Pandas:
> 0 1
> Name: c0, dtype: int64
> PySpark Pandas:
> Traceback (most recent call last):
> File "/test.py", line 25, in <module>
> result = np.gcd(2, t0['c0'])
> ^^^^^^^^^^^^^^^^^^^
> File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py",
> line 555, in __array_ufunc__
> result = numpy_compat.maybe_dispatch_ufunc_to_spark_func(
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/numpy_compat.py",
> line 226, in maybe_dispatch_ufunc_to_spark_func
> return column_op(convert_arguments)(*inputs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py",
> line 238, in wrapper
> if all(not should_alignment_for_column_op(self, col) for col in cols):
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py",
> line 238, in <genexpr>
> if all(not should_alignment_for_column_op(self, col) for col in cols):
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py",
> line 82, in should_alignment_for_column_op
> return self._internal.spark_frame is not other._internal.spark_frame
> ^^^^^^^^^^^^^^
> AttributeError: 'int' object has no attribute '_internal'
> {code}
> *Expected behavior*
> np.gcd(2, a Series containing 1) should return an integer Series containing
> 1, matching Pandas. A scalar left operand should not be treated as a
> distributed Series.
> *Actual behavior*
> The pandas-on-Spark call raises AttributeError: 'int' object has no attribute
> '_internal'. The traceback passes through maybe_dispatch_ufunc_to_spark_func
> and column_op, then should_alignment_for_column_op accesses self._internal
> when self is the Python integer.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
