Narendra Singh created SPARK-61562:
--------------------------------------

             Summary: Spark Connect Column operators fail with NumPy scalar 
operands
                 Key: SPARK-61562
                 URL: https://issues.apache.org/jira/browse/SPARK-61562
             Project: Spark
          Issue Type: Bug
          Components: Connect, PySpark
    Affects Versions: 4.3.0, 5.0.0
            Reporter: Narendra Singh


In Spark Connect, Column operators and methods raise an AttributeError when the 
other operand is a NumPy scalar, while classic PySpark accepts it.

{code:python}
import numpy as np
from pyspark.sql import functions as F

df = spark.range(3).select(F.col("id").alias("x"))
df.select(F.col("x") + np.int64(1)).collect()
# Classic: [Row((x + 1)=1), Row((x + 1)=2), Row((x + 1)=3)]
# Connect: AttributeError: 'numpy.int64' object has no attribute '_expr'
{code}

The same happens for np.int32, np.float32 and np.bool_ operands of arithmetic 
and comparison operators, and for Column.between. np.float64 works only because 
it is a subclass of Python float. F.lit(np.int64(1)) already works in Connect.

Cause: _bin_op in python/pyspark/sql/connect/column.py turns only built-in 
Python types (bool, int, float, str, date/time, Decimal) into literals and 
treats anything else as a Column, although LiteralExpression already supports 
NumPy scalars (np.generic).

Proposed fix: also send NumPy scalars to LiteralExpression in _bin_op and the 
other Column methods that take literal values, and add tests that run on both 
classic and Connect.

Related: SPARK-61473 records this AttributeError behavior of PySpark's _bin_op 
for the Spark Connect Rust client. Classic PySpark accepts NumPy scalars here, 
so this proposes accepting them in PySpark Connect too.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to