[ 
https://issues.apache.org/jira/browse/SPARK-61562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144254#comment-18144254
 ] 

Narendra Singh commented on SPARK-61562:
----------------------------------------

I've opened a PR that sends NumPy scalars to LiteralExpression in the Connect 
Column operators, matching classic PySpark.

One design point for review: accept NumPy scalars (as classic does), or raise 
UNSUPPORTED_DATA_TYPE instead of the AttributeError? The PR accepts them. 
Unsigned NumPy types (e.g. np.uint8) are still unsupported by Connect literal 
inference, which could be a follow-up. Happy to adjust the direction if 
preferred.


> Spark Connect Column operators fail with NumPy scalar operands
> --------------------------------------------------------------
>
>                 Key: SPARK-61562
>                 URL: https://issues.apache.org/jira/browse/SPARK-61562
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect, PySpark
>    Affects Versions: 4.3.0, 5.0.0
>            Reporter: Narendra Singh
>            Priority: Major
>              Labels: pull-request-available
>
> In Spark Connect, Column operators and methods raise an AttributeError when 
> the other operand is a NumPy scalar, while classic PySpark accepts it.
> {code:python}
> import numpy as np
> from pyspark.sql import functions as F
> df = spark.range(3).select(F.col("id").alias("x"))
> df.select(F.col("x") + np.int64(1)).collect()
> # Classic: [Row((x + 1)=1), Row((x + 1)=2), Row((x + 1)=3)]
> # Connect: AttributeError: 'numpy.int64' object has no attribute '_expr'
> {code}
> The same happens for np.int32, np.float32 and np.bool_ operands of arithmetic 
> and comparison operators, and for Column.between. np.float64 works only 
> because it is a subclass of Python float. F.lit(np.int64(1)) already works in 
> Connect.
> Cause: _bin_op in python/pyspark/sql/connect/column.py turns only built-in 
> Python types (bool, int, float, str, date/time, Decimal) into literals and 
> treats anything else as a Column, although LiteralExpression already supports 
> NumPy scalars (np.generic).
> Proposed fix: also send NumPy scalars to LiteralExpression in _bin_op and the 
> other Column methods that take literal values, and add tests that run on both 
> classic and Connect.
> Related: SPARK-61473 records this AttributeError behavior of PySpark's 
> _bin_op for the Spark Connect Rust client. Classic PySpark accepts NumPy 
> scalars here, so this proposes accepting them in PySpark Connect too.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to