[
https://issues.apache.org/jira/browse/SPARK-61562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61562:
-----------------------------------
Labels: pull-request-available (was: )
> Spark Connect Column operators fail with NumPy scalar operands
> --------------------------------------------------------------
>
> Key: SPARK-61562
> URL: https://issues.apache.org/jira/browse/SPARK-61562
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
> Affects Versions: 4.3.0, 5.0.0
> Reporter: Narendra Singh
> Priority: Major
> Labels: pull-request-available
>
> In Spark Connect, Column operators and methods raise an AttributeError when
> the other operand is a NumPy scalar, while classic PySpark accepts it.
> {code:python}
> import numpy as np
> from pyspark.sql import functions as F
> df = spark.range(3).select(F.col("id").alias("x"))
> df.select(F.col("x") + np.int64(1)).collect()
> # Classic: [Row((x + 1)=1), Row((x + 1)=2), Row((x + 1)=3)]
> # Connect: AttributeError: 'numpy.int64' object has no attribute '_expr'
> {code}
> The same happens for np.int32, np.float32 and np.bool_ operands of arithmetic
> and comparison operators, and for Column.between. np.float64 works only
> because it is a subclass of Python float. F.lit(np.int64(1)) already works in
> Connect.
> Cause: _bin_op in python/pyspark/sql/connect/column.py turns only built-in
> Python types (bool, int, float, str, date/time, Decimal) into literals and
> treats anything else as a Column, although LiteralExpression already supports
> NumPy scalars (np.generic).
> Proposed fix: also send NumPy scalars to LiteralExpression in _bin_op and the
> other Column methods that take literal values, and add tests that run on both
> classic and Connect.
> Related: SPARK-61473 records this AttributeError behavior of PySpark's
> _bin_op for the Spark Connect Rust client. Classic PySpark accepts NumPy
> scalars here, so this proposes accepting them in PySpark Connect too.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]