asddfl created SPARK-60647:
------------------------------
Summary: pandas-on-Spark np.exp on a boolean Series fails with a
SQL type mismatch
Key: SPARK-60647
URL: https://issues.apache.org/jira/browse/SPARK-60647
Project: Spark
Issue Type: Bug
Components: Pandas API on Spark, PySpark
Affects Versions: 4.2.0
Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
Python: 3.11.17
pyspark 4.2.0
findspark 2.0.1
pandas 3.0.6
numpy 2.4.6
Reporter: asddfl
Applying np.exp to a boolean Series succeeds in Pandas/NumPy but fails in the
pandas API on Spark. The generated SQL applies EXP directly to the BOOLEAN
column instead of promoting it to a numeric input.
{code:python}
import pandas as pd
import numpy as np
from pyspark.sql import SparkSession
import pyspark.pandas as ps
t0 = pd.DataFrame({'c0': [False]})
print("Pandas:", flush=True)
result = np.exp(t0['c0'])
print(result, flush=True)
spark = (
SparkSession.builder
.master("local[2]")
.config("spark.sql.ansi.enabled", "false")
.config("spark.sql.shuffle.partitions", "2")
.getOrCreate()
)
t0 = ps.DataFrame({'c0': [False]})
print("PySpark Pandas:", flush=True)
result = np.exp(t0['c0'])
print(result, flush=True)
spark.stop()
{code}
{code:bash}
Pandas:
0 1.0
Name: c0, dtype: float16
PySpark Pandas:
Traceback (most recent call last):
File "/test.py", line 25, in <module>
result = np.exp(t0['c0'])
^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py",
line 555, in __array_ufunc__
result = numpy_compat.maybe_dispatch_ufunc_to_spark_func(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/numpy_compat.py",
line 226, in maybe_dispatch_ufunc_to_spark_func
return column_op(convert_arguments)(*inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py",
line 246, in wrapper
self._internal.spark_frame.select(scol).schema[0],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/classic/dataframe.py",
line 1101, in select
jdf = self._jdf.select(self._jcols(*cols))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/py4j/java_gateway.py",
line 1362, in __call__
return_value = get_return_value(
^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py",
line 257, in deco
raise converted from None
pyspark.errors.exceptions.captured.AnalysisException:
[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "EXP(c0)" due to data
type mismatch: The first parameter requires the "DOUBLE" type, however "c0" has
the type "BOOLEAN". SQLSTATE: 42K09;
'Project [unresolvedalias(EXP(c0#1))]
+- Project [__index_level_0__#0L, c0#1, monotonically_increasing_id() AS
__natural_order__#2L]
+- LocalRelation [__index_level_0__#0L, c0#1]
{code}
*Expected behavior*
np.exp(False) should return the numeric value 1.0 using NumPy-compatible type
promotion, as in the Pandas result.
*Actual behavior*
The call raises AnalysisException with condition
DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE and SQLSTATE 42K09. The message states
that EXP(c0) requires DOUBLE but c0 is BOOLEAN.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
