asddfl created SPARK-60653:
------------------------------
Summary: pandas-on-Spark np.lcm declares a double result and fails
for an int64 value above 2**53
Key: SPARK-60653
URL: https://issues.apache.org/jira/browse/SPARK-60653
Project: Spark
Issue Type: Bug
Components: Pandas API on Spark, PySpark
Affects Versions: 4.2.0
Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
Python: 3.11.17
pyspark 4.2.0
findspark 2.0.1
pandas 3.0.6
numpy 2.4.6
Reporter: asddfl
When using the pandas API on Spark (pyspark.pandas), np.lcm fails for the valid
int64 value 2**53 + 1. The same operation on a Pandas Series returns the
integer unchanged.
The failure occurs while converting the integer result to Arrow double in the
Python worker. This input and its least common multiple are both within the
int64 range, but 2**53 + 1 cannot be represented exactly as float64.
{code:python}
import pandas as pd
import numpy as np
from pyspark.sql import SparkSession
import pyspark.pandas as ps
t0 = pd.DataFrame({'c0': [2**53 + 1]})
print("Pandas:", flush=True)
result = np.lcm(t0['c0'], t0['c0'])
print(result, flush=True)
spark = (
SparkSession.builder
.master("local[2]")
.config("spark.sql.ansi.enabled", "false")
.config("spark.sql.shuffle.partitions", "2")
.getOrCreate()
)
t0 = ps.DataFrame({'c0': [2**53 + 1]})
print("PySpark Pandas:", flush=True)
result = np.lcm(t0['c0'], t0['c0'])
print(result, flush=True)
spark.stop()
{code}
{code:bash}
Pandas:
0 9007199254740993
Name: c0, dtype: int64
PySpark Pandas:
Traceback (most recent call last):
File "/test.py", line 26, in <module>
print(result, flush=True)
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/series.py",
line 7481, in __repr__
pser =
self._psdf._get_or_create_repr_pandas_cache(max_display_count)[self.name]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py",
line 13988, in _get_or_create_repr_pandas_cache
self, "_repr_pandas_cache", {n: self.head(n + 1)._to_internal_pandas()}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py",
line 13983, in _to_internal_pandas
return self._internal.to_pandas_frame
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/utils.py",
line 608, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/internal.py", line
1080, in to_pandas_frame
pdf = sdf._to_pandas(pandasStructHandlingMode="row")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/classic/dataframe.py",
line 1974, in _to_pandas
return PandasConversionMixin._to_pandas(self, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/pandas/conversion.py",
line 367, in _to_pandas
batches = self._collect_as_arrow(
^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/pandas/conversion.py",
line 537, in _collect_as_arrow
with unwrap_spark_exception():
File "/opt/conda/envs/lib/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py",
line 280, in unwrap_spark_exception
raise converted from None
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
line 410, in convert_column
File "pyarrow/array.pxi", line 1341, in pyarrow.lib.Array.from_pandas
File "pyarrow/array.pxi", line 414, in pyarrow.lib.array
File "pyarrow/array.pxi", line 97, in pyarrow.lib._ndarray_to_array
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyspark.errors.exceptions.captured.PythonException: An exception was thrown
from the Python worker:
Traceback (most recent call last):
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
line 410, in convert_column
return pa.Array.from_pandas(
^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/array.pxi", line 1341, in pyarrow.lib.Array.from_pandas
File "pyarrow/array.pxi", line 414, in pyarrow.lib.array
File "pyarrow/array.pxi", line 97, in pyarrow.lib._ndarray_to_array
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 9007199254740993 not in range:
-9007199254740992 to 9007199254740992
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
line 415, in convert_column
return pa.Array.from_pandas(series, mask=mask).cast(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/array.pxi", line 1217, in pyarrow.lib.Array.cast
File "/opt/conda/envs/lib/python3.11/site-packages/pyarrow/compute.py", line
415, in cast
return call_function("cast", [arr], options, memory_pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_compute.pyx", line 610, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 405, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 155, in
pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 9007199254740993 not in range:
-9007199254740992 to 9007199254740992
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py",
line 3622, in main
process()
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py",
line 3612, in process
serializer.dump_stream(out_iter, outfile)
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
line 546, in dump_stream
return ArrowStreamSerializer.dump_stream(self, batches, stream)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
line 146, in dump_stream
for batch in iterator:
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
line 168, in _write_stream_start
first = next(batch_iterator, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
line 543, in <genexpr>
(create_batch(_normalize_packed(packed)) for packed in iterator),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
line 529, in create_batch
return PandasToArrowConversion.convert(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
line 442, in convert
arrays = [convert_column(col, field) for col, field in zip(columns,
schema.fields)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
line 442, in <listcomp>
arrays = [convert_column(col, field) for col, field in zip(columns,
schema.fields)]
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/conversion.py",
line 440, in convert_column
raise PySparkValueError(error_msg) from e
pyspark.errors.exceptions.base.PySparkValueError: Failed to convert the value
of the column '_0' with type 'int64' to Arrow type 'double'. It can be caused
by overflows or other unsafe conversions warned by Arrow. Arrow safe type check
can be disabled by using SQL config
`spark.sql.execution.pandas.convertToArrowArraySafely`.
{code}
*Expected behavior*
np.lcm applied to two Series containing 9007199254740993 should return an
integer Series containing 9007199254740993, matching Pandas without an unsafe
conversion to double.
*Actual behavior*
The computation raises pyspark.errors.exceptions.captured.PythonException. Its
worker traceback contains pyarrow.lib.ArrowInvalid: Integer value
9007199254740993 not in range: -9007199254740992 to 9007199254740992, followed
by a failure to convert an int64 column to Arrow double.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
