asddfl created SPARK-60658:
------------------------------
Summary: pandas-on-Spark np.floor converts negative infinity to
the minimum int64 value
Key: SPARK-60658
URL: https://issues.apache.org/jira/browse/SPARK-60658
Project: Spark
Issue Type: Bug
Components: Pandas API on Spark, PySpark
Affects Versions: 4.2.0
Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
Python: 3.11.17
pyspark 4.2.0
findspark 2.0.1
pandas 3.0.6
numpy 2.4.6
Reporter: asddfl
When using the pandas API on Spark (pyspark.pandas), np.floor on a float64
Series containing negative infinity returns -9223372036854775808 with dtype
int64. Pandas/NumPy preserves negative infinity and returns dtype float64.
A single float64 input with explicit negative infinity isolates the floor
operation. Both queries complete successfully, but the value and result dtype
differ.
{code:python}
import pandas as pd
import numpy as np
from pyspark.sql import SparkSession
import pyspark.pandas as ps
t0 = pd.DataFrame({'c0': [float('-inf')]})
print("Pandas:", flush=True)
result = np.floor(t0['c0'])
print(result, flush=True)
spark = (
SparkSession.builder
.master("local[2]")
.config("spark.sql.ansi.enabled", "false")
.config("spark.sql.shuffle.partitions", "2")
.getOrCreate()
)
ps.set_option("compute.ops_on_diff_frames", True)
t0 = ps.DataFrame({'c0': [float('-inf')]})
print("PySpark Pandas:", flush=True)
result = np.floor(t0['c0'])
print(result, flush=True)
spark.stop()
{code}
{code:bash}
Pandas:
0 -inf
Name: c0, dtype: float64
PySpark Pandas:
0 -9223372036854775808
Name: c0, dtype: int64
{code}
*Expected behavior*
np.floor(float('-inf')) should preserve negative infinity and produce a float64
result, matching Pandas/NumPy.
*Actual behavior*
Pandas returns -inf with dtype float64. pandas-on-Spark returns
-9223372036854775808, the minimum signed 64-bit integer, with dtype int64.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
