asddfl created SPARK-60643:
------------------------------
Summary: pandas-on-Spark drop_duplicates cannot resolve a column
name containing a dot
Key: SPARK-60643
URL: https://issues.apache.org/jira/browse/SPARK-60643
Project: Spark
Issue Type: Bug
Components: Pandas API on Spark, PySpark
Affects Versions: 4.2.0
Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
Python: 3.11.17
pyspark 4.2.0
findspark 2.0.1
pandas 3.0.6
numpy 2.4.6
Reporter: asddfl
Renaming a column from c0 to the literal name t0.c0 and calling drop_duplicates
raises an unresolved-column error in the pandas API on Spark. Pandas returns
the one-row DataFrame successfully.
The error treats the dot as a qualifier separator instead of preserving the
complete column name.
{code:python}
import pandas as pd
from pyspark.sql import SparkSession
import pyspark.pandas as ps
t0 = pd.DataFrame({'c0': [1]})
print("Pandas:", flush=True)
result = t0.rename(columns={'c0': 't0.c0'}).drop_duplicates()
print(result, flush=True)
spark = (
SparkSession.builder
.master("local[2]")
.config("spark.sql.ansi.enabled", "false")
.config("spark.sql.shuffle.partitions", "2")
.getOrCreate()
)
t0 = ps.DataFrame({'c0': [1]})
print("PySpark Pandas:", flush=True)
result = t0.rename(columns={'c0': 't0.c0'}).drop_duplicates()
print(result, flush=True)
spark.stop()
{code}
{code:bash}
Pandas:
t0.c0
0 1
PySpark Pandas:
Traceback (most recent call last):
File "/test.py", line 24, in <module>
result = t0.rename(columns={'c0': 't0.c0'}).drop_duplicates()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py",
line 10311, in drop_duplicates
sdf, column = self._mark_duplicates(subset, keep)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py",
line 5155, in _mark_duplicates
sdf = sdf.withColumn(column, F.row_number().over(window) > 1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/classic/dataframe.py",
line 1801, in withColumn
return DataFrame(self._jdf.withColumn(colName, col._jc), self.sparkSession)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/py4j/java_gateway.py",
line 1362, in __call__
return_value = get_return_value(
^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py",
line 257, in deco
raise converted from None
pyspark.errors.exceptions.captured.AnalysisException:
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter
with name `t0`.`c0` cannot be resolved. Did you mean one of the following?
[`t0.c0`, `__index_level_0__`, `__natural_order__`]. SQLSTATE: 42703;
'Project [__index_level_0__#0L, t0.c0#3L, __natural_order__#2L,
'`>`(row_number() windowspecdefinition('t0.c0, __natural_order__#2L ASC NULLS
FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())),
1) AS __duplicated__#4]
+- Project [__index_level_0__#0L, c0#1L AS t0.c0#3L, __natural_order__#2L]
+- Project [__index_level_0__#0L, c0#1L, monotonically_increasing_id() AS
__natural_order__#2L]
+- LocalRelation [__index_level_0__#0L, c0#1L]
{code}
*Expected behavior*
Return one row with value 1 under the literal column name t0.c0, matching
Pandas.
*Actual behavior*
The call raises AnalysisException with condition
UNRESOLVED_COLUMN.WITH_SUGGESTION and SQLSTATE 42703. Spark attempts to resolve
`t0`.`c0`, while its suggestions include the existing literal column `t0.c0`.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
