Hyukjin Kwon created SPARK-61011:
------------------------------------
Summary: Refer to the columns of a DataFrame by its plan ID in the
Python client as PySpark does
Key: SPARK-61011
URL: https://issues.apache.org/jira/browse/SPARK-61011
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, the columns of a DataFrame refer to it by the plan ID of
its plan: {{df["name"]}}, {{df.name}} and {{df._col(name)}} are the
{{ColumnReference}} of the name and the plan ID, {{df["*"]}} the
{{UnresolvedStar}} of the plan ID, {{df.metadataColumn(name)}} the metadata
{{ColumnReference}} of it, and {{df.colRegex(regex)}} the Column of the
{{UnresolvedRegex}} of it. The server resolves them against that plan, so that
{{df1.join(df2, df1["id"] == df2["id"]).select(df1["id"], df2["w"])}} selects
the {{id}} of {{df1}}, and a self-join of a DataFrame and one derived from it
raises {{AMBIGUOUS_COLUMN_REFERENCE}}.
The Rust-backed Python client refers to them by name only, so the join above
raises {{AMBIGUOUS_REFERENCE}}, as does {{df1.join(df2, df1.id == df2.id)}};
and {{colRegex}} returns a DataFrame rather than a Column. Refer to the columns
of a DataFrame as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
