Hyukjin Kwon created SPARK-61120:
------------------------------------
Summary: Add DataFrame._session, SparkSession._client and the
results of execute_command that vendored modules read, in the Python client as
PySpark does
Key: SPARK-61120
URL: https://issues.apache.org/jira/browse/SPARK-61120
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Modules vendored from Apache Spark v4.3.0 read private attributes of PySpark
Connect's DataFrame, session and client that the Python client lacks:
* {{DataFrame._session}}, the session of a DataFrame, which these read:
** {{pyspark.ml.util._cache_spark_dataset}}, which {{CrossValidator}} and
{{TrainValidationSplit}} use with {{SPARKML_TEMP_DFS_PATH}}
** {{pyspark.sql.plot}}, for the maximum rows of a plot
** {{select_dtypes}} of {{pyspark.pandas}}
On the client, {{df._session}} is a Column of that name, so with
{{SPARKML_TEMP_DFS_PATH}} set, {{CrossValidator(...).fit(df)}} raises
{{TypeError: 'Column' object is not callable}}.
* {{SparkSession._client}} and {{SparkConnectClient._resources()}}, of which
{{TorchDistributor}} reads the resources of the cluster on Spark Connect. The
client has neither.
* PySpark's {{SparkConnectClient.execute_command}} returns the result of a
{{get_resources_command}} as a dict of {{ResourceInformation}}, of a
{{create_resource_profile_command}} as its profile ID, and of a
{{checkpoint_command}} as a DataFrame of its cached relation, and it also
returns the {{pipeline_event_result}} of a response. The client returns the
first as its message, and none of the others.
The fix is to add them, as PySpark has them.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
