Hyukjin Kwon created SPARK-61867:
------------------------------------
Summary: Give the protoc-generated modules of Spark Connect
plugins the classes of the Rust protobuf runtime, so that Delta Lake's DML
plans take the drop-in's relations
Key: SPARK-61867
URL: https://issues.apache.org/jira/browse/SPARK-61867
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Since SPARK-61786, pyspark.sql.connect.proto exports the messages of the
drop-in's Rust protobuf runtime. A Spark Connect plugin's Python client, such
as delta-spark 4.0.0's delta.connect, has protoc-generated modules that import
Spark Connect's protos and build their messages with protobuf's own classes.
Their fields of Spark Connect's types then refuse the drop-in's messages:
DeltaTable.delete, update and merge raise TypeError at
relation.delete_from_table.target.CopyFrom(self._target.plan(session)), where
the official client works.
Also, a plugin's relation or command that holds df._plan.plan(session), as
Delta's merge does with its source, holds the drop-in's placeholder of the
DataFrame's relation inside the extension's Any, which the core did not replace.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]