Hyukjin Kwon created SPARK-61867:
------------------------------------

             Summary: Give the protoc-generated modules of Spark Connect 
plugins the classes of the Rust protobuf runtime, so that Delta Lake's DML 
plans take the drop-in's relations
                 Key: SPARK-61867
                 URL: https://issues.apache.org/jira/browse/SPARK-61867
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Since SPARK-61786, pyspark.sql.connect.proto exports the messages of the 
drop-in's Rust protobuf runtime. A Spark Connect plugin's Python client, such 
as delta-spark 4.0.0's delta.connect, has protoc-generated modules that import 
Spark Connect's protos and build their messages with protobuf's own classes. 
Their fields of Spark Connect's types then refuse the drop-in's messages: 
DeltaTable.delete, update and merge raise TypeError at 
relation.delete_from_table.target.CopyFrom(self._target.plan(session)), where 
the official client works.
Also, a plugin's relation or command that holds df._plan.plan(session), as 
Delta's merge does with its source, holds the drop-in's placeholder of the 
DataFrame's relation inside the extension's Any, which the core did not replace.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to