Hyukjin Kwon created SPARK-61786:
------------------------------------

             Summary: Always use the Rust protobuf runtime in 
pyspark.sql.connect.proto, and remove PYSPARK_CONNECT_PROTO_IMPLEMENTATION
                 Key: SPARK-61786
                 URL: https://issues.apache.org/jira/browse/SPARK-61786
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


pyspark.sql.connect.proto exports the messages of the drop-in's Rust protobuf 
runtime (SPARK-61755) whether or not the protobuf package is installed, and 
PYSPARK_CONNECT_PROTO_IMPLEMENTATION is removed, with the vendored generated 
*_pb2.py modules.

Where protobuf is installed, the files of the protos are added to its default 
descriptor pool, as the generated modules add them, so that the modules that 
protoc generates of other protos that import them (a plugin's, such as Delta 
Lake's Connect client) build, and protobuf's json_format finds the types of an 
Any. The runtime gains what protobuf's own modules read of a message: 
GetOptions and CopyToProto of the descriptors, the pool of a file, service and 
method descriptors, the methods of protobuf's Any and Timestamp (a Timestamp 
field takes a datetime), HasExtension/ClearExtension/Extensions as upb's, and 
the names of the generated modules in the modules of the protos. 
example_plugins.proto is compiled into the runtime.

A message field of a message of protobuf's own classes (of a plugin's generated 
module) takes a message of pyspark.sql.connect.proto through its bytes or a 
dict only: protobuf's messages take their own classes only, so their CopyFrom 
raises TypeError for one of the runtime's.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to