Hyukjin Kwon created SPARK-61786:
------------------------------------
Summary: Always use the Rust protobuf runtime in
pyspark.sql.connect.proto, and remove PYSPARK_CONNECT_PROTO_IMPLEMENTATION
Key: SPARK-61786
URL: https://issues.apache.org/jira/browse/SPARK-61786
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
pyspark.sql.connect.proto exports the messages of the drop-in's Rust protobuf
runtime (SPARK-61755) whether or not the protobuf package is installed, and
PYSPARK_CONNECT_PROTO_IMPLEMENTATION is removed, with the vendored generated
*_pb2.py modules.
Where protobuf is installed, the files of the protos are added to its default
descriptor pool, as the generated modules add them, so that the modules that
protoc generates of other protos that import them (a plugin's, such as Delta
Lake's Connect client) build, and protobuf's json_format finds the types of an
Any. The runtime gains what protobuf's own modules read of a message:
GetOptions and CopyToProto of the descriptors, the pool of a file, service and
method descriptors, the methods of protobuf's Any and Timestamp (a Timestamp
field takes a datetime), HasExtension/ClearExtension/Extensions as upb's, and
the names of the generated modules in the modules of the protos.
example_plugins.proto is compiled into the runtime.
A message field of a message of protobuf's own classes (of a plugin's generated
module) takes a message of pyspark.sql.connect.proto through its bytes or a
dict only: protobuf's messages take their own classes only, so their CopyFrom
raises TypeError for one of the runtime's.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]