Hyukjin Kwon created SPARK-61992:
------------------------------------
Summary: Take the map fields of a protobuf message's constructor
as upb's InitMapAttributes, and make the runtime's DecodeError a subclass of
protobuf's
Key: SPARK-61992
URL: https://issues.apache.org/jira/browse/SPARK-61992
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Two documented gaps of the drop-in's protobuf runtime against PySpark 4.3.0
with upb (protobuf 7.36.2):
- A map field of a message's constructor takes only a mapping, and raises
"TypeError: ... is not iterable" for any other value. upb's InitMapAttributes
takes, for a map of scalars, anything that MutableMapping.update takes: pairs,
an iterator of pairs, lists of 2-character strings and [], with Python's
ValueError for a pair of another shape, and a tuple as the arguments of update.
For a map of messages it takes the keys that the value iterates, each of
value[key]: a dict sets the fields of the key's message, and anything else goes
through CopyFrom.
- The runtime's DecodeError is not a google.protobuf.message.DecodeError, so
"except google.protobuf.message.DecodeError" does not catch a failed parse,
where it catches upb's. The error also does not pickle (attribute lookup
DecodeError on pyspark.sql.connect.proto failed).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]