Hyukjin Kwon created SPARK-61992:
------------------------------------

             Summary: Take the map fields of a protobuf message's constructor 
as upb's InitMapAttributes, and make the runtime's DecodeError a subclass of 
protobuf's
                 Key: SPARK-61992
                 URL: https://issues.apache.org/jira/browse/SPARK-61992
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Two documented gaps of the drop-in's protobuf runtime against PySpark 4.3.0 
with upb (protobuf 7.36.2):
- A map field of a message's constructor takes only a mapping, and raises 
"TypeError: ... is not iterable" for any other value. upb's InitMapAttributes 
takes, for a map of scalars, anything that MutableMapping.update takes: pairs, 
an iterator of pairs, lists of 2-character strings and [], with Python's 
ValueError for a pair of another shape, and a tuple as the arguments of update. 
For a map of messages it takes the keys that the value iterates, each of 
value[key]: a dict sets the fields of the key's message, and anything else goes 
through CopyFrom.
- The runtime's DecodeError is not a google.protobuf.message.DecodeError, so 
"except google.protobuf.message.DecodeError" does not catch a failed parse, 
where it catches upb's. The error also does not pickle (attribute lookup 
DecodeError on pyspark.sql.connect.proto failed).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to