[
https://issues.apache.org/jira/browse/SPARK-62010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62010:
------------------------------------
Assignee: Hyukjin Kwon
> Make the maps of the Rust protobuf runtime merge their arguments, fail an
> iteration that they change, and order integer keys as upb's maps do
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62010
> URL: https://issues.apache.org/jira/browse/SPARK-62010
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Type: Bug Component: Connect
> Problem
> pyspark.sql.connect.proto is the drop-in's protobuf runtime in Rust
> (crates/pyspark-rs/src/pbmsg),
> which follows upb, the implementation of a protobuf install that
> pyspark-client 4.3.0 uses. Its map
> containers differ from upb's in four ways:
> 1. MapContainer.MergeFrom of a map of the runtime of other key or value types
> took its entries
> as they were, without converting them. So a map held keys and values of
> other types:
> sql.args.MergeFrom(sql.named_arguments) put an Expression into the map of
> Literals, which
> serialized as a Literal of other fields, and options.MergeFrom(sql.args)
> put a message into a
> map of strings. A map of int32 keys merged from a map of uint32 keys held
> two entries of the
> key 4, and one of bool keys held True. upb converts each entry, with its
> errors.
> 2. MapContainer.MergeFrom of any other argument but a dict iterated it as
> (key, value) pairs. upb
> takes a mapping by its items (protobuf's maps, MappingProxyType, UserDict,
> an object with keys),
> as MutableMapping.update does. So the map of a message of protobuf's class
> of the type, which a
> plugin's module that keeps protobuf's classes gives (Delta Lake's
> base_pb2), merged
> {"ab": "zz", "cd": "yy"} as {"a": "b", "c": "d"}, and failed for keys of
> other lengths. Of a
> map of messages, upb reads each key's value and copies it into the key's
> message (CopyFrom), or
> sets its fields of a dict, and raises "Argument for field ... is not
> iterable".
> upb's MergeFrom is a dict check, then upb's InitMapAttributes, which
> SPARK-61992 gave the
> runtime's constructor: it also copies a message into the message that the
> map has for its
> key, so that a message read before (x = map[key]) has the new fields,
> where the runtime
> replaced it. And map.update(None) did nothing, where upb's
> MutableMapping.update raises the
> TypeError of iterating None.
> 3. A map changed during its iteration did not fail the iteration. upb raises
> "RuntimeError: Map modified during iteration." at the next step, once a
> method of the map that
> may change its keys ran. The runtime iterated a copy of the keys, so
> items() and values() read
> the value of a key that the loop had removed, which inserted the key
> again, of the default
> value, into the message.
> 4. The keys of a signed integer type were ordered as signed numbers in the
> serialization and in
> str(). upb orders them as unsigned numbers (0, 1, 3, 100, -5, -1), so the
> bytes of
> SerializeToString(deterministic=True) differed. The protos of Spark
> Connect have maps of string
> keys only, so this affects the maps of plugins' protos.
> docs/compatibility.md also:
> - described upb's limit of 100 nested messages as a limit of its parse only;
> - described the serialization order of maps and the outcome of a truncated
> varint imprecisely;
> - said that no message that the runtime makes of a parse or a copy is tracked
> by the garbage
> collector, where the top-level messages are;
> - gave the server's limit of the depth of a plan as 100 levels, where it is
> 1,024 nested messages
> (spark.connect.grpc.marshallerRecursionLimit), from about 510 nested
> expressions.
> Repro (master 9cf73197; scan is a message of a plugin's module of the
> runtime's classes, whose
> field table is of a type of base_pb2, a plugin's module that keeps protobuf's
> classes, as Delta
> Lake's DeltaScan holds a DeltaTable)
> >>> s = pb.SQL(); s.args.MergeFrom(pb.SQL(named_arguments={"k":
> {"unresolved_attribute": {"unparsed_identifier": "c"}}}).named_arguments)
> >>> pb.SQL.FromString(s.SerializeToString()).args["k"]
> binary: "\n\001c" (the Expression's bytes, read as a
> Literal)
> >>> d = pb.Read.DataSource();
> d.options.MergeFrom(types.MappingProxyType({"ab": "zz"})); dict(d.options)
> {'a': 'b'}
> >>> scan.table.options.MergeFrom(base_pb2.Table(options={"ab":
> "zz"}).options); dict(scan.table.options)
> {'a': 'b'}
> >>> d = pb.Read.DataSource(options={"k0": "v0", "k1": "v1", "k2": "v2"})
> >>> for k, v in d.options.items():
> ... if k == "k0": del d.options["k1"]
> >>> dict(d.options)
> {'k0': 'v0', 'k2': 'v2', 'k1': ''}
> Expected (as with pyspark-client 4.3.0)
> The first MergeFrom raises TypeError: Parameter to CopyFrom() must be
> instance of same class:
> expected <class '...Expression.Literal'> got <class '...Expression'>.
> The two other MergeFrom calls give {'ab': 'zz'}.
> The loop raises RuntimeError: Map modified during iteration., and k1 stays
> deleted.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]