Hyukjin Kwon created SPARK-62019:
------------------------------------
Summary: Compare the messages of the Rust protobuf runtime as
upb's: floats of their bits, unknown fields of their contents, maps as
mappings, and take only protobuf's messages of its default pool
Key: SPARK-62019
URL: https://issues.apache.org/jira/browse/SPARK-62019
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Problem
pyspark.sql.connect.proto is the drop-in's protobuf runtime in Rust
(crates/pyspark-rs/src/pbmsg),
which follows upb, the implementation of a protobuf install that pyspark-client
4.3.0 uses. Its
equality and its type checks differ from upb's:
1. == of two messages compared float and double fields by value. upb
(upb_Message_IsEqual) compares
them by their bits. So a message of a NaN never equaled its copy or its
parse, `in` and
`remove` of a container missed it, and -0.0 equaled 0.0 and the unset
default of a field
without presence.
2. == compared the unknown fields of two messages byte for byte. upb compares
their fields, sorted
by their tags, of their values. So the same unknown fields in another order,
or a varint of
another encoding, were unequal.
3. CopyFrom, MergeFrom, the constructors and the containers took any object
that has the
DESCRIPTOR.full_name of the type and a SerializeToString: protobuf's message
of a type of the
same name of another pool, or a lookalike. upb takes a message of its class
only, and refuses
those with the TypeError of another class. == already refused them, since
SPARK-61958.
4. == of two maps of other key or value types (a plugin's maps of int32 and of
int64 keys) was
False for equal keys and values, and a map of a NaN value equaled itself.
upb's == of maps is
Mapping.__eq__, of the dicts of their items.
docs/compatibility.md and dev/parity/known-differences.md also gave no reason
why a plugin's
file that imports no proto of Spark Connect (Delta Lake's base.proto) keeps
protobuf's classes.
Repro (master 9cf73197)
>>> import copy; import pyspark.sql.connect.proto as pb
>>> m = pb.Expression.Literal(double=float("nan")); m == copy.deepcopy(m)
False
>>> pb.Sample(lower_bound=-0.0) == pb.Sample()
True
>>> a = pb.Expression.Literal.FromString(bytes.fromhex("a81f05b21f03616263"))
>>> b = pb.Expression.Literal.FromString(bytes.fromhex("b21f03616263a81f05"))
>>> a == b # the same unknown fields 501 and 502, in the other
order
False
>>> other = <protobuf's class of spark.connect.StorageLevel of another
DescriptorPool>
>>> m = pb.StorageLevel(); m.CopyFrom(other(use_disk=True)); m.use_disk
True
Expected (as with pyspark-client 4.3.0)
True, False, True, and CopyFrom raises TypeError: Parameter to CopyFrom()
must be instance of
same class: expected <class '...StorageLevel'> got <class 'StorageLevel'>.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]