[
https://issues.apache.org/jira/browse/SPARK-62019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62019:
------------------------------------
Assignee: Hyukjin Kwon
> Compare the messages of the Rust protobuf runtime as upb's: floats of their
> bits, unknown fields of their contents, maps as mappings, and take only
> protobuf's messages of its default pool
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62019
> URL: https://issues.apache.org/jira/browse/SPARK-62019
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Problem
> pyspark.sql.connect.proto is the drop-in's protobuf runtime in Rust
> (crates/pyspark-rs/src/pbmsg),
> which follows upb, the implementation of a protobuf install that
> pyspark-client 4.3.0 uses. Its
> equality and its type checks differ from upb's:
> 1. == of two messages compared float and double fields by value. upb
> (upb_Message_IsEqual) compares
> them by their bits. So a message of a NaN never equaled its copy or its
> parse, `in` and
> `remove` of a container missed it, and -0.0 equaled 0.0 and the unset
> default of a field
> without presence.
> 2. == compared the unknown fields of two messages byte for byte. upb compares
> their fields, sorted
> by their tags, of their values. So the same unknown fields in another
> order, or a varint of
> another encoding, were unequal.
> 3. CopyFrom, MergeFrom, the constructors and the containers took any object
> that has the
> DESCRIPTOR.full_name of the type and a SerializeToString: protobuf's
> message of a type of the
> same name of another pool, or a lookalike. upb takes a message of its
> class only, and refuses
> those with the TypeError of another class. == already refused them, since
> SPARK-61958.
> 4. == of two maps of other key or value types (a plugin's maps of int32 and
> of int64 keys) was
> False for equal keys and values, and a map of a NaN value equaled itself.
> upb's == of maps is
> Mapping.__eq__, of the dicts of their items.
> docs/compatibility.md and dev/parity/known-differences.md also gave no reason
> why a plugin's
> file that imports no proto of Spark Connect (Delta Lake's base.proto) keeps
> protobuf's classes.
> Repro (master 9cf73197)
> >>> import copy; import pyspark.sql.connect.proto as pb
> >>> m = pb.Expression.Literal(double=float("nan")); m == copy.deepcopy(m)
> False
> >>> pb.Sample(lower_bound=-0.0) == pb.Sample()
> True
> >>> a =
> pb.Expression.Literal.FromString(bytes.fromhex("a81f05b21f03616263"))
> >>> b =
> pb.Expression.Literal.FromString(bytes.fromhex("b21f03616263a81f05"))
> >>> a == b # the same unknown fields 501 and 502, in the other
> order
> False
> >>> other = <protobuf's class of spark.connect.StorageLevel of another
> DescriptorPool>
> >>> m = pb.StorageLevel(); m.CopyFrom(other(use_disk=True)); m.use_disk
> True
> Expected (as with pyspark-client 4.3.0)
> True, False, True, and CopyFrom raises TypeError: Parameter to CopyFrom()
> must be instance of
> same class: expected <class '...StorageLevel'> got <class 'StorageLevel'>.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]