Hyukjin Kwon created SPARK-62019:
------------------------------------

             Summary: Compare the messages of the Rust protobuf runtime as 
upb's: floats of their bits, unknown fields of their contents, maps as 
mappings, and take only protobuf's messages of its default pool
                 Key: SPARK-62019
                 URL: https://issues.apache.org/jira/browse/SPARK-62019
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Problem
pyspark.sql.connect.proto is the drop-in's protobuf runtime in Rust 
(crates/pyspark-rs/src/pbmsg),
which follows upb, the implementation of a protobuf install that pyspark-client 
4.3.0 uses. Its
equality and its type checks differ from upb's:

1. == of two messages compared float and double fields by value. upb 
(upb_Message_IsEqual) compares
   them by their bits. So a message of a NaN never equaled its copy or its 
parse, `in` and
   `remove` of a container missed it, and -0.0 equaled 0.0 and the unset 
default of a field
   without presence.
2. == compared the unknown fields of two messages byte for byte. upb compares 
their fields, sorted
   by their tags, of their values. So the same unknown fields in another order, 
or a varint of
   another encoding, were unequal.
3. CopyFrom, MergeFrom, the constructors and the containers took any object 
that has the
   DESCRIPTOR.full_name of the type and a SerializeToString: protobuf's message 
of a type of the
   same name of another pool, or a lookalike. upb takes a message of its class 
only, and refuses
   those with the TypeError of another class. == already refused them, since 
SPARK-61958.
4. == of two maps of other key or value types (a plugin's maps of int32 and of 
int64 keys) was
   False for equal keys and values, and a map of a NaN value equaled itself. 
upb's == of maps is
   Mapping.__eq__, of the dicts of their items.

docs/compatibility.md and dev/parity/known-differences.md also gave no reason 
why a plugin's
file that imports no proto of Spark Connect (Delta Lake's base.proto) keeps 
protobuf's classes.

Repro (master 9cf73197)
  >>> import copy; import pyspark.sql.connect.proto as pb
  >>> m = pb.Expression.Literal(double=float("nan")); m == copy.deepcopy(m)
  False
  >>> pb.Sample(lower_bound=-0.0) == pb.Sample()
  True
  >>> a = pb.Expression.Literal.FromString(bytes.fromhex("a81f05b21f03616263"))
  >>> b = pb.Expression.Literal.FromString(bytes.fromhex("b21f03616263a81f05"))
  >>> a == b             # the same unknown fields 501 and 502, in the other 
order
  False
  >>> other = <protobuf's class of spark.connect.StorageLevel of another 
DescriptorPool>
  >>> m = pb.StorageLevel(); m.CopyFrom(other(use_disk=True)); m.use_disk
  True

Expected (as with pyspark-client 4.3.0)
  True, False, True, and CopyFrom raises TypeError: Parameter to CopyFrom() 
must be instance of
  same class: expected <class '...StorageLevel'> got <class 'StorageLevel'>.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to