[ 
https://issues.apache.org/jira/browse/SPARK-62056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-62056.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 873
[https://github.com/apache/spark-connect-rust/pull/873]

> Take the arguments of the methods of the maps and repeated containers of the 
> Rust protobuf runtime as upb's, with upb's errors
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62056
>                 URL: https://issues.apache.org/jira/browse/SPARK-62056
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> pyspark.sql.connect.proto is the drop-in's protobuf runtime in Rust 
> (crates/pyspark-rs/src/pbmsg),
> which follows upb, the implementation of a protobuf install that 
> pyspark-client 4.3.0 uses. The
> methods of its containers took their arguments as PyO3's defaults make them, 
> with PyO3's errors,
> where upb's take them as their C functions and collections.abc's Python 
> methods do:
> - upb's maps are of its C types ScalarMapContainer and MessageMapContainer.
>   - MergeFrom, clear, GetEntryClass and get_or_create take positional 
> arguments only, with
>     CPython's errors naming the class: "ScalarMapContainer.MergeFrom() takes 
> exactly one argument
>     (0 given)", "... takes no keyword arguments".
>   - setdefault takes no keyword arguments and counts its arguments as 
> PyArg_UnpackTuple does. A
>     map of messages refuses it whatever its arguments.
>   - get parses its key and default as PyArg_ParseTupleAndKeywords("O|O") does.
>   - pop, popitem, update, keys, values and items are MutableMapping's and 
> Mapping's, in Python.
> - upb's repeated containers take positional arguments only:
>   - append, extend, remove and MergeFrom take exactly one;
>   - insert and pop parse theirs as PyArg_ParseTuple's "nO" and "|n";
>   - __deepcopy__ takes no keyword arguments.
> The runtime's methods took keyword arguments, so these all succeeded where 
> upb raises TypeError:
> MergeFrom(other=...), append(value=...), insert(key=..., value=...), 
> pop(key=...),
> setdefault(key, default=...), __deepcopy__(memo=...). Their errors were 
> PyO3's, and the maps'
> errors named MapContainer: "MapContainer.MergeFrom() missing 1 required 
> positional argument:
> 'other'".
> Repro (master 80046f2f)
>   >>> import pyspark.sql.connect.proto as pb
>   >>> d = pb.Read.DataSource(); d.options.MergeFrom()
>   TypeError: MapContainer.MergeFrom() missing 1 required positional argument: 
> 'other'
>   >>> d.options.MergeFrom(other=[("a", "b")]); dict(d.options)
>   {'a': 'b'}
>   >>> q = pb.StatApproxQuantile(); q.probabilities.append(value=0.5); 
> list(q.probabilities)
>   [0.5]
>   >>> q.probabilities.insert(0)
>   TypeError: RepeatedScalarContainer.insert() missing 1 required positional 
> argument: 'value'
> Expected (as with pyspark-client 4.3.0)
>   TypeError: ScalarMapContainer.MergeFrom() takes exactly one argument (0 
> given)
>   TypeError: ScalarMapContainer.MergeFrom() takes no keyword arguments
>   TypeError: RepeatedScalarContainer.append() takes no keyword arguments
>   TypeError: function takes exactly 2 arguments (1 given)



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to