[ 
https://issues.apache.org/jira/browse/SPARK-61524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61524.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 560
[https://github.com/apache/spark-connect-rust/pull/560]

> Raise the errors of protobuf's default backend for the values that PySpark 
> sets in proto fields, as PySpark does
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61524
>                 URL: https://issues.apache.org/jira/browse/SPARK-61524
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> The Python drop-in raises the errors of protobuf's pure-Python backend where 
> PySpark sets a value in a proto field. A default install of pyspark-client 
> uses protobuf's upb backend, whose errors and conversions differ. The 
> pure-Python backend is used only when 
> PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python is set.
> Under upb, official PySpark 4.3.0 behaves as below, with protobuf 7.36, which 
> a fresh install of pyspark-client 4.3.0 resolves through grpcio-status:
> - A str that UTF-8 cannot encode (a lone surrogate), in a literal, an option, 
> a fill, the runtime conf, a table name or the text of sql, raises the 
> UnicodeEncodeError of its encoding: "'utf-8' codec can't encode character 
> '\ud800' in position 1: surrogates not allowed". The drop-in raised a 
> ValueError "... isn't a valid unicode string and can't be encoded in UTF-8."
> - Bytes in a string field are decoded as UTF-8, and bytes that are not UTF-8 
> raise the UnicodeDecodeError of their decoding. This covers the keys of the 
> runtime conf, of the builder's options and of the writers' options, the text 
> of sql, the names of _joinAsOf's `on` and of replace's subset, and the values 
> of the conf and of createTable's options. The drop-in refused bytes there, or 
> raised a ValueError "... has type bytes, but isn't valid UTF-8 encoding".
> - Any other object in a string field raises TypeError "bad argument type for 
> built-in operation". The drop-in raised "Cannot set 
> spark.connect.KeyValue.value to ..." or "... has type ..., but expected one 
> of: (<class 'bytes'>, <class 'str'>)".
> - A bool in the int fields of dropna's thresh and sampleBy's seed raises 
> TypeError "Field spark.connect.NADrop.min_non_nulls: Expected an int, got a 
> boolean." (protobuf 6.33 sent it as 1 or 0 with a DeprecationWarning). The 
> drop-in raised the pure-Python backend's "Cannot set ... (int)."
> - StreamingQuery.explain(extended) reads its argument as a C long, and a 
> NumPy bool by its truth. Other values raise the TypeError or OverflowError of 
> that conversion, and a NumPy array raises "[1] has type ndarray, but expected 
> one of: bool".
> Also as PySpark:
> - DataFrameReader.option and DataStreamReader.option assert, in the 
> DataSource plan, that a key is a str: AssertionError.
> - The option of a writer or of DataStreamReader returns before it looks at 
> the key when the value is None.
> - A dict-kept key that is unhashable raises its TypeError.
> - An as-of column of _joinAsOf that is not a Column or a str raises an 
> AssertionError, before `on` is looked at.
> - conf.set(None, v) sends the empty key.
> - sql() checks `args` and formats the query object as it is before it sets 
> the query, so bytes with keyword arguments raise "expected str, got bytes".



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to