[
https://issues.apache.org/jira/browse/SPARK-61524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61524:
------------------------------------
Assignee: Hyukjin Kwon
> Raise the errors of protobuf's default backend for the values that PySpark
> sets in proto fields, as PySpark does
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61524
> URL: https://issues.apache.org/jira/browse/SPARK-61524
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The Python drop-in raises the errors of protobuf's pure-Python backend where
> PySpark sets a value in a proto field. A default install of pyspark-client
> uses protobuf's upb backend, whose errors and conversions differ. The
> pure-Python backend is used only when
> PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python is set.
> Under upb, official PySpark 4.3.0 behaves as below, with protobuf 7.36, which
> a fresh install of pyspark-client 4.3.0 resolves through grpcio-status:
> - A str that UTF-8 cannot encode (a lone surrogate), in a literal, an option,
> a fill, the runtime conf, a table name or the text of sql, raises the
> UnicodeEncodeError of its encoding: "'utf-8' codec can't encode character
> '\ud800' in position 1: surrogates not allowed". The drop-in raised a
> ValueError "... isn't a valid unicode string and can't be encoded in UTF-8."
> - Bytes in a string field are decoded as UTF-8, and bytes that are not UTF-8
> raise the UnicodeDecodeError of their decoding. This covers the keys of the
> runtime conf, of the builder's options and of the writers' options, the text
> of sql, the names of _joinAsOf's `on` and of replace's subset, and the values
> of the conf and of createTable's options. The drop-in refused bytes there, or
> raised a ValueError "... has type bytes, but isn't valid UTF-8 encoding".
> - Any other object in a string field raises TypeError "bad argument type for
> built-in operation". The drop-in raised "Cannot set
> spark.connect.KeyValue.value to ..." or "... has type ..., but expected one
> of: (<class 'bytes'>, <class 'str'>)".
> - A bool in the int fields of dropna's thresh and sampleBy's seed raises
> TypeError "Field spark.connect.NADrop.min_non_nulls: Expected an int, got a
> boolean." (protobuf 6.33 sent it as 1 or 0 with a DeprecationWarning). The
> drop-in raised the pure-Python backend's "Cannot set ... (int)."
> - StreamingQuery.explain(extended) reads its argument as a C long, and a
> NumPy bool by its truth. Other values raise the TypeError or OverflowError of
> that conversion, and a NumPy array raises "[1] has type ndarray, but expected
> one of: bool".
> Also as PySpark:
> - DataFrameReader.option and DataStreamReader.option assert, in the
> DataSource plan, that a key is a str: AssertionError.
> - The option of a writer or of DataStreamReader returns before it looks at
> the key when the value is None.
> - A dict-kept key that is unhashable raises its TypeError.
> - An as-of column of _joinAsOf that is not a Column or a str raises an
> AssertionError, before `on` is looked at.
> - conf.set(None, v) sends the empty key.
> - sql() checks `args` and formats the query object as it is before it sets
> the query, so bytes with keyword arguments raise "expected str, got bytes".
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]