Hyukjin Kwon created SPARK-61524:
------------------------------------
Summary: Raise the errors of protobuf's default backend for the
values that PySpark sets in string fields, and take a bool in an int field, as
PySpark does
Key: SPARK-61524
URL: https://issues.apache.org/jira/browse/SPARK-61524
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The Python drop-in raises the errors of protobuf's pure-Python backend where
PySpark sets a value in a proto field. A default install of pyspark-client uses
protobuf's upb backend, whose errors and conversions differ. The pure-Python
backend is used only when PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python is set.
Under upb (protobuf 6.33), official PySpark 4.3.0:
- A str that UTF-8 cannot encode (a lone surrogate), in a literal, an option, a
fill, the runtime conf, a table name or the text of sql, raises the
UnicodeEncodeError of its encoding: "'utf-8' codec can't encode character
'\ud800' in position 1: surrogates not allowed". The drop-in raised a
ValueError "... isn't a valid unicode string and can't be encoded in UTF-8."
- Bytes in a string field are decoded as UTF-8, and bytes that are not UTF-8
raise the UnicodeDecodeError of their decoding. This covers the keys of the
runtime conf, of the builder's options and of the writers' options, the text of
sql, the names of _joinAsOf's `on` and of replace's subset, and the values of
the conf and of createTable's options. The drop-in refused bytes there, or
raised a ValueError "... has type bytes, but isn't valid UTF-8 encoding".
- Any other object in a string field raises TypeError "bad argument type for
built-in operation". The drop-in raised "Cannot set
spark.connect.KeyValue.value to ..." or "... has type ..., but expected one of:
(<class 'bytes'>, <class 'str'>)".
- A bool in the int fields of dropna's thresh and sampleBy's seed is sent as 1
or 0, with a DeprecationWarning that the default filters hide. The drop-in
raised a TypeError.
- StreamingQuery.explain(extended) reads its argument as a C long, and a NumPy
bool by its truth. Other values raise the TypeError or OverflowError of that
conversion, and a NumPy array raises "[1] has type ndarray, but expected one
of: bool".
Also as PySpark:
- DataFrameReader.option and DataStreamReader.option assert, in the DataSource
plan, that a key is a str: AssertionError.
- The option of a writer or of DataStreamReader returns before it looks at the
key when the value is None.
- A dict-kept key that is unhashable raises its TypeError.
- An as-of column of _joinAsOf that is not a Column or a str raises an
AssertionError, before `on` is looked at.
- conf.set(None, v) sends the empty key.
- sql() checks `args` and formats the query object as it is before it sets the
query, so bytes with keyword arguments raise "expected str, got bytes".
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]