Hyukjin Kwon created SPARK-61524:
------------------------------------

             Summary: Raise the errors of protobuf's default backend for the 
values that PySpark sets in string fields, and take a bool in an int field, as 
PySpark does
                 Key: SPARK-61524
                 URL: https://issues.apache.org/jira/browse/SPARK-61524
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


The Python drop-in raises the errors of protobuf's pure-Python backend where 
PySpark sets a value in a proto field. A default install of pyspark-client uses 
protobuf's upb backend, whose errors and conversions differ. The pure-Python 
backend is used only when PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python is set.

Under upb (protobuf 6.33), official PySpark 4.3.0:
- A str that UTF-8 cannot encode (a lone surrogate), in a literal, an option, a 
fill, the runtime conf, a table name or the text of sql, raises the 
UnicodeEncodeError of its encoding: "'utf-8' codec can't encode character 
'\ud800' in position 1: surrogates not allowed". The drop-in raised a 
ValueError "... isn't a valid unicode string and can't be encoded in UTF-8."
- Bytes in a string field are decoded as UTF-8, and bytes that are not UTF-8 
raise the UnicodeDecodeError of their decoding. This covers the keys of the 
runtime conf, of the builder's options and of the writers' options, the text of 
sql, the names of _joinAsOf's `on` and of replace's subset, and the values of 
the conf and of createTable's options. The drop-in refused bytes there, or 
raised a ValueError "... has type bytes, but isn't valid UTF-8 encoding".
- Any other object in a string field raises TypeError "bad argument type for 
built-in operation". The drop-in raised "Cannot set 
spark.connect.KeyValue.value to ..." or "... has type ..., but expected one of: 
(<class 'bytes'>, <class 'str'>)".
- A bool in the int fields of dropna's thresh and sampleBy's seed is sent as 1 
or 0, with a DeprecationWarning that the default filters hide. The drop-in 
raised a TypeError.
- StreamingQuery.explain(extended) reads its argument as a C long, and a NumPy 
bool by its truth. Other values raise the TypeError or OverflowError of that 
conversion, and a NumPy array raises "[1] has type ndarray, but expected one 
of: bool".

Also as PySpark:
- DataFrameReader.option and DataStreamReader.option assert, in the DataSource 
plan, that a key is a str: AssertionError.
- The option of a writer or of DataStreamReader returns before it looks at the 
key when the value is None.
- A dict-kept key that is unhashable raises its TypeError.
- An as-of column of _joinAsOf that is not a Column or a str raises an 
AssertionError, before `on` is looked at.
- conf.set(None, v) sends the empty key.
- sql() checks `args` and formats the query object as it is before it sets the 
query, so bytes with keyword arguments raise "expected str, got bytes".




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to