Hyukjin Kwon created SPARK-61485:
------------------------------------
Summary: Pickle the methods of Row, VariantVal, Geometry and
Geography by reference to pyspark.sql.types, and read an Arrow stream of one
schema in the invalid-UTF-8 scan
Key: SPARK-61485
URL: https://issues.apache.org/jira/browse/SPARK-61485
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Follow-ups of a post-hoc review of SPARK-61402 and SPARK-61405, checked against
PySpark 4.3.0:
- A UDF that is, or captures, a method of Row, VariantVal, Geometry or
Geography (F.udf(Geography.getSrid, "int"), F.udf(VariantVal.toJson, "string"),
F.udf(variant.toJson, "string"), a UDF capturing Row.asDict) failed on the
server with PythonException ModuleNotFoundError: No module named
'pyspark._values' (or 'pyspark._row'). The extension loads these classes from
the embedded modules pyspark._values and pyspark._row, which set the classes'
__module__ to pyspark.sql.types but leave their methods' __module__ as the
embedded module, so cloudpickle pickled the methods by reference to a module
that exists on the client only. PySpark's methods are of pyspark.sql.types,
which the server's Python workers import. The methods now report
pyspark.sql.types.
- The invalid-UTF-8 scan of a result (first_invalid_utf8) took the column count
from the first batch and the field types from the last schema message, so a
stream with a second, narrower schema (which a valid Arrow IPC stream never
has) panicked with an index out of bounds. A stream of more than one schema now
does not read there, and the decoding error is reported as is.
- docs/compatibility.md did not describe SPARK-61405's changes: the strata that
sample_by rejects, the seed that it draws on the client, and the Value::Map
fill of fillna_value.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]