Hyukjin Kwon created SPARK-61485:
------------------------------------

             Summary: Pickle the methods of Row, VariantVal, Geometry and 
Geography by reference to pyspark.sql.types, and read an Arrow stream of one 
schema in the invalid-UTF-8 scan
                 Key: SPARK-61485
                 URL: https://issues.apache.org/jira/browse/SPARK-61485
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Follow-ups of a post-hoc review of SPARK-61402 and SPARK-61405, checked against 
PySpark 4.3.0:
- A UDF that is, or captures, a method of Row, VariantVal, Geometry or 
Geography (F.udf(Geography.getSrid, "int"), F.udf(VariantVal.toJson, "string"), 
F.udf(variant.toJson, "string"), a UDF capturing Row.asDict) failed on the 
server with PythonException ModuleNotFoundError: No module named 
'pyspark._values' (or 'pyspark._row'). The extension loads these classes from 
the embedded modules pyspark._values and pyspark._row, which set the classes' 
__module__ to pyspark.sql.types but leave their methods' __module__ as the 
embedded module, so cloudpickle pickled the methods by reference to a module 
that exists on the client only. PySpark's methods are of pyspark.sql.types, 
which the server's Python workers import. The methods now report 
pyspark.sql.types.
- The invalid-UTF-8 scan of a result (first_invalid_utf8) took the column count 
from the first batch and the field types from the last schema message, so a 
stream with a second, narrower schema (which a valid Arrow IPC stream never 
has) panicked with an index out of bounds. A stream of more than one schema now 
does not read there, and the decoding error is reported as is.
- docs/compatibility.md did not describe SPARK-61405's changes: the strata that 
sample_by rejects, the seed that it draws on the client, and the Value::Map 
fill of fillna_value.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to