[ 
https://issues.apache.org/jira/browse/SPARK-61485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61485:
-----------------------------------
    Labels: pull-request-available  (was: )

> Pickle the methods of Row, VariantVal, Geometry and Geography by reference to 
> pyspark.sql.types, and read an Arrow stream of one schema in the 
> invalid-UTF-8 scan
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61485
>                 URL: https://issues.apache.org/jira/browse/SPARK-61485
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Follow-ups of a post-hoc review of SPARK-61402 and SPARK-61405, checked 
> against PySpark 4.3.0:
> - A UDF that is, or captures, a method of Row, VariantVal, Geometry or 
> Geography (F.udf(Geography.getSrid, "int"), F.udf(VariantVal.toJson, 
> "string"), F.udf(variant.toJson, "string"), a UDF capturing Row.asDict) 
> failed on the server with PythonException ModuleNotFoundError: No module 
> named 'pyspark._values' (or 'pyspark._row'). The extension loads these 
> classes from the embedded modules pyspark._values and pyspark._row, which set 
> the classes' __module__ to pyspark.sql.types but leave their methods' 
> __module__ as the embedded module, so cloudpickle pickled the methods by 
> reference to a module that exists on the client only. PySpark's methods are 
> of pyspark.sql.types, which the server's Python workers import. The methods 
> now report pyspark.sql.types.
> - The invalid-UTF-8 scan of a result (first_invalid_utf8) took the column 
> count from the first batch and the field types from the last schema message, 
> so a stream with a second, narrower schema (which a valid Arrow IPC stream 
> never has) panicked with an index out of bounds. A stream of more than one 
> schema now does not read there, and the decoding error is reported as is.
> - docs/compatibility.md did not describe SPARK-61405's changes: the strata 
> that sample_by rejects, the seed that it draws on the client, and the 
> Value::Map fill of fillna_value.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to