[
https://issues.apache.org/jira/browse/SPARK-62015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-62015.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 858
[https://github.com/apache/spark-connect-rust/pull/858]
> Convert the local rows of createDataFrame with the type objects of the
> schema, inferred too, and read tuple rows by their len and iter, as PySpark
> does
> -------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62015
> URL: https://issues.apache.org/jira/browse/SPARK-62015
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark 4.3.0's createDataFrame of local rows converts them with converters
> of the type objects of
> the schema (LocalDataToArrowConversion of python/pyspark/sql/conversion.py),
> and reads the rows
> with the Python protocol. The drop-in's conversion
> (crates/pyspark-rs/src/infer.rs, and
> create_from_local of crates/pyspark-rs/src/session.rs) differs in three ways:
> 1. The values of a user-defined type of an inferred schema. PySpark's
> _infer_type returns
> obj.__UDT__ (types.py:2654-2655), _merge_type keeps the first of two UDTs
> of one class
> (types.py:2921-2974), and the converter serializes every value with that
> object
> (conversion.py:846-877). The drop-in serializes each value with its own
> __UDT__ (udt_of), as the
> core type of an inferred schema holds no object. Values whose __UDT__ has
> other parameters, such
> as a scale, send other data, without an error: in a field, an array, a
> map, a struct, and the
> SQL type of another UDT.
> 2. Rows of a tuple subclass of its own __len__ or __iter__. PySpark's to_row
> takes a tuple by
> len(item), which must be the number of fields, and tuple(item)
> (conversion.py:944-973);
> _infer_schema names a tuple by len(row) and zips the names with row
> (types.py:2810-2821); the
> converter of a struct reads a nested tuple by len(value) and value[i]
> (conversion.py:620-640).
> The drop-in reads the values as the tuple stores them (row_values,
> convert_struct), while its
> inference iterates the row. Such a row sends its storage, a __len__ of
> another length or that
> raises raises nothing, and an __iter__ of values of other types than the
> storage fails with
> ArrowInvalid.
> 3. The type objects of a given schema (KD-types C7). PySpark's converters
> name the type object in
> their errors (f"input for {dataType} must not be None",
> conversion.py:599-925), and the
> converter of a spatial type calls dataType.toInternal(value)
> (conversion.py:893-916). The
> drop-in names its core type (BooleanType() of a subclass SubBool of
> BooleanType), and does not
> call the toInternal of a subclass of GeometryType or GeographyType.
> Repro (master 50594b2e, against a Spark 4.3.0 server)
> class ScaleUDT(UserDefinedType):
> def __init__(self, scale=1): self.scale = scale
> @classmethod
> def sqlType(cls): return LongType()
> @classmethod
> def module(cls): return __name__
> def serialize(self, obj): return obj.v * self.scale
> def deserialize(self, datum): return datum
> class Val:
> def __init__(self, v, s): self.v, self.s = v, s
> @property
> def __UDT__(self): return ScaleUDT(self.s)
> class T(tuple):
> def __iter__(self): return iter((99, "z"))
> class Len3(tuple):
> def __len__(self): return 3
> class SubBool(BooleanType): pass
> >>> spark.createDataFrame([(Val(3, 2),), (Val(4,
> 10),)]).toArrow().to_pylist()
> [{'_1': 6}, {'_1': 40}]
> >>> spark.createDataFrame([T((1, "a"))], "a long, b string").collect()
> [Row(a=1, b='a')]
> >>> spark.createDataFrame([Len3((1, "a"))], "a long, b string").collect()
> [Row(a=1, b='a')]
> >>> spark.createDataFrame([(None,)], StructType([StructField("a",
> SubBool(), False)]))
> PySparkValueError: input for BooleanType() must not be None
> A GeometryType subclass whose toInternal returns another dict sends the
> value's own SRID and WKB,
> and its toInternal is not called.
> Expected (as with pyspark-client 4.3.0)
> [{'_1': 6}, {'_1': 8}]
> [Row(a=99, b='z')]
> PySparkValueError: [AXIS_LENGTH_MISMATCH] Length mismatch: Expected axis
> has 2 elements, new
> values have 3 elements.
> PySparkValueError: input for SubBool() must not be None
> and the toInternal of the GeometryType subclass is called, whose result is
> sent.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]