[ 
https://issues.apache.org/jira/browse/SPARK-62015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-62015:
------------------------------------

    Assignee: Hyukjin Kwon

> Convert the local rows of createDataFrame with the type objects of the 
> schema, inferred too, and read tuple rows by their len and iter, as PySpark 
> does
> -------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62015
>                 URL: https://issues.apache.org/jira/browse/SPARK-62015
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Problem
> PySpark 4.3.0's createDataFrame of local rows converts them with converters 
> of the type objects of
> the schema (LocalDataToArrowConversion of python/pyspark/sql/conversion.py), 
> and reads the rows
> with the Python protocol. The drop-in's conversion 
> (crates/pyspark-rs/src/infer.rs, and
> create_from_local of crates/pyspark-rs/src/session.rs) differs in three ways:
> 1. The values of a user-defined type of an inferred schema. PySpark's 
> _infer_type returns
>    obj.__UDT__ (types.py:2654-2655), _merge_type keeps the first of two UDTs 
> of one class
>    (types.py:2921-2974), and the converter serializes every value with that 
> object
>    (conversion.py:846-877). The drop-in serializes each value with its own 
> __UDT__ (udt_of), as the
>    core type of an inferred schema holds no object. Values whose __UDT__ has 
> other parameters, such
>    as a scale, send other data, without an error: in a field, an array, a 
> map, a struct, and the
>    SQL type of another UDT.
> 2. Rows of a tuple subclass of its own __len__ or __iter__. PySpark's to_row 
> takes a tuple by
>    len(item), which must be the number of fields, and tuple(item) 
> (conversion.py:944-973);
>    _infer_schema names a tuple by len(row) and zips the names with row 
> (types.py:2810-2821); the
>    converter of a struct reads a nested tuple by len(value) and value[i] 
> (conversion.py:620-640).
>    The drop-in reads the values as the tuple stores them (row_values, 
> convert_struct), while its
>    inference iterates the row. Such a row sends its storage, a __len__ of 
> another length or that
>    raises raises nothing, and an __iter__ of values of other types than the 
> storage fails with
>    ArrowInvalid.
> 3. The type objects of a given schema (KD-types C7). PySpark's converters 
> name the type object in
>    their errors (f"input for {dataType} must not be None", 
> conversion.py:599-925), and the
>    converter of a spatial type calls dataType.toInternal(value) 
> (conversion.py:893-916). The
>    drop-in names its core type (BooleanType() of a subclass SubBool of 
> BooleanType), and does not
>    call the toInternal of a subclass of GeometryType or GeographyType.
> Repro (master 50594b2e, against a Spark 4.3.0 server)
>   class ScaleUDT(UserDefinedType):
>       def __init__(self, scale=1): self.scale = scale
>       @classmethod
>       def sqlType(cls): return LongType()
>       @classmethod
>       def module(cls): return __name__
>       def serialize(self, obj): return obj.v * self.scale
>       def deserialize(self, datum): return datum
>   class Val:
>       def __init__(self, v, s): self.v, self.s = v, s
>       @property
>       def __UDT__(self): return ScaleUDT(self.s)
>   class T(tuple):
>       def __iter__(self): return iter((99, "z"))
>   class Len3(tuple):
>       def __len__(self): return 3
>   class SubBool(BooleanType): pass
>   >>> spark.createDataFrame([(Val(3, 2),), (Val(4, 
> 10),)]).toArrow().to_pylist()
>   [{'_1': 6}, {'_1': 40}]
>   >>> spark.createDataFrame([T((1, "a"))], "a long, b string").collect()
>   [Row(a=1, b='a')]
>   >>> spark.createDataFrame([Len3((1, "a"))], "a long, b string").collect()
>   [Row(a=1, b='a')]
>   >>> spark.createDataFrame([(None,)], StructType([StructField("a", 
> SubBool(), False)]))
>   PySparkValueError: input for BooleanType() must not be None
> A GeometryType subclass whose toInternal returns another dict sends the 
> value's own SRID and WKB,
> and its toInternal is not called.
> Expected (as with pyspark-client 4.3.0)
>   [{'_1': 6}, {'_1': 8}]
>   [Row(a=99, b='z')]
>   PySparkValueError: [AXIS_LENGTH_MISMATCH] Length mismatch: Expected axis 
> has 2 elements, new
>   values have 3 elements.
>   PySparkValueError: input for SubBool() must not be None
> and the toInternal of the GeometryType subclass is called, whose result is 
> sent.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to