[ 
https://issues.apache.org/jira/browse/SPARK-61434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61434:
-----------------------------------
    Labels: pull-request-available  (was: )

> Raise the error of a converter before that of PyArrow in createDataFrame of 
> local rows, as PySpark does
> -------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61434
>                 URL: https://issues.apache.org/jira/browse/SPARK-61434
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Tim Meehan
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> createDataFrame of local rows in the Python client of spark-connect-rust 
> raises the error of the first bad value it meets, in column order. PySpark 
> first runs every value of every column through its converters 
> ({{LocalDataToArrowConversion}}: the type assertions and the non-null checks) 
> and only then hands the columns to PyArrow, so the errors of PyArrow (type 
> and range) come last. For rows with two bad values, one of each kind, the 
> clients raise different errors. Both reject the data.
> {code:python}
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> spark.createDataFrame([(128, 0)], "a tinyint, b timestamp")
> {code}
> Against an OSS Spark 4.3.0 Connect server on loopback (the error is raised in 
> the client, before any request):
> * pyspark-client 4.3.0: {{AssertionError}}, for the {{0}}, which is not a 
> datetime.
> * spark-connect-rust master 125654b: {{pyarrow.lib.ArrowInvalid: Value 128 
> too large to fit in C integer type}}.
> The same happens for the following, where spark-connect-rust raises the first 
> error and PySpark the second (the last row has one bad value and an Arrow 
> type that PyArrow cannot build):
> || rows || schema || pyspark-client 4.3.0 || spark-connect-rust master 
> 125654b ||
> | {{[(128, None)]}} | {{a tinyint, b int not null}} | {{PySparkValueError}} 
> input for IntegerType() must not be None | {{ArrowInvalid}} Value 128 too 
> large |
> | {{[("abc", 0)]}} | {{a int, b timestamp}} | {{AssertionError}} | 
> {{ArrowInvalid}} Could not convert 'abc' |
> | {{[(128, datetime(2020,1,1)), (1, 0)]}} | {{a tinyint, b timestamp}} | 
> {{AssertionError}} | {{ArrowInvalid}} |
> | {{[([128], "x")]}} | {{a array<tinyint>, b decimal(5,2)}} | 
> {{AssertionError}} | {{ArrowInvalid}} |
> | {{[((128, 0),)]}} | {{a struct<x:tinyint,y:timestamp>}} | 
> {{AssertionError}} | {{ArrowInvalid}} |
> | {{[(128, None)]}} | {{a tinyint, b interval year to month}} | 
> {{PySparkTypeError}} UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION | 
> {{ArrowInvalid}} |
> When the only bad values are the ones that PyArrow rejects, such as {{[(128, 
> "x")]}} for {{a tinyint, b int}}, both clients raise {{ArrowInvalid: Value 
> 128 too large}}. A fuzzer of createDataFrame over 18,000 cases found 319 
> (1.8%) with a different error class, in 17 pairs of classes; every case 
> reduced to two bad values.
> From source, at 125654b: {{infer::convert}} 
> (crates/pyspark-rs/src/infer.rs:1151) does the work of both PySpark's 
> converters and PyArrow's conversion of one value, and returns the first error 
> of either. The caller loops over the columns and the rows and propagates that 
> error at once (crates/pyspark-rs/src/session.rs:1005-1009). The arms of the 
> types that PySpark has no converter of (boolean, the integers, float and 
> double, day-time interval, date, time; infer.rs:1285 onward) raise the errors 
> of PyArrow there, so a bad value in an early column hides a failed assertion 
> in a later one. PySpark's {{convert}} (pyspark/sql/conversion.py, 
> {{LocalDataToArrowConversion.convert}}) builds {{pylist}} with all the 
> converters first, then {{to_arrow_schema}}, then {{pa.Table.from_arrays}}.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to