[
https://issues.apache.org/jira/browse/SPARK-61434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61434:
------------------------------------
Assignee: Tim Meehan
> Raise the error of a converter before that of PyArrow in createDataFrame of
> local rows, as PySpark does
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61434
> URL: https://issues.apache.org/jira/browse/SPARK-61434
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Tim Meehan
> Assignee: Tim Meehan
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> createDataFrame of local rows in the Python client of spark-connect-rust
> raises the error of the first bad value it meets, in column order. PySpark
> first runs every value of every column through its converters
> ({{LocalDataToArrowConversion}}: the type assertions and the non-null checks)
> and only then hands the columns to PyArrow, so the errors of PyArrow (type
> and range) come last. For rows with two bad values, one of each kind, the
> clients raise different errors. Both reject the data.
> {code:python}
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> spark.createDataFrame([(128, 0)], "a tinyint, b timestamp")
> {code}
> Against an OSS Spark 4.3.0 Connect server on loopback (the error is raised in
> the client, before any request):
> * pyspark-client 4.3.0: {{AssertionError}}, for the {{0}}, which is not a
> datetime.
> * spark-connect-rust master 125654b: {{pyarrow.lib.ArrowInvalid: Value 128
> too large to fit in C integer type}}.
> The same happens for the following, where spark-connect-rust raises the first
> error and PySpark the second (the last row has one bad value and an Arrow
> type that PyArrow cannot build):
> || rows || schema || pyspark-client 4.3.0 || spark-connect-rust master
> 125654b ||
> | {{[(128, None)]}} | {{a tinyint, b int not null}} | {{PySparkValueError}}
> input for IntegerType() must not be None | {{ArrowInvalid}} Value 128 too
> large |
> | {{[("abc", 0)]}} | {{a int, b timestamp}} | {{AssertionError}} |
> {{ArrowInvalid}} Could not convert 'abc' |
> | {{[(128, datetime(2020,1,1)), (1, 0)]}} | {{a tinyint, b timestamp}} |
> {{AssertionError}} | {{ArrowInvalid}} |
> | {{[([128], "x")]}} | {{a array<tinyint>, b decimal(5,2)}} |
> {{AssertionError}} | {{ArrowInvalid}} |
> | {{[((128, 0),)]}} | {{a struct<x:tinyint,y:timestamp>}} |
> {{AssertionError}} | {{ArrowInvalid}} |
> | {{[(128, None)]}} | {{a tinyint, b interval year to month}} |
> {{PySparkTypeError}} UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION |
> {{ArrowInvalid}} |
> When the only bad values are the ones that PyArrow rejects, such as {{[(128,
> "x")]}} for {{a tinyint, b int}}, both clients raise {{ArrowInvalid: Value
> 128 too large}}. A fuzzer of createDataFrame over 18,000 cases found 319
> (1.8%) with a different error class, in 17 pairs of classes; every case
> reduced to two bad values.
> From source, at 125654b: {{infer::convert}}
> (crates/pyspark-rs/src/infer.rs:1151) does the work of both PySpark's
> converters and PyArrow's conversion of one value, and returns the first error
> of either. The caller loops over the columns and the rows and propagates that
> error at once (crates/pyspark-rs/src/session.rs:1005-1009). The arms of the
> types that PySpark has no converter of (boolean, the integers, float and
> double, day-time interval, date, time; infer.rs:1285 onward) raise the errors
> of PyArrow there, so a bad value in an early column hides a failed assertion
> in a later one. PySpark's {{convert}} (pyspark/sql/conversion.py,
> {{LocalDataToArrowConversion.convert}}) builds {{pylist}} with all the
> converters first, then {{to_arrow_schema}}, then {{pa.Table.from_arrays}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]