Tim Meehan created SPARK-61434:
----------------------------------
Summary: Raise the error of a converter before that of PyArrow in
createDataFrame of local rows, as PySpark does
Key: SPARK-61434
URL: https://issues.apache.org/jira/browse/SPARK-61434
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
createDataFrame of local rows in the Python client of spark-connect-rust raises
the error of the first bad value it meets, in column order. PySpark first runs
every value of every column through its converters
({{LocalDataToArrowConversion}}: the type assertions and the non-null checks)
and only then hands the columns to PyArrow, so the errors of PyArrow (type and
range) come last. For rows with two bad values, one of each kind, the clients
raise different errors. Both reject the data.
{code:python}
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
spark.createDataFrame([(128, 0)], "a tinyint, b timestamp")
{code}
Against an OSS Spark 4.3.0 Connect server on loopback (the error is raised in
the client, before any request):
* pyspark-client 4.3.0: {{AssertionError}}, for the {{0}}, which is not a
datetime.
* spark-connect-rust master 125654b: {{pyarrow.lib.ArrowInvalid: Value 128 too
large to fit in C integer type}}.
The same happens for the following, where spark-connect-rust raises the first
error and PySpark the second (the last row has one bad value and an Arrow type
that PyArrow cannot build):
|| rows || schema || pyspark-client 4.3.0 || spark-connect-rust master 125654b
||
| {{[(128, None)]}} | {{a tinyint, b int not null}} | {{PySparkValueError}}
input for IntegerType() must not be None | {{ArrowInvalid}} Value 128 too large
|
| {{[("abc", 0)]}} | {{a int, b timestamp}} | {{AssertionError}} |
{{ArrowInvalid}} Could not convert 'abc' |
| {{[(128, datetime(2020,1,1)), (1, 0)]}} | {{a tinyint, b timestamp}} |
{{AssertionError}} | {{ArrowInvalid}} |
| {{[([128], "x")]}} | {{a array<tinyint>, b decimal(5,2)}} |
{{AssertionError}} | {{ArrowInvalid}} |
| {{[((128, 0),)]}} | {{a struct<x:tinyint,y:timestamp>}} | {{AssertionError}}
| {{ArrowInvalid}} |
| {{[(128, None)]}} | {{a tinyint, b interval year to month}} |
{{PySparkTypeError}} UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION |
{{ArrowInvalid}} |
When the only bad values are the ones that PyArrow rejects, such as {{[(128,
"x")]}} for {{a tinyint, b int}}, both clients raise {{ArrowInvalid: Value 128
too large}}. A fuzzer of createDataFrame over 18,000 cases found 319 (1.8%)
with a different error class, in 17 pairs of classes; every case reduced to two
bad values.
>From source, at 125654b: {{infer::convert}}
>(crates/pyspark-rs/src/infer.rs:1151) does the work of both PySpark's
>converters and PyArrow's conversion of one value, and returns the first error
>of either. The caller loops over the columns and the rows and propagates that
>error at once (crates/pyspark-rs/src/session.rs:1005-1009). The arms of the
>types that PySpark has no converter of (boolean, the integers, float and
>double, day-time interval, date, time; infer.rs:1285 onward) raise the errors
>of PyArrow there, so a bad value in an early column hides a failed assertion
>in a later one. PySpark's {{convert}} (pyspark/sql/conversion.py,
>{{LocalDataToArrowConversion.convert}}) builds {{pylist}} with all the
>converters first, then {{to_arrow_schema}}, then {{pa.Table.from_arrays}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]