Tim Meehan created SPARK-61434:
----------------------------------

             Summary: Raise the error of a converter before that of PyArrow in 
createDataFrame of local rows, as PySpark does
                 Key: SPARK-61434
                 URL: https://issues.apache.org/jira/browse/SPARK-61434
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Tim Meehan


createDataFrame of local rows in the Python client of spark-connect-rust raises 
the error of the first bad value it meets, in column order. PySpark first runs 
every value of every column through its converters 
({{LocalDataToArrowConversion}}: the type assertions and the non-null checks) 
and only then hands the columns to PyArrow, so the errors of PyArrow (type and 
range) come last. For rows with two bad values, one of each kind, the clients 
raise different errors. Both reject the data.

{code:python}
from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
spark.createDataFrame([(128, 0)], "a tinyint, b timestamp")
{code}

Against an OSS Spark 4.3.0 Connect server on loopback (the error is raised in 
the client, before any request):
* pyspark-client 4.3.0: {{AssertionError}}, for the {{0}}, which is not a 
datetime.
* spark-connect-rust master 125654b: {{pyarrow.lib.ArrowInvalid: Value 128 too 
large to fit in C integer type}}.

The same happens for the following, where spark-connect-rust raises the first 
error and PySpark the second (the last row has one bad value and an Arrow type 
that PyArrow cannot build):
|| rows || schema || pyspark-client 4.3.0 || spark-connect-rust master 125654b 
||
| {{[(128, None)]}} | {{a tinyint, b int not null}} | {{PySparkValueError}} 
input for IntegerType() must not be None | {{ArrowInvalid}} Value 128 too large 
|
| {{[("abc", 0)]}} | {{a int, b timestamp}} | {{AssertionError}} | 
{{ArrowInvalid}} Could not convert 'abc' |
| {{[(128, datetime(2020,1,1)), (1, 0)]}} | {{a tinyint, b timestamp}} | 
{{AssertionError}} | {{ArrowInvalid}} |
| {{[([128], "x")]}} | {{a array<tinyint>, b decimal(5,2)}} | 
{{AssertionError}} | {{ArrowInvalid}} |
| {{[((128, 0),)]}} | {{a struct<x:tinyint,y:timestamp>}} | {{AssertionError}} 
| {{ArrowInvalid}} |
| {{[(128, None)]}} | {{a tinyint, b interval year to month}} | 
{{PySparkTypeError}} UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION | 
{{ArrowInvalid}} |

When the only bad values are the ones that PyArrow rejects, such as {{[(128, 
"x")]}} for {{a tinyint, b int}}, both clients raise {{ArrowInvalid: Value 128 
too large}}. A fuzzer of createDataFrame over 18,000 cases found 319 (1.8%) 
with a different error class, in 17 pairs of classes; every case reduced to two 
bad values.

>From source, at 125654b: {{infer::convert}} 
>(crates/pyspark-rs/src/infer.rs:1151) does the work of both PySpark's 
>converters and PyArrow's conversion of one value, and returns the first error 
>of either. The caller loops over the columns and the rows and propagates that 
>error at once (crates/pyspark-rs/src/session.rs:1005-1009). The arms of the 
>types that PySpark has no converter of (boolean, the integers, float and 
>double, day-time interval, date, time; infer.rs:1285 onward) raise the errors 
>of PyArrow there, so a bad value in an early column hides a failed assertion 
>in a later one. PySpark's {{convert}} (pyspark/sql/conversion.py, 
>{{LocalDataToArrowConversion.convert}}) builds {{pylist}} with all the 
>converters first, then {{to_arrow_schema}}, then {{pa.Table.from_arrays}}.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to