HyukjinKwon opened a new pull request, #304: URL: https://github.com/apache/spark-connect-rust/pull/304
### What changes were proposed in this pull request? [SPARK-61106](https://issues.apache.org/jira/browse/SPARK-61106): `spark.createDataFrame` of local rows with a schema now converts their values, and raises their errors, as PyArrow does in PySpark Connect. PySpark Connect passes the values of most atomic types to PyArrow as they are. The client converts them in Rust (`crates/pyspark-rs/src/infer.rs`), emulating PyArrow. This change makes the emulation follow PyArrow's code: - An int, date, day-time interval or time field takes a value as PyArrow's `CIntFromPython` does: - an int, else the `__index__` of the value, else its `__int__` (`PyObjectToPyInt`), such as of a `Decimal`, a float or a NumPy scalar - a bool raises `ArrowTypeError` ("Expected integer, got bool") - a value out of range raises `ArrowInvalid` ("Value X too large to fit in C integer type", or "Integer too large for date32") - A NumPy scalar in a day-time interval field is taken only as a `timedelta64[us]`, as PyArrow's `duration[us]` takes it. Another NumPy scalar raises `ArrowNotImplementedError`. - The errors are PyArrow's classes, `ArrowInvalid`, `ArrowTypeError` and `ArrowNotImplementedError`, with PyArrow's messages: - "tried to convert to float32" for a float field - "PyLong is too large to fit int64" for an int beyond int64 in a float or double field - "was not a utf8 string" for bytes of a string field - Without PyArrow, they are the built-in classes these derive from. - A date field takes the days of any int32, as PyArrow does; it no longer stops at the dates of Python. `collect` turns days outside those dates into `date(1970, 1, 1) + timedelta(days=d)`, as PyArrow's `Date32Scalar.as_py` does. That raises its `OverflowError`: "date value out of range", or "days=1000000000; must have magnitude <= 999999999". - A `char` or `varchar` field raises `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION`, after the values are converted, as PySpark's `to_arrow_schema` does: it has no Arrow type for these types. ### Why are the changes needed? Of 374 cases of a value of each kind in a field of each atomic type, 119 differed from PySpark 4.3.0 on a local Spark 4.3.0 server. For example: | Input | master | This change and PySpark 4.3.0 | | --- | --- | --- | | `Decimal("1.5")` into `int`, `date` or `interval day to second` | `ValueError` or `ArrowTypeError` | 1, `1970-01-02`, 1 microsecond | | `"x"` into `int` | `ValueError: Could not convert 'x' ...` | `pyarrow.lib.ArrowInvalid: Could not convert 'x' ...` | | `True` into `int` | `TypeError` | `ArrowTypeError` | | `"x"` into `float` | `... tried to convert to float` | `... tried to convert to float32` | | `2**70` into `double` | `ValueError: Integer value ... is outside of the range ...` | `ArrowInvalid: PyLong is too large to fit int64` | | `2**40` into `date` | `OverflowError: days=...` | `ArrowInvalid: Integer too large for date32` | | `2932897` days into `date` | `OverflowError` in `createDataFrame` | created; `OverflowError: date value out of range` when collected | | `True` or `NaN` into `time` | taken | `ArrowTypeError`, `ValueError` | | bytes, or an int, into `char(3)` | the server's `UNSUPPORTED_CHAR_OR_VARCHAR_AS_STRING`, or `TypeError` | `UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` | ### Does this PR introduce _any_ user-facing change? Yes. The values of local rows convert, or fail, as in PySpark 4.3.0. ### How was this patch tested? - A new live test checks: - a Decimal in an int and a date field - the `ArrowInvalid` of a str in an int field - the error of a char field - the error of a date out of the dates of Python It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks the values sent for a Decimal in each kind of field and for a `timedelta64[us]`, the errors and their exact classes, and the error of char and varchar fields. - The test of ints in date fields expected the errors of days outside the dates of Python from `createDataFrame`. It now expects the days to be sent, as PySpark does. - On a local server, 401 cases give the same results as official PySpark 4.3.0: 374 of a value of each kind into each atomic type, and 27 of char, varchar and nested fields, NumPy scalars and edge values. The only differences are the addresses of `object()`s in the messages. - The rows of the client convert their values when they are read. So the `OverflowError` of a date outside the dates of Python is raised when the value is read, where PySpark raises it in `collect`. - The lib tests of both crates (588 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2054 passed, 12 skipped), and the live drop-in and Arrow suites (158 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
