HyukjinKwon opened a new pull request, #326: URL: https://github.com/apache/spark-connect-rust/pull/326
### What changes were proposed in this pull request? [SPARK-61133](https://issues.apache.org/jira/browse/SPARK-61133): `pyspark.sql.types` of the client gains the private helpers of PySpark's, as of Apache Spark v4.3.0: - the inference, merging, conversion and verification of the types of Python objects: `_infer_type`, `_infer_schema`, `_merge_type`, `_has_nulltype`, `_need_converter`, `_create_converter`, `_make_type_verifier`, `_create_row_inbound_converter` and `_int_size_to_type` - their tables: `_type_mappings`, `_array_type_mappings`, `_array_signed_int_typecode_ctype_mappings`, `_array_unsigned_int_typecode_ctype_mappings`, `_acceptable_types`, `_atomic_types`, `_complex_types`, `_all_complex_types`, `_all_mappable_types` and `JVM_INT_MAX` They are PySpark's code, since the client's types share the classes and the attributes of PySpark's. There is one change: the client's `Row` is not a tuple. So wherever PySpark takes a tuple of a `Row`, these helpers take the client's `Row` too, in five places: - `_infer_schema` - the converter of a struct in `_create_converter` - the acceptable types of a struct and of a variant - the verifier of a struct The client's own `createDataFrame` still infers and converts in Rust, as before. ### Why are the changes needed? PySpark Connect's `session.py` imports `_infer_schema`, `_has_nulltype` and `_merge_type`. Apache Spark's `test_types.py` imports `_infer_type`, `_make_type_verifier`, `_merge_type` and the typecode tables, as do some libraries. On this client, `from pyspark.sql.types import _merge_type` raised `ImportError`, so Apache Spark's `test_parity_types.py` could not even be collected. ### Does this PR introduce _any_ user-facing change? Yes. These helpers can be imported from `pyspark.sql.types`, and behave as PySpark's. ### How was this patch tested? - New offline tests check: - the helpers and their tables - the verifiers of each type, with the success and failure cases of Apache Spark's `DataTypeVerificationTests`, also of the client's `Row`, and their errors with the names of the fields - the inference of each type of value, also nested and of the options, the merging of types, and the converters They pass with official PySpark 4.3.0 and fail without this change. With them, the coverage of the offline suites stays at 83%. - Apache Spark's `test_parity_types.py` now runs on this client against a local server. 61 tests pass and 21 are skipped. Of the 9 that fail, one fails with official PySpark too. The other 8 are gaps of the types that the file could not show before, now filed: - SPARK-61134: the fields of the interval types, and the arguments of the types without parameters - SPARK-61135: the JSON of `GeometryType` and `GeographyType` - SPARK-61136: the decimals of a negative scale - SPARK-61137: the `StructField` objects of a `StructType` - SPARK-61138: `str` of a `VariantVal`, and rows that are `VariantVal`s - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2198 passed, 8 skipped), and the live drop-in and Arrow suites (181 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
