HyukjinKwon opened a new pull request, #208: URL: https://github.com/apache/spark-connect-rust/pull/208
### What changes were proposed in this pull request? This replaces #144 by DB Tsai, which does not allow edits by maintainers, with its commits plus the merge with master and the review changes below. It is stacked on #207 (SPARK-60727), with which it shares the inline IPC bridge and the dispatcher; until #207 merges, its diff includes #207's. `createDataFrame(pandas_df, schema)`, for a pandas DataFrame of at least 10,000 rows whose columns are all object columns of exact `str` values and `None`, with an explicit `StructType` of `StringType` (UTF8_BINARY) fields, now encodes the Arrow IPC in Rust (`_pandas_string_ipc` in `crates/pyspark-rs/src/pandas_input.rs`) from the NumPy object arrays, and sends it inline as the row path does, instead of materializing Python rows. Everything else keeps the row path, with its errors: subclasses of `str`, other objects (including `NaN`, `pd.NA` and numbers), `None` in a non-nullable field, strings with lone surrogates (which cannot be encoded to UTF-8), other collations, smaller frames, other dtypes and schemas, and customized pandas or `pyspark.sql.types` classes, as for #121 (SPARK-60676). Review changes on top of #144: - `_pandas_string_ipc` encoded every string to UTF-8 twice, in a validation pass and while building; it now builds in one pass and falls back on the first string that cannot be encoded. - Dropped the checks of the aliases of the Arrow importer and of `pyspark.sql.pandas.types`: the path calls the Rust encoder with the native schema and never calls them. The two tests that only exercised them went with them. - A test that required a second row conversion to read `Index._data` exactly as often as the first now checks that the path declines without more reads than the row path, as for #121. ### Why are the changes needed? [SPARK-60720](https://issues.apache.org/jira/browse/SPARK-60720): converting each string through Python rows dominates `createDataFrame` of large text frames. Measured on macOS against a local Spark 4.3.0 server, as the client CPU time of `createDataFrame` of 200,000 rows of three string columns (ASCII, empty, NUL, accents, CJK, emoji, combining marks and nulls) with an explicit schema (median of 5): 216 ms on the current master, 60 ms with #144 as it was, and 43 ms with this change (official PySpark 4.3.0: 51 ms). All give the same rows. ### Does this PR introduce _any_ user-facing change? No, other than speed. ### How was this patch tested? - New tests in `python/tests/test_arrow_handoff.py`, against the mock Arrow input service, check every IPC cell (Unicode, NUL, empty strings, nulls, all-null columns), field names, nullability and metadata, the inline transport with any relation-cache setting, that no configuration is read, that surrogates fall back before any configuration read, and that each unsupported or customized input keeps the row path with its errors. - Against the current master, merged into #207 and this branch, and a local Spark 4.3.0 server: the lib and every integration test of `apache-spark-connect` and `apache-spark-connect-core`, e2e included (789 passed), the build of all tests with `wasm-udf`, the offline Python suites (1,443 passed, 9 skipped), the live drop-in and Arrow suites (83 passed) and the live cases (10 passed). `cargo fmt --all --check` passes. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6); the merge with master, the review changes and the measurement by Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
