HyukjinKwon opened a new pull request, #249: URL: https://github.com/apache/spark-connect-rust/pull/249
### What changes were proposed in this pull request? This replaces #145 by DB Tsai, which does not allow edits by maintainers, with its commits plus the merge with master and the review changes below. `createDataFrame(pandas_df, schema)`, for a pandas DataFrame of at least 10,000 rows whose columns are all kept in pandas' Arrow string storage (`StringDtype` with `pyarrow` storage, as `string[pyarrow]`, including the NumPy-semantics variant of pandas 2.2 and the `str` dtype of pandas 3), with an explicit `StructType` of `StringType` (UTF8_BINARY) fields, now passes that storage to the Arrow importer as Arrow string columns (`_arrow_string_table` in `python/pyspark/sql/_pandas_input.py`), instead of materializing Python rows. pandas converts such a column for the row path from that same storage, with its missing values as `None`, so the values are the same. Everything else keeps the row path: `string[python]` storage, mixed dtypes, a null in a non-nullable field, subclasses of pandas' Arrow string arrays, smaller frames, other schemas, and the cases that the guards of #208 (SPARK-60720) already decline. Review changes on top of #145: - The storage is read through pandas' public API (`Series.array`, `__arrow_array__`), and only pandas' own array classes are trusted, instead of walking pandas' block manager (`_mgr`, `iget_values`, `blknos`, `blklocs`) and reading `_pa_array` behind about 200 lines of checks of pandas internals. - The table goes to the Arrow importer, as the int64 and date tables do, so it is uploaded as cached artifacts at `spark.sql.session.localRelationCacheThreshold`; #145 sent it inline, and a large inline relation fails (an HTTP/2 error above about 128 MB). - It works on pandas 2.2, the lower bound of the supported range, whose `StringDtype` has a `pyarrow_numpy` storage instead of `na_value`. `test_arrow_string_input.py`, which required pandas 2.3, is replaced by tests in `test_arrow_handoff.py`. - The architecture docs describe the path. ### Why are the changes needed? [SPARK-60745](https://issues.apache.org/jira/browse/SPARK-60745): converting Arrow-backed strings into Python rows and back to Arrow dominates `createDataFrame` of such frames. Measured on macOS against a local Spark 4.3.0 server, as the client CPU time of `createDataFrame` of 200,000 rows of three `string[pyarrow]` columns (ASCII, empty, NUL, accents, CJK, emoji, combining marks and nulls) with an explicit schema (median of 5): 266 ms on the current master, 26 ms with this change (official PySpark 4.3.0: 23 ms). All give the same rows. ### Does this PR introduce _any_ user-facing change? No, other than speed. ### How was this patch tested? - New tests in `python/tests/test_arrow_handoff.py`, against the mock Arrow input service: every cell, field name, nullability and metadata of `string[pyarrow]`, NumPy-semantics, `large_string`-backed and sliced columns, inline below the relation cache threshold and uploaded at it, without materializing rows; the same table as the row path; and each unsupported input keeps the row path. - Against the current master and a local Spark 4.3.0 server: the full suites (results below the merge). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6); the merge with master, the review changes and the measurement by Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
