HyukjinKwon commented on PR #145: URL: https://github.com/apache/spark-connect-rust/pull/145#issuecomment-6683196169
Thanks for this. As agreed with the maintainers, I am leaving it open for DB Tsai. #144, which it is stacked on, is being replaced by a PR from a maintainer's fork, since these branches do not allow maintainer edits. Before this lands: - `_prepare_arrow_string_input` walks pandas' block manager (`_mgr`, `iget_values`, `blknos`, `blklocs`) and reads `_pa_array` and `_dtype` directly. The public API gives the same values: `column.array.__arrow_array__(type=pa.string())`, then `.cast(pa.string())`, matched the row path for `string[pyarrow]`, sliced frames and `ArrowDtype(large_string)`. Gating on `type(dtype) is StringDtype and dtype.storage == "pyarrow"` could replace about 200 lines. - On pandas 2.2, the lower bound of the supported range, the hook checks fail (`StringDtype.name` is a plain class attribute there), so the path declines a canonical `pd.NA` column, and `test_arrow_string_input.py` fails at `StringDtype(na_value=...)`, which 2.2 does not take. - It encodes the IPC in Python, while #144 adds a Rust encoder for strings; one of the two would do. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
