HyukjinKwon opened a new pull request, #310: URL: https://github.com/apache/spark-connect-rust/pull/310
### What changes were proposed in this pull request? [SPARK-61113](https://issues.apache.org/jira/browse/SPARK-61113): `DataFrame.__arrow_c_stream__` is added, as PySpark's. PySpark's DataFrame implements the stream of the Arrow PyCapsule interface. `__arrow_c_stream__(requested_schema=None)` returns an Arrow C stream of the rows of the DataFrame, of the Arrow schema of its schema. `pyspark.sql.interchange.SparkArrowCStreamer` reads one partition at a time through `mapInArrow` and `toLocalIterator`, so the client never holds all the rows at once. A requested schema is passed to the stream, which casts to it. `pyspark.sql.interchange` is vendored byte for byte from Apache Spark v4.3.0 (and added to the drift check). `DataFrame.__arrow_c_stream__` returns its stream. ### Why are the changes needed? The client had neither `DataFrame.__arrow_c_stream__` nor `pyspark.sql.interchange`. Libraries that read Arrow streams could not take a Spark DataFrame: `pa.RecordBatchReader.from_stream(df)` raised `TypeError: Expected an object implementing the Arrow PyCapsule Protocol for streams`. ### Does this PR introduce _any_ user-facing change? Yes. `pa.RecordBatchReader.from_stream(df)`, Polars and DuckDB read a DataFrame directly, as with PySpark 4.3.0. ### How was this patch tested? - A new live test reads the stream of a DataFrame of three partitions, with nested columns, through `pa.RecordBatchReader.from_stream`. It checks: - the schema, which is that of `toArrow` - one batch per partition - the rows - the stream of a requested schema, cast to it - the stream of an empty DataFrame It passes with official PySpark 4.3.0 and fails without this change. - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2105 passed, 8 skipped), and the live drop-in and Arrow suites (165 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
