viirya commented on code in PR #108: URL: https://github.com/apache/spark-connect-rust/pull/108#discussion_r4418537573
########## docs/architecture.md: ########## @@ -102,6 +102,24 @@ triggers `ExecutePlan`. If the response stream breaks mid-flight, the core's reattachable-execute iterator resumes from the last response it saw, so long results survive transient disconnects. +### Python Arrow table handoff + +`toArrow()` collects decoded batches and uses arrow-rs's `IntoPyArrow` for one +boxed `RecordBatchReader`. Compatible value and offset buffers are shared; +network decoding, sliced validity bitmaps, and pandas conversion may still copy. +Batch schema consistency is validated before export. Field names or timezones +containing NUL use IPC over the collected batches, without another RPC. Empty +batch vectors retain the empty-schema behavior. + +The Arrow handoff has been tested locally with PyArrow 10.0.0 (Python 3.9) Review Comment: `docs/` is the published user site, and this paragraph is maintainer information (which PyArrow versions were tested locally, which CI job runs the smoke tests, what the parity gate covers). Could it move to `dev/design/architecture.md`? There's also an extra blank line after it (L121-122). ########## dev/design/arrow-handoff-rss.md: ########## @@ -0,0 +1,71 @@ +# Arrow handoff repeated-call RSS control Review Comment: This reads as a one-off measurement record for this PR (dated, single runs on a shared desktop) rather than design documentation. The reproducer also depends on test internals (`runpy.run_path` on the test file, `protocol.__wrapped__`, `_dataframe`), so it will quietly stop working when the tests change, and nothing checks it. Could the numbers and the script live in the PR description instead of the repo? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
