DB Tsai created SPARK-60701:
-------------------------------
Summary: Decode owned Arrow IPC result buffers without
intermediate stream copies in Spark Connect Rust
Key: SPARK-60701
URL: https://issues.apache.org/jira/browse/SPARK-60701
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Assignee: DB Tsai
Fix For: connect-rust-4.3.0
After PR108 C Stream export, frozen combined toArrow return-long-w50-b10000
still15.86ms vsPython11.59ms and large-string11.30vs8.38ms.
DataFrame.collect_record_batches owns each protocol ArrowBatch Vec but
decode_arrow_record_batches borrows it into Cursor+StreamReader, which
allocates and copies message bodies. Evaluate Arrow58.3 StreamDecoder with an
owned Arrow Buffer to reuse compatible underlying bytes, retaining automatic
alignment copying, validation and lifetime ownership. Scope the record-batch
result decoding path, distinct from PR108 C Stream export and PR112 typed row
decode. Preserve multi-batch IPC/dictionaries/schema/error/truncated/empty
stream semantics and capture_execution. Verify exported values remain valid
after stream/response/DataFrame drop. Require same PR108 dependency on both
frozen benchmark sides, unchanged harness and independent all-cell/schema
correctness including nested/null/string/dictionary/alignment cases. Publish
only if meaningful measured throughput/RSS gains without guard regression. No
unsafe or unchecked Arrow validation, no JVM.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
