[
https://issues.apache.org/jira/browse/SPARK-60917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-60917:
------------------------------------
Assignee: DB Tsai
> Reduce cached artifact upload copies and accelerate SHA-256 in Spark Connect
> Rust
> ---------------------------------------------------------------------------------
>
> Key: SPARK-60917
> URL: https://issues.apache.org/jira/browse/SPARK-60917
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Following SPARK-60675, cached Arrow input upload has two avoidable client
> costs: sha2 0.10 uses software SHA-256 on aarch64 without its asm feature,
> and cache artifact framing copies serialized IPC buffers and eagerly
> materializes all request frames.
> Enable sha2 assembly acceleration on non-Windows aarch64 while retaining
> runtime CPU detection/software fallback. Move cache blobs into shared Bytes
> and yield the existing 32 KiB artifact frames lazily. Keep hashes, CRCs,
> cache lookup/deduplication, acknowledgements and generic file-artifact
> framing unchanged.
> Local macOS arm64 measurements on 136 MB of Arrow input: isolated SHA hashing
> improved from about 302 to 53 ms. In a separate five-sample framing
> comparison with accelerated SHA in both native clients, median
> createDataFrame time improved from 404.43 to 341.43 ms and sampled extra peak
> RSS fell from 288.59 to 135.81 MiB. Spark 4.2.0 local[2], 3 GiB driver,
> Python 3.12.14 and PyArrow 26; input preparation and count/eight-column-sum
> validation were outside the timed interval. These are local creation
> measurements, not end-to-end query or remote-network claims.
> This issue covers SHA acceleration and lazy shared-buffer framing only.
> Serialization/upload pipelining is excluded. SPARK-60898 tracks the
> independent stable-plan-ID/server-plan-cache problem.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
