leixm opened a new pull request, #55230: URL: https://github.com/apache/spark/pull/55230
### What changes were proposed in this pull request? This PR adds a fast path in `Platform.copyMemory()` for copies where `length <= UNSAFE_COPY_THRESHOLD` (1MB). When the copy fits within a single threshold-sized chunk, we directly call `_UNSAFE.copyMemory()` and return immediately, bypassing the direction check and while loop. This is functionally equivalent to the original code because when `length <= UNSAFE_COPY_THRESHOLD`, both the forward and backward copy paths produce the exact same single call to `_UNSAFE.copyMemory()` — the loop executes exactly once regardless of direction. The fast path simply avoids the unnecessary branch, loop setup, and `Math.min` computation. ### Why are the changes needed? The vast majority of `Platform.copyMemory` calls in Spark (shuffle, sort, serialization, etc.) copy far less than 1MB. For these small copies, the current code still pays the overhead of: - A conditional branch to determine copy direction (`dstOffset < srcOffset`) - While loop initialization and condition check - `Math.min(length, UNSAFE_COPY_THRESHOLD)` computation While each individual overhead is small, `Platform.copyMemory` is on an extremely hot path — it is called millions of times per task during shuffle and sort operations. In our production testing, this optimization resulted in a measurable reduction in CPU time. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing unit tests. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
