sunchao commented on PR #58763: URL: https://github.com/apache/spark/pull/58763#issuecomment-5673016893
@viirya Yes: the intended policy is that a consumer explicitly opts disposable, unused work into reclamation for ordinary allocations, including another task's allocations. It does not mean that completed I/O was free, and this mechanism does not incorporate workload priority. The revision narrows when that cost is paid: keep buffers when the existing capacity or the same immediate share-bound partial grant suffices; reclaim only the affected memory mode; stop once admission can proceed; skip drained owners; never drain on remove/clear or the unroll-to-storage transfer. The startup setting defaults to false. A latency-sensitive consumer can decline optional reservations, or limit them to unused work it can discard synchronously. It must keep backing-memory credit for buffers still used by decoding or in-flight I/O, and its callback must not wait for I/O or close the whole reader. The included TaskMemoryManager consumer tests and downstream Comet JNI fixture validate ownership, same/peer-task reclamation, fairness, rollback and cleanup. The no-owner allocator measurement is in the reply to r3998074904. These do not measure scan benefit, duplicate I/O, the owner's cost, or owner/requester p95/p99 latency. I have made those limits explicit in the PR description and am leaving the first-consumer design discussion open. Representative reader measurements remain necessary before enabling the policy for a workload. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
