peterxcli commented on PR #5916: URL: https://github.com/apache/datafusion-comet/pull/5916#issuecomment-5723042805
> Thanks @peterxcli . LGTM and merging > > A few suggested follow-ups: > > 1. **Broader performance and scaling coverage**: benchmark on SSD/NVMe with many spill rounds and concurrent tasks, reporting spill/merge time, peak memory, open file counts, and per-disk peak space. > 2. **Range-index memory accounting**: measure the retained range metadata under heavy spilling and account for it in memory reservations where appropriate. > 3. **Evaluate spill-file rotation**: compare the current task-wide file with files per spill session or rotation at a byte threshold. If limiting individual file size is the objective, a byte threshold addresses that more directly than session boundaries. Any rotation should keep open merge readers bounded and preserve each partition’s write order. @sunchao thanks for the suggestion, filed https://github.com/apache/datafusion-comet/issues/6010 and https://github.com/apache/datafusion-comet/issues/6009. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
