zhuqi-lucas opened a new issue, #27254: URL: https://github.com/apache/datafusion/issues/27254
With `datafusion.execution.parquet.enable_exact_reverse_scan` (#27079), the scan buffers one decoded row group and charges each batch's `get_array_memory_size()` to the memory pool. Batches of the same row group can share buffers, such as a page dictionary, so a shared buffer is counted once per batch. The scan cannot spill, so a query can fail with a resources error well below its real memory use. Example from the #27079 review: 1M rows with 4000 distinct strings (about 25 MB). A 100 MB pool fails in the reverse scan, while the plan with the option off finishes at 30 MB by spilling in the sort. The reservation should count each distinct buffer once across the batches it holds. This should be fixed before the option is enabled by default. Related: #27079, #19048 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
