2010YOUY01 commented on PR #23777: URL: https://github.com/apache/datafusion/pull/23777#issuecomment-5202012722
> > Is it possible to implement it entirely within the `record_output`, and avoid large changes to different operators inner logic? > > I don't see how, since `RecordBatch::record_output` doesn't keep any state related to previous `RecordBatch`es that have been processed, and we need this information so that we can deduplicate the sliced `RecordBatch`es to avoid summing up the memory for the same underlying original large RecordBatch. This is a limitation that we need to deal with until we get #15591 Perhaps we can use something like `RecordBatchMemoryCounter` (with internal buffer tracking to avoid double-counting) to replace the below `Counter`: https://github.com/apache/datafusion/blob/f27e50c13647efa5292d5ef5b1e4ed6af3015962/datafusion/physical-expr-common/src/metrics/baseline.rs#L69 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
