alamb commented on issue #22189: URL: https://github.com/apache/datafusion/issues/22189#issuecomment-6504228183
> What we saw. Peak memory grew linearly with the number of files scanned, about 32 MB per day (about 29 KB per file). A 27-day scan peaked at about 915 MB. A jemalloc heap profile, unsampled, put about 85% of the peak under ParquetFileMetrics::new. The plan's ExecutionPlanMetricsSet keeps every registered metric until the query ends, so these sets add up over the whole scan rather than being freed file by file. That sounds embarassing. Maybe we should have some limit or something that caps the number of metrics (as keeping per file statsistics for 14 x 1,000 files is expensive and will likely be of limited use -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
