mitchsw commented on issue #23644: URL: https://github.com/apache/datafusion/issues/23644#issuecomment-4999033253
Thanks for the suggestion! I'm not 100% sure I understand your proposal, but attempting to answer: 1. We can remove per-file metrics entirely, making registration `O(partitions)` without adding a new config. However, that would break `EXPLAIN ANALYZE VERBOSE`, which currently outputs per-file metrics. 2. We can retain the `O(files)` registrations but replace the query-wide `ExecutionPlanMetricsSet` map and mutex with a two-tier structure: `ExecutionPlan -> partition -> file`. Each partition would have its own metric map and lock, which should eliminate the query-wide contention that dominated my profile. However, we would still allocate and maintain per-file metrics that are never read in the common case. If we want compact metrics by default while retaining per-file metrics for verbose analysis, we need some execution-time signal to select between the two behaviors. I am happy to explore alternative 2 if the config is undesirable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
