lifulong commented on PR #12127: URL: https://github.com/apache/gluten/pull/12127#issuecomment-4623073343
> @lifulong Does it only impact driver memory or executor memory as well? We usually use a driver nodes with more memory and cores. So driver memory OOM rarely happens. > > Can we set the config to true by default and disable it in your case? @FelixYBW Thanks for the review. I discovered this issue when troubleshooting driver OOM issues. Theoretically it has little impact on executor memory usage, but I haven’t analyzed actual executor memory usage data. I’ll adjust the default value of spark.gluten.sql.scan.detailedMetrics.enabled: keep all metrics collected by default, and allow reducing collected metrics via this configuration toggle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
