rezasafi opened a new pull request #23306: [SPARK-26357][Core] Expose executors' procfs metrics to Metrics system URL: https://github.com/apache/spark/pull/23306 This change exposes the executors' proscfs metrics that were introduced in SPARK-24958 to Metrics system. To do this a new metric source is defined and a new config is also added. Using the configs user can choose whether they want to see procfs metrics using REST API and Spark UI or to see them through Metrics system. To avoid overhead user can't choose to report these metrics to both UI and Metrics system. This was tested manually and I verified that procfs metrics are reporting in Metrics system using Console sink: application_1544653637885_0014.driver.procfs.processTree.ProcessTreeJVMRSSMemory value = 558977024 application_1544653637885_0014.driver.procfs.processTree.ProcessTreeJVMVMemory value = 4888707072 application_1544653637885_0014.driver.procfs.processTree.ProcessTreeOtherRSSMemory value = 0 application_1544653637885_0014.driver.procfs.processTree.ProcessTreeOtherVMemory value = 0 application_1544653637885_0014.driver.procfs.processTree.ProcessTreePythonRSSMemory value = 33546240 application_1544653637885_0014.driver.procfs.processTree.ProcessTreePythonVMemory value = 401543168 And they got updated as well: application_1544653637885_0014.driver.procfs.processTree.ProcessTreeJVMRSSMemory value = 659648512 application_1544653637885_0014.driver.procfs.processTree.ProcessTreeJVMVMemory value = 4965343232 application_1544653637885_0014.driver.procfs.processTree.ProcessTreeOtherRSSMemory value = 0 application_1544653637885_0014.driver.procfs.processTree.ProcessTreeOtherVMemory value = 0 application_1544653637885_0014.driver.procfs.processTree.ProcessTreePythonRSSMemory value = 33546240 application_1544653637885_0014.driver.procfs.processTree.ProcessTreePythonVMemory value = 401543168
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org