[GitHub] spark pull request #22167: [SPARK-25170][DOC] Add list and short description...
Github user LucaCanali closed the pull request at: https://github.com/apache/spark/pull/22167 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22167: [SPARK-25170][DOC] Add list and short description...
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/22167#discussion_r215315965 --- Diff: docs/monitoring.md --- @@ -388,6 +388,163 @@ value triggering garbage collection on jobs, and `spark.ui.retainedStages` that Note that the garbage collection takes place on playback: it is possible to retrieve more entries by increasing these values and restarting the history server. +### Executor Task Metrics + +The REST API exposes the values of the Task Metrics collected by Spark executors at the +task execution level. The metrics can be used for performance troubleshooting. +A list of the available metrics with a short description: + + + Spark Executor Task Metric name + Short description + + +executorRunTime +Time the executor spent running this task. This includes time fetching shuffle data. +The value is expressed in milliseconds. + + +executorCpuTime +CPU Time the executor spent running this task. This includes time fetching shuffle data. --- End diff -- nit: `CPU time`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22167: [SPARK-25170][DOC] Add list and short description...
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/22167#discussion_r215315657 --- Diff: docs/monitoring.md --- @@ -388,6 +388,163 @@ value triggering garbage collection on jobs, and `spark.ui.retainedStages` that Note that the garbage collection takes place on playback: it is possible to retrieve more entries by increasing these values and restarting the history server. +### Executor Task Metrics + +The REST API exposes the values of the Task Metrics collected by Spark executors at the +task execution level. The metrics can be used for performance troubleshooting. +A list of the available metrics with a short description: + + + Spark Executor Task Metric name + Short description + + +executorRunTime +Time the executor spent running this task. This includes time fetching shuffle data. +The value is expressed in milliseconds. + + +executorCpuTime +CPU Time the executor spent running this task. This includes time fetching shuffle data. +The value is expressed in nanoseconds. + + +executorDeserializeTime +Time taken on the executor to deserialize this task. +The value is expressed in milliseconds. + + +executorDeserializeCpuTime +CPU Time taken on the executor to deserialize this task. + The value is expressed in nanoseconds. + + +resultSize +The number of bytes this task transmitted back to the driver as the TaskResult. + + +jvmGCTime +Amount of time the JVM spent in garbage collection while executing this task. --- End diff -- Why do we start with `amount of` while the above parameters start `Time` or `CPU Time`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22167: [SPARK-25170][DOC] Add list and short description...
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/22167#discussion_r215315711 --- Diff: docs/monitoring.md --- @@ -388,6 +388,163 @@ value triggering garbage collection on jobs, and `spark.ui.retainedStages` that Note that the garbage collection takes place on playback: it is possible to retrieve more entries by increasing these values and restarting the history server. +### Executor Task Metrics + +The REST API exposes the values of the Task Metrics collected by Spark executors at the +task execution level. The metrics can be used for performance troubleshooting. +A list of the available metrics with a short description: + + + Spark Executor Task Metric name + Short description + + +executorRunTime +Time the executor spent running this task. This includes time fetching shuffle data. +The value is expressed in milliseconds. + + +executorCpuTime +CPU Time the executor spent running this task. This includes time fetching shuffle data. +The value is expressed in nanoseconds. + + +executorDeserializeTime +Time taken on the executor to deserialize this task. +The value is expressed in milliseconds. + + +executorDeserializeCpuTime +CPU Time taken on the executor to deserialize this task. + The value is expressed in nanoseconds. + + +resultSize +The number of bytes this task transmitted back to the driver as the TaskResult. + + +jvmGCTime +Amount of time the JVM spent in garbage collection while executing this task. +The value is expressed in milliseconds. + + +resultSerializationTime +Amount of time spent serializing the task result. --- End diff -- ditto --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22167: [SPARK-25170][DOC] Add list and short description...
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/22167#discussion_r215314710 --- Diff: docs/monitoring.md --- @@ -388,6 +388,163 @@ value triggering garbage collection on jobs, and `spark.ui.retainedStages` that Note that the garbage collection takes place on playback: it is possible to retrieve more entries by increasing these values and restarting the history server. +### Executor Task Metrics + +The REST API exposes the values of the Task Metrics collected by Spark executors at the +task execution level. The metrics can be used for performance troubleshooting. +A list of the available metrics with a short description: + + + Spark Executor Task Metric name + Short description + + +executorRunTime +Time the executor spent running this task. This includes time fetching shuffle data. +The value is expressed in milliseconds. + + +executorCpuTime --- End diff -- Do we miss `` in these two lines? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22167: [SPARK-25170][DOC] Add list and short description...
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/22167#discussion_r215314527 --- Diff: docs/monitoring.md --- @@ -388,6 +388,163 @@ value triggering garbage collection on jobs, and `spark.ui.retainedStages` that Note that the garbage collection takes place on playback: it is possible to retrieve more entries by increasing these values and restarting the history server. +### Executor Task Metrics + +The REST API exposes the values of the Task Metrics collected by Spark executors at the +task execution level. The metrics can be used for performance troubleshooting. +A list of the available metrics with a short description: + + + Spark Executor Task Metric name + Short description + + +executorRunTime +Time the executor spent running this task. This includes time fetching shuffle data. --- End diff -- Does `Time` mean `elapsed time` or other `time`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22167: [SPARK-25170][DOC] Add list and short description...
GitHub user LucaCanali opened a pull request: https://github.com/apache/spark/pull/22167 [SPARK-25170][DOC] Add list and short description of Spark Executor Task Metrics to the documentation ## What changes were proposed in this pull request? Add description of Task Metrics to the documentation. ## How was this patch tested? None. You can merge this pull request into a Git repository by running: $ git pull https://github.com/LucaCanali/spark docMonitoringTaskMetrics Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22167.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22167 commit a8db1605adbc271c785fda24b4945bf87149a4cd Author: LucaCanali Date: 2018-08-20T14:12:52Z Document Spark Executor Task Metrics --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
