This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a073bf38c7d [SPARK-45209][CORE][UI] Flame Graph Support For Executor
Thread Dump Page
a073bf38c7d is described below
commit a073bf38c7d8802e2ab12c54299e1541a48a394e
Author: Kent Yao <y...@apache.org>
AuthorDate: Wed Oct 25 18:29:43 2023 +0800
[SPARK-45209][CORE][UI] Flame Graph Support For Executor Thread Dump Page
### What changes were proposed in this pull request?
This PR draws a CPU Flame Graph by Java stack traces for executors and
drivers. Currently, the Java stack traces is just a SNAPSHOT, not sampling at a
certain frequency for a period. Sampling might be considered an upcoming
feature out of the scope of this PR.
If you are new to flame graphs, there are also some references you can
refer to learn about the basic concepts and details.
[1] [Flame Graphs](https://www.brendangregg.com/flamegraphs.html)
[2] [FLIP-165: Operator's Flame
Graphs](https://cwiki.apache.org/confluence/display/FLINK/FLIP-165%3A+Operator%27s+Flame+Graphs)
[3] [Java in Flames. mixed-mode flame graphs provide a… | by Netflix
Technology Blog](https://netflixtechblog.com/java-in-flames-e763b3d32166)
[4]
[HProf](https://docs.oracle.com/javase/7/docs/technotes/samples/hprof.html)
#### Pending features
This PR mainly focuses on the UI, independent of the profiling steps. What
we might have in the future are:
- Flame Graph Support For Task Thread Page which SPARK-45151 added
- Add `ProfilingExecutor(max, interval)` message to profile whole executor
- Add `ProfileTask(taskId, max, interval)` message to profile an certain
task
- Different views for on/off/full CPUs
- Mixed mode profiling, which might rely upon some ext libs at runtime
- And so on.
### Why are the changes needed?
Performance is always an important design factor in Spark. It is desirable
to provide better visibility into the distribution of CPU resources while
executing user code alongside the Spark kernel. One of the most visually
effective means to do that is [Flame
Graphs](http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html), which
visually presents the data gathered by performance profiling tools used by
developers for performance tuning their applications.
### Does this PR introduce _any_ user-facing change?
yes
### How was this patch tested?
locally
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #42988 from yaooqinn/SPARK-45209.
Authored-by: Kent Yao <y...@apache.org>
Signed-off-by: yangjie01 <yangji...@baidu.com>
---
LICENSE | 3 +-
LICENSE-binary | 3 +-
.../org/apache/spark/ui/static/d3-flamegraph.css | 47 ++++++++++++++++++++
.../apache/spark/ui/static/d3-flamegraph.min.js | 2 +
.../org/apache/spark/ui/static/flamegraph.js | 36 ++++++++++++++++
.../spark/ui/exec/ExecutorThreadDumpPage.scala | 20 +++++++++
.../spark/ui/flamegraph/FlamegraphNode.scala | 50 ++++++++++++++++++++++
dev/.rat-excludes | 2 +
8 files changed, 161 insertions(+), 2 deletions(-)
diff --git a/LICENSE b/LICENSE
index 44983fd1259..3216134fd4b 100644
--- a/LICENSE
+++ b/LICENSE
@@ -216,7 +216,8 @@
core/src/main/resources/org/apache/spark/ui/static/bootstrap*
core/src/main/resources/org/apache/spark/ui/static/vis*
docs/js/vendor/bootstrap.js
connector/spark-ganglia-lgpl/src/main/java/com/codahale/metrics/ganglia/GangliaReporter.java
-
+core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.min.js
+core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.css
Python Software Foundation License
----------------------------------
diff --git a/LICENSE-binary b/LICENSE-binary
index 30fca96a883..c6f291f1108 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -413,7 +413,8 @@
core/src/main/java/org/apache/spark/util/collection/TimSort.java
core/src/main/resources/org/apache/spark/ui/static/bootstrap*
core/src/main/resources/org/apache/spark/ui/static/vis*
docs/js/vendor/bootstrap.js
-
+core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.min.js
+core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.css
------------------------------------------------------------------------------------
This product bundles various third-party components under other open source
licenses.
diff --git
a/core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.css
b/core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.css
new file mode 100644
index 00000000000..92821ef10e0
--- /dev/null
+++ b/core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.css
@@ -0,0 +1,47 @@
+/** https://cdn.jsdelivr.net/npm/d3-flame-graph@4.1.3/dist/d3-flamegraph.css */
+.d3-flame-graph rect {
+ stroke: #EEEEEE;
+ fill-opacity: .8;
+}
+
+.d3-flame-graph rect:hover {
+ stroke: #474747;
+ stroke-width: 0.5;
+ cursor: pointer;
+}
+
+.d3-flame-graph-label {
+ pointer-events: none;
+ white-space: nowrap;
+ text-overflow: ellipsis;
+ overflow: hidden;
+ font-size: 12px;
+ font-family: Verdana;
+ margin-left: 4px;
+ margin-right: 4px;
+ line-height: 1.5;
+ padding: 0 0 0;
+ font-weight: 400;
+ color: black;
+ text-align: left;
+}
+
+.d3-flame-graph .fade {
+ opacity: 0.6 !important;
+}
+
+.d3-flame-graph .title {
+ font-size: 20px;
+ font-family: Verdana;
+}
+
+.d3-flame-graph-tip {
+ background-color: black;
+ border: none;
+ border-radius: 3px;
+ padding: 5px 10px 5px 10px;
+ min-width: 250px;
+ text-align: left;
+ color: white;
+ z-index: 10;
+}
\ No newline at end of file
diff --git
a/core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.min.js
b/core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.min.js
new file mode 100644
index 00000000000..babc683f6b6
--- /dev/null
+++ b/core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.min.js
@@ -0,0 +1,2 @@
+/**
https://cdn.jsdelivr.net/npm/d3-flame-graph@4.1.3/dist/d3-flamegraph.min.js */
+!function(t,n){"object"==typeof exports&&"object"==typeof
module?module.exports=n():"function"==typeof
define&&define.amd?define([],n):"object"==typeof
exports?exports.flamegraph=n():t.flamegraph=n()}(self,(function(){return(()=>{"use
strict";var t={d:(n,e)=>{for(var r in
e)t.o(e,r)&&!t.o(n,r)&&Object.defineProperty(n,r,{enumerable:!0,get:e[r]})},o:(t,n)=>Object.prototype.hasOwnProperty.call(t,n)},n={};function
e(){}function r(t){return null==t?e:function(){return this.querySelector(t)}}
[...]
\ No newline at end of file
diff --git a/core/src/main/resources/org/apache/spark/ui/static/flamegraph.js
b/core/src/main/resources/org/apache/spark/ui/static/flamegraph.js
new file mode 100644
index 00000000000..c298dbaeed6
--- /dev/null
+++ b/core/src/main/resources/org/apache/spark/ui/static/flamegraph.js
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/* global d3, flamegraph */
+
+/* eslint-disable no-unused-vars */
+function drawFlamegraph() {
+ const width = (window.innerWidth * 95) / 100;
+ const chart = flamegraph()
+ .width(width)
+ .cellHeight(18)
+ .transitionEase(d3.easeCubic)
+ .sort(true)
+ .title("");
+ const jsonStr = d3.select("#executor-flamegraph-data").text().trim()
+ const jsonData = JSON.parse(jsonStr);
+ d3.select("#executor-flamegraph-chart")
+ .datum(jsonData)
+ .call(chart);
+ window.onresize = () => chart.width(width);
+}
+/* eslint-enable no-unused-vars */
diff --git
a/core/src/main/scala/org/apache/spark/ui/exec/ExecutorThreadDumpPage.scala
b/core/src/main/scala/org/apache/spark/ui/exec/ExecutorThreadDumpPage.scala
index 0be9df921d1..4a00777c509 100644
--- a/core/src/main/scala/org/apache/spark/ui/exec/ExecutorThreadDumpPage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/exec/ExecutorThreadDumpPage.scala
@@ -22,7 +22,10 @@ import javax.servlet.http.HttpServletRequest
import scala.xml.{Node, Text}
import org.apache.spark.SparkContext
+import org.apache.spark.status.api.v1.ThreadStackTrace
import org.apache.spark.ui.{SparkUITab, UIUtils, WebUIPage}
+import org.apache.spark.ui.UIUtils.prependBaseUri
+import org.apache.spark.ui.flamegraph.FlamegraphNode
private[ui] class ExecutorThreadDumpPage(
parent: SparkUITab,
@@ -67,8 +70,10 @@ private[ui] class ExecutorThreadDumpPage(
<div class="row">
<div class="col-12">
<p>Updated at {UIUtils.formatDate(time)}</p>
+ {drawExecutorFlamegraph(request, threadDump)}
{
// scalastyle:off
+ <p></p>
<div style="display: flex; align-items: center;">
<a class="expandbutton"
onClick="expandAllThreadStackTrace(true)">Expand All</a>
<a class="expandbutton d-none"
onClick="collapseAllThreadStackTrace(true)">Collapse All</a>
@@ -106,4 +111,19 @@ private[ui] class ExecutorThreadDumpPage(
}.getOrElse(Text("Error fetching thread dump"))
UIUtils.headerSparkPage(request, s"Thread dump for executor $executorId",
content, parent)
}
+
+ // scalastyle:off
+ private def drawExecutorFlamegraph(request: HttpServletRequest, thread:
Array[ThreadStackTrace]): Seq[Node] = {
+ <div>
+ <div id="executor-flamegraph-data"
class="d-none">{FlamegraphNode(thread).toJsonString}</div>
+ <div id="executor-flamegraph-chart">
+ <link rel="stylesheet" type="text/css" href={prependBaseUri(request,
"/static/d3-flamegraph.css")}></link>
+ <script src={UIUtils.prependBaseUri(request,
"/static/d3-flamegraph.min.js")}></script>
+ <script src={UIUtils.prependBaseUri(request,
"/static/d3.min.js")}></script>
+ <script src={UIUtils.prependBaseUri(request,
"/static/flamegraph.js")}></script>
+ <script>drawFlamegraph()</script>
+ </div>
+ </div>
+ }
+ // scalastyle:off
}
diff --git
a/core/src/main/scala/org/apache/spark/ui/flamegraph/FlamegraphNode.scala
b/core/src/main/scala/org/apache/spark/ui/flamegraph/FlamegraphNode.scala
new file mode 100644
index 00000000000..a0a59f0572d
--- /dev/null
+++ b/core/src/main/scala/org/apache/spark/ui/flamegraph/FlamegraphNode.scala
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.ui.flamegraph
+
+import scala.collection.mutable.HashMap
+
+import org.apache.commons.text.StringEscapeUtils
+
+import org.apache.spark.status.api.v1.ThreadStackTrace
+
+case class FlamegraphNode(name: String) {
+ private val children = new HashMap[String, FlamegraphNode]()
+ private var value: Int = 0
+ def toJsonString: String = {
+ // scalastyle:off line.size.limit
+
s"""{"name":"$name","value":$value,"children":[${children.map(_._2.toJsonString).mkString(",")}]}"""
+ // scalastyle:on line.size.limit
+ }
+}
+
+object FlamegraphNode {
+ def apply(stacks: Array[ThreadStackTrace]): FlamegraphNode = {
+ val root = FlamegraphNode("root")
+ stacks.foreach { stack =>
+ root.value += 1
+ var cur = root
+ stack.stackTrace.elems.reverse.foreach { e =>
+ val head = e.split("\n").head
+ val name = StringEscapeUtils.escapeJson(head)
+ cur = cur.children.getOrElseUpdate(name, FlamegraphNode(name))
+ cur.value += 1
+ }
+ }
+ root
+ }
+}
diff --git a/dev/.rat-excludes b/dev/.rat-excludes
index a27319f16aa..4f0fcf085be 100644
--- a/dev/.rat-excludes
+++ b/dev/.rat-excludes
@@ -26,6 +26,8 @@ bootstrap.bundle.min.js
bootstrap.min.css
jquery-3.5.1.min.js
d3.min.js
+d3-flamegraph.css
+d3-flamegraph.min.js
dagre-d3.min.js
graphlib-dot.min.js
sorttable.js
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
