[ https://issues.apache.org/jira/browse/SPARK-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14098758#comment-14098758 ]
Phuoc Do commented on SPARK-975: -------------------------------- Cheng Lian, I saw that latest UI displays stack trace for each stage. Is there a way to filter out function calls that we don't display in debugger. There seems to be a lot of native code calls in there. See stack below. I did some work with d3 force layout. See here: https://github.com/dnprock/spark-debugger Stack: org.apache.spark.rdd.RDD.count(RDD.scala:904) $line9.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:15) $line9.$read$$iwC$$iwC$$iwC.<init>(<console>:20) $line9.$read$$iwC$$iwC.<init>(<console>:22) $line9.$read$$iwC.<init>(<console>:24) $line9.$read.<init>(<console>:26) $line9.$read$.<init>(<console>:30) $line9.$read$.<clinit>(<console>) $line9.$eval$.<init>(<console>:7) $line9.$eval$.<clinit>(<console>) $line9.$eval.$print(<console>) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:483) org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789) org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062) org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615) org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646) org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610) > Spark Replay Debugger > --------------------- > > Key: SPARK-975 > URL: https://issues.apache.org/jira/browse/SPARK-975 > Project: Spark > Issue Type: New Feature > Components: Spark Core > Affects Versions: 0.9.0 > Reporter: Cheng Lian > Labels: arthur, debugger > Attachments: IMG_20140722_184149.jpg, RDD DAG.png > > > The Spark debugger was first mentioned as {{rddbg}} in the [RDD technical > report|http://www.cs.berkeley.edu/~matei/papers/2011/tr_spark.pdf]. > [Arthur|https://github.com/mesos/spark/tree/arthur], authored by [Ankur > Dave|https://github.com/ankurdave], is an old implementation of the Spark > debugger, which demonstrated both the elegance and power behind the RDD > abstraction. Unfortunately, the corresponding GitHub branch was not merged > into the master branch and had stopped 2 years ago. For more information > about Arthur, please refer to [the Spark Debugger Wiki > page|https://github.com/mesos/spark/wiki/Spark-Debugger] in the old GitHub > repository. > As a useful tool for Spark application debugging and analysis, it would be > nice to have a complete Spark debugger. In > [PR-224|https://github.com/apache/incubator-spark/pull/224], I propose a new > implementation of the Spark debugger, the Spark Replay Debugger (SRD). > [PR-224|https://github.com/apache/incubator-spark/pull/224] is only a preview > for discussion. In the current version, I only implemented features that can > illustrate the basic mechanisms. There are still features appeared in Arthur > but missing in SRD, such as checksum based nondeterminsm detection and single > task debugging with conventional debugger (like {{jdb}}). However, these > features can be easily built upon current SRD framework. To minimize code > review effort, I didn't include them into the current version intentionally. > Attached is the visualization of the MLlib ALS application (with 1 iteration) > generated by SRD. For more information, please refer to [the SRD overview > document|http://spark-replay-debugger-overview.readthedocs.org/en/latest/]. -- This message was sent by Atlassian JIRA (v6.2#6252) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org