[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2016-02-19 Thread yaooqinn
Github user yaooqinn commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r53543399 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -349,6 +365,23 @@ private[ui] class StagePage(parent: StagesTab) extends WebUIP

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36189276 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -773,16 +773,26 @@ class DAGScheduler( stage.pendingTasks.clear()

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36135556 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -773,16 +773,26 @@ class DAGScheduler( stage.pendingTasks.clear()

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36135056 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -773,16 +773,26 @@ class DAGScheduler( stage.pendingTasks.clear

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127411771 Other concerns aside ... can somebody explain the logic for resetting the accumulator values to me? IIUC you reset the values: a) on the initial instance

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127410184 Hey Patrick, that sounds good. I agree that there are probably a couple different concerns here. My biggest concern was some overall direction we head on metri

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/7770 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127409439 [Test build #1314 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1314/console) for PR 7770 at commit [`f5b0d68`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127409208 Now that this has passed tests as of the latest commit, I'm going to merge it to master and branch-1.5. Thanks! --- If your project is set up for it, you can reply t

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127408534 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127408428 [Test build #39571 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39571/console) for PR 7770 at commit [`9abecb9`](https://github.

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127406881 [Test build #1316 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1316/console) for PR 7770 at commit [`f5b0d68`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127402731 [Test build #1313 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1313/console) for PR 7770 at commit [`f5b0d68`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127401874 [Test build #1310 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1310/console) for PR 7770 at commit [`d7df332`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127401517 [Test build #1311 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1311/console) for PR 7770 at commit [`d7df332`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127400950 [Test build #1315 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1315/console) for PR 7770 at commit [`f5b0d68`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127398592 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127398453 [Test build #39568 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39568/console) for PR 7770 at commit [`f5b0d68`](https://github.

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127387737 [Test build #1312 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1312/console) for PR 7770 at commit [`d7df332`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127378665 Hey Imran, So I think there is a larger design discussion at hand that we can probably break out into a mailing list thread or maybe discuss offline - about as

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127365342 Accumulators handle task failures, but have super confusing semantics around recomputation from shared lineage, speculative execution, and stage retries. (eg., I don't u

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127362926 [Test build #39571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39571/consoleFull) for PR 7770 at commit [`9abecb9`](https://gith

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127362471 [Test build #1315 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1315/consoleFull) for PR 7770 at commit [`f5b0d68`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127362504 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127362520 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127362381 [Test build #1314 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1314/consoleFull) for PR 7770 at commit [`f5b0d68`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127362468 [Test build #1316 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1316/consoleFull) for PR 7770 at commit [`f5b0d68`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127357209 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127354214 [Test build #1312 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1312/consoleFull) for PR 7770 at commit [`d7df332`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127354538 [Test build #1313 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1313/consoleFull) for PR 7770 at commit [`f5b0d68`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127354001 [Test build #39568 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39568/consoleFull) for PR 7770 at commit [`f5b0d68`](https://gith

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127354192 [Test build #1311 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1311/consoleFull) for PR 7770 at commit [`d7df332`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127353734 [Test build #1307 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1307/console) for PR 7770 at commit [`d7df332`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127353710 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127353656 [Test build #1310 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1310/consoleFull) for PR 7770 at commit [`d7df332`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127353757 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127351943 [Test build #1309 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1309/consoleFull) for PR 7770 at commit [`d7df332`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127351953 [Test build #1308 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1308/consoleFull) for PR 7770 at commit [`d7df332`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127351729 [Test build #1307 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1307/consoleFull) for PR 7770 at commit [`d7df332`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127350868 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127350821 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127349734 LGTM pending Jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127349371 Reviewed 2 of 2 files at r17, 3 of 5 files at r18, 1 of 1 files at r20, 4 of 4 files at r21. Review status: all files reviewed at latest revisio

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36111035 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala --- @@ -258,6 +259,24 @@ class SQLQuerySuite extends QueryTest with BeforeAndAft

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36104996 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/joins/BroadcastJoinSuite.scala --- @@ -0,0 +1,94 @@ +/* +* Licensed to the Apache

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127329348 To clarify slightly, (@andrewor14, correct me if I'm wrong), these new internal metrics are considered to be completely `private[spark]` right now, so there's no `@Dev

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36104593 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/joins/BroadcastJoinSuite.scala --- @@ -0,0 +1,94 @@ +/* +* Licensed to the Apache

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127328436 @squito we won't remove `TaskMetrics` before 2.0 because it breaks compatibility, as you mentioned. Also, in my mind the semantics of accumulators during retries are

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127285180 [Test build #1305 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1305/console) for PR 7770 at commit [`9de2a12`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127276478 [Test build #39539 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39539/console) for PR 7770 at commit [`9de2a12`](https://github.

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127276533 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36087907 --- Diff: core/src/main/scala/org/apache/spark/ui/ToolTips.scala --- @@ -62,6 +62,13 @@ private[spark] object ToolTips { """Time that the executor spen

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127243422 I didn't realize the point of that other PR was so we could start switching all task metrics over ... accumulators have their own set of confusing semantics around retrie

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127229048 [Test build #39539 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39539/consoleFull) for PR 7770 at commit [`9de2a12`](https://gith

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127228657 [Test build #1305 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1305/consoleFull) for PR 7770 at commit [`9de2a12`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127228266 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127164852 To add to @pwendell's comment, the biggest advantage of accumulators is the ability to deal with failed and resubmitted tasks. Double counting has been a perennial is

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127164405 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36065313 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/joins/BroadcastJoinSuite.scala --- @@ -0,0 +1,94 @@ +/* +* Licensed to the Apache

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-03 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127163815 I was able to reproduce the test failure locally and fixed it. There were many logical merge conflicts that changed the underlying assumptions of a few of my tests. T

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36056278 --- Diff: core/src/main/scala/org/apache/spark/ui/ToolTips.scala --- @@ -62,6 +62,13 @@ private[spark] object ToolTips { """Time that the executor s

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36056120 --- Diff: core/src/main/scala/org/apache/spark/ui/ToolTips.scala --- @@ -62,6 +62,13 @@ private[spark] object ToolTips { """Time that the executor sp

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36056048 --- Diff: core/src/main/scala/org/apache/spark/ui/ToolTips.scala --- @@ -62,6 +62,13 @@ private[spark] object ToolTips { """Time that the executor s

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36056021 --- Diff: core/src/main/scala/org/apache/spark/ui/ToolTips.scala --- @@ -62,6 +62,13 @@ private[spark] object ToolTips { """Time that the executor spent

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36056014 --- Diff: core/src/main/scala/org/apache/spark/ui/ToolTips.scala --- @@ -62,6 +62,13 @@ private[spark] object ToolTips { """Time that the executor spent

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127122956 @squito regarding accumulators, there was an earlier patch that supports incremental updates of accumulators, so we can use them for this type of measurement now. Over

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36055960 --- Diff: core/src/main/scala/org/apache/spark/ui/ToolTips.scala --- @@ -62,6 +62,13 @@ private[spark] object ToolTips { """Time that the executor s

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36055866 --- Diff: core/src/main/scala/org/apache/spark/ui/ToolTips.scala --- @@ -62,6 +62,13 @@ private[spark] object ToolTips { """Time that the executor sp

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36055833 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/joins/BroadcastJoinSuite.scala --- @@ -0,0 +1,94 @@ +/* +* Licensed to the Apache

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36055835 --- Diff: core/src/main/scala/org/apache/spark/ui/ToolTips.scala --- @@ -62,6 +62,13 @@ private[spark] object ToolTips { """Time that the executor sp

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127120882 It looks like this is now passing all tests except for one of its own, `BytesToBytesMapSuite.testTotalMemoryConsumption`. --- If your project is set up for it, you ca

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127120493 [Test build #1298 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1298/console) for PR 7770 at commit [`d0fef87`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127119434 [Test build #1297 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1297/console) for PR 7770 at commit [`d0fef87`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127119209 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127119192 [Test build #39494 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39494/console) for PR 7770 at commit [`40b4802`](https://github.

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127118845 [Test build #1296 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1296/console) for PR 7770 at commit [`d0fef87`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36055191 --- Diff: core/src/main/scala/org/apache/spark/ui/ToolTips.scala --- @@ -62,6 +62,13 @@ private[spark] object ToolTips { """Time that the executor spen

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36055045 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -773,16 +773,26 @@ class DAGScheduler( stage.pendingTasks.clear()

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127117982 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127117976 **[Test build #39484 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39484/console)** for PR 7770 at commit [`0625d73`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127114536 Whats the reason for using accumulators instead of TaskMetrics? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127114307 **[Test build #1291 timed out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1291/console)** for PR 7770 at commit [`c00a197`](https://githu

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127114074 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127114062 **[Test build #39479 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39479/console)** for PR 7770 at commit [`c00a197`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/7770#discussion_r36054177 --- Diff: core/src/main/scala/org/apache/spark/Aggregator.scala --- @@ -89,13 +84,18 @@ case class Aggregator[K, V, C] ( } else { val combin

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127110512 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127110459 **[Test build #39477 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39477/console)** for PR 7770 at commit [`10da1cd`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127108559 [Test build #1292 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1292/console) for PR 7770 at commit [`c00a197`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127107581 [Test build #1290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1290/console) for PR 7770 at commit [`c00a197`](https://github

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127103973 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127102207 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127101555 [Test build #39494 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39494/consoleFull) for PR 7770 at commit [`40b4802`](https://gith

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127101482 **[Test build #1289 timed out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1289/console)** for PR 7770 at commit [`17f4c2d`](https://githu

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127101390 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127101402 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127101361 [Test build #1298 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1298/consoleFull) for PR 7770 at commit [`d0fef87`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127101344 [Test build #1297 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1297/consoleFull) for PR 7770 at commit [`d0fef87`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127101341 [Test build #1296 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1296/consoleFull) for PR 7770 at commit [`d0fef87`](https://git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127101224 Haha, looks like I was wrong about commenting code out like that :( --- If your project is set up for it, you can reply to this email and have your reply appear on Git

[GitHub] spark pull request: [SPARK-8735] [SQL] Expose memory usage for shu...

2015-08-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7770#issuecomment-127101167 [Test build #1294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1294/console) for PR 7770 at commit [`d0fef87`](https://github

  1   2   3   >