Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-149215414
https://issues.apache.org/jira/browse/SPARK-11185
Note I do believe adding something like max task time would be very useful
for debugging. I can do this same
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-148844132
Yeah @tgravescs would you mind filing a new JIRA for the second issue you
brought up (that you want a new metric in the all stages page), so we can keep
that separ
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-148843534
I see... is the JIRA title / description outdated then?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-148842974
@andrewor14 "Duration" already is a column in the "All Stages Page". The
problem @tgravescs is pointing out is that it's not the right metric for his
use case ((1
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-148842285
It seems that there are two separate issues here: (1) the stage duration is
misleading, and (2) there's no way to see from the "All Stages Page" which
stages took lon
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-148809195
(incidentally, I just realized this info is totally missing from the json,
filed https://issues.apache.org/jira/browse/SPARK-11155)
---
If your project is set up for it,
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-148808343
Also jumping in late, but I agree with @andrewor14 , I think we should just
change duration to (1), that would be the most useful. My vote is for (last
task end) - (firs
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-148715800
I'm also a bit hesitant to change Duration but if everyone agrees its a bug
we can change it. The question is it supposed to time taken across all tasks
or is it time
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-148576976
@andrewor14 did you see @tgravescs 's comment saying that (1) and (2) don't
really help his use case? He wants an easy way to spot stages that took a lot
of resou
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-148552339
Chiming in a little late here. I actually like (1) in @pwendell's
[comment](https://github.com/apache/spark/pull/9051#issuecomment-147493632),
where we fix the semant
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-148471371
@d2r can we get a screenshot here to help visualize the change? It'd be
great to do that for all UI changes.
---
If your project is set up for it, you can reply to this
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-148084696
Ping... thoughts on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147523442
I definitely agree with not cluttering up the page. I'm fine with addition
metrics or something.
So do we want to add 2 columns, 1 for the "active time" and
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147514208
Ok that makes sense to me. I'm still a little concerned about cluterring up
the main page with something that's not necessarily broadly useful...what about
hiding
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147509387
So the stages page already has the # of tasks per stage and the Duration so
personally if I saw a large duration with many tasks and a small max task
duration I wouldn
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147499886
I was thinking it's misleading in the sense that a stage could have a small
max task duration, but a huge number of tasks, such that its total time is
still long.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147498736
Or perhaps we need multiple columns added to cover both or if you have
better ideas on to show I'm definitely open to suggestions.
---
If your project is set up for i
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147498589
So one thing I would I want from this is if I have 100's of stages I want
to be able to quickly (I don't want to click on each individual stage) look and
see which one
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147495618
I like the idea of adding 2 ("active time"?). Then users can infer 3 from
this. I agree that the metric added here is misleading.
---
If your project is set up fo
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147493632
I see the underlying problem posed in the JIRA - it's difficult to assess
duration since it currently includes the time spent waiting on dependent
stages. However, this
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147489092
@vanzin @JoshRosen just wanted to check to get your input to if this would
be useful to you or if you had other ideas? I basically want some other
indicator that sho
Github user d2r commented on a diff in the pull request:
https://github.com/apache/spark/pull/9051#discussion_r41787721
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala ---
@@ -150,6 +152,10 @@ private[ui] class StageTableBase(
}
val formatte
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/9051#discussion_r41787102
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala ---
@@ -150,6 +152,10 @@ private[ui] class StageTableBase(
}
val fo
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147475887
Jenkins, this is okay to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147439696
[Test build #43568 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43568/console)
for PR 9051 at commit
[`5588157`](https://github.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147439789
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147439788
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147404975
[Test build #43568 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43568/consoleFull)
for PR 9051 at commit
[`5588157`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147403468
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147403438
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147402862
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147401309
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147401247
[Test build #43567 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43567/console)
for PR 9051 at commit
[`5588157`](https://github.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147401311
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147396129
[Test build #43567 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43567/consoleFull)
for PR 9051 at commit
[`5588157`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147394979
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147394925
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-147394955
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9051#issuecomment-146994822
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pr
GitHub user d2r opened a pull request:
https://github.com/apache/spark/pull/9051
[SPARK-10930] Adds max task duration to all stages page
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/d2r/spark spark-10930-ui-max-task-dur
Alter
40 matches
Mail list logo
