[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1384 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-54101534 I think this was ultimately fixed by #1545 so we can close this issue. But feel free to open another PR if that one did not fix this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-50149066 @tsudukim yes,SPARK-2298 is that i want to. but i think a simple way is on this PR add a jobid column to stage table.it is very easy to achieve it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-49944917 @lianhuiwang It appears to be a different problem to SPARK-2298. Is your aim same as this ticket? https://issues.apache.org/jira/browse/SPARK-1362 If so, how about creating another PR to modify it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-49944747 @rxin Surely we can also fix them all in one patch. But it can be a little bit hard work to modify them compatibly in one patch so I just have thought to separate into several tasks and to make #1384 as first step by showing only attemptId to distinguish attempts. You can take whichever is convenient for you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-49860984 i think you can add jobid to stageTable. because jobid is very useful when a application has many jobs.that can distinguish every job's stages. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-49850442 It turned out much trickier than I thought to add attempt id. I submitted a PR here #1545 That PR already modifies the UI, since that's the only way I could test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-49495727 Modified PR as your comments. thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-49209857 @rxin in #1262, can I expect the key of the stagedata in JobProgressListener become stageId + attemptId instead of stageId only? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-49208410 Let's hold off merging this one until we merge #1262. Then it will be easier to index the information based on stage + attempt. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-49209319 @rxin OK. After that, I think I can make this patch better. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user tsudukim commented on a diff in the pull request:
https://github.com/apache/spark/pull/1384#discussion_r15019018
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -478,6 +479,7 @@ private[spark] object JsonProtocol {
def stageInfoFromJson(json: JValue): StageInfo = {
val stageId = (json \ "Stage ID").extract[Int]
+val attemptId = (json \ "Attempt ID").extract[Int]
--- End diff --
Sure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-49207731 @pwendell I agree that there are many room for improvement about handling of stageId and attemptId. It might be better to break this problems into some sub-tasks. I think this patch should be one of them. Or did you mean we should fix all of this problem in one patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1384#discussion_r14977192
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -478,6 +479,7 @@ private[spark] object JsonProtocol {
def stageInfoFromJson(json: JValue): StageInfo = {
val stageId = (json \ "Stage ID").extract[Int]
+val attemptId = (json \ "Attempt ID").extract[Int]
--- End diff --
Could you add a test in `JsonProtocolSuite`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-49111726 @tsudukim I created a JIRA to deal with the broader issue. If you want to take that on as well, let me know: https://issues.apache.org/jira/browse/SPARK-2501 it might make sense to wrap it into this patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-49111342 @rxin is this something you've thought about in your various schedule refactoring things? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-4990 @tsudukim @kayousterhout so I think in general here, our handling of stage re-submissions is broken in the UI. For instance, I looked in the `JobProgressListener` and we index many things on `StageId` that might better be indexed on `StageId, AttemptId`. Also, we should probably give the `AttemptId` when start a task so that we understand which stage attempt to associated it with. I also don't understand exactly what happens when a stage gets re-attempted, do we send a "stage completed" event? It might be good to fix the way we deal with stage re-submissions to make this work better overall. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user tsudukim commented on a diff in the pull request:
https://github.com/apache/spark/pull/1384#discussion_r14976665
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -478,6 +479,7 @@ private[spark] object JsonProtocol {
def stageInfoFromJson(json: JValue): StageInfo = {
val stageId = (json \ "Stage ID").extract[Int]
+val attemptId = (json \ "Attempt ID").extract[Int]
--- End diff --
Ah, no it's not. How about this?
```
val attemptId = (json \ "Attempt ID").extractOpt[Int].getOrElse(0)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1384#discussion_r14975903
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -478,6 +479,7 @@ private[spark] object JsonProtocol {
def stageInfoFromJson(json: JValue): StageInfo = {
val stageId = (json \ "Stage ID").extract[Int]
+val attemptId = (json \ "Attempt ID").extract[Int]
--- End diff --
Is this backwards compatible?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-48963364 I'm wondering how to show it. I gave it a shot. Is it smart?  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-48804157 Yup - but let's avoid exposing the concept of TaskSet to users in the UI. That's only for internal engineering. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-48804094 @xrin OK, thanks. Then attempt id is still required in the web ui for users to know stage + attempt. Have I got that right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-48803729 @tsudukim The concept of TaskSet should be internal to Spark. Users shouldn't have to aware of task set. Users should only care about stage + attempt. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-48803681 @andrewor14 Thank you for your comment. I think it is more weird if the display style of ID/attempt changes by conditions. Surely most stages will only have 1 attempt, but as the set of stage id and stage attempt id is the identifier of the taskset, I'd like to show the attempt id even if the stage is the first one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-48802862 Hm the latest screenshot looks a little funky to me. Most stages will only have 1 attempt, so I think it makes sense to only show the attempt if this is not the first one. Something like: ``` ID 4 2 3 (Attempt 2) 1 0 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-48802747 @pwendell Thank you for your response. You mean like this?  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-48801078 /cc @rxin who is interested in this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-48800948 To make this a bit more concise, what about having one column on the left whose header is `ID: Attempt` and separating them with a colon. Current the word "Stage" is redundant there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-48800698 Attempt Id shows up in web ui. Submitted and Duration became individual value to stage attempts.  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1384#issuecomment-48800648 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2298: Show stage attempt in UI
GitHub user tsudukim opened a pull request: https://github.com/apache/spark/pull/1384 SPARK-2298: Show stage attempt in UI Added attempt ID column into stage page of webUI. Added attemptId handling code into StageInfo, JsonProtocol. Modified DAGScheduler to identify stages whose stageId is same but attemptId is different. Modified testcode for stage attempt ID. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tsudukim/spark feature/SPARK-2298 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1384.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1384 commit 0ea38ae39e34dacb8adef96cfbb7ce728bb3406a Author: Masayoshi TSUZUKI Date: 2014-07-11T19:02:35Z SPARK-2298 Show stage attempt in UI Added attempt ID column into stage page of webUI. Added attemptId handling code into StageInfo, JsonProtocol. Modified DAGScheduler to identify stages whose stageId is same but attemptId is different. Modified testcode for stage attempt ID. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
