[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118763844 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/7038 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118954773 Ok, I'm merging this into master. Thanks @thegiive. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7038#discussion_r33873825
--- Diff:
core/src/main/scala/org/apache/spark/deploy/master/WorkerInfo.scala ---
@@ -107,4 +107,6 @@ private[spark] class WorkerInfo(
def setState(state: WorkerState.Value): Unit = {
this.state = state
}
+
+ def isAlive(): Boolean = this.state == WorkerState.ALIVE
--- End diff --
we probably don't need this since we only use it in 1 place
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7038#discussion_r33873807
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -76,12 +76,13 @@ private[deploy] object JsonProtocol {
}
def writeMasterState(obj: MasterStateResponse): JObject = {
+val alive_workers = obj.workers.filter(_.isAlive())
--- End diff --
in Spark we generally use camel case, so:
```
val aliveWorkers = obj.workers.filter(_.state == WorkerState.ALIVE)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user thegiive commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118449368 HI @andrewor14 The test is passed. If there is any problem, please let me know. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118438713 [Test build #36515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36515/console) for PR 7038 at commit [`9e54bf0`](https://github.com/apache/spark/commit/9e54bf09c75594941ff46a767af4d66d285fb63f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118438727 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user thegiive commented on a diff in the pull request:
https://github.com/apache/spark/pull/7038#discussion_r33881284
--- Diff:
core/src/main/scala/org/apache/spark/deploy/master/WorkerInfo.scala ---
@@ -107,4 +107,6 @@ private[spark] class WorkerInfo(
def setState(state: WorkerState.Value): Unit = {
this.state = state
}
+
+ def isAlive(): Boolean = this.state == WorkerState.ALIVE
--- End diff --
Hi Andrew
Same reason as my comment. I think this is only way we can easily add one
line code without too much trouble
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user thegiive commented on a diff in the pull request:
https://github.com/apache/spark/pull/7038#discussion_r33881472
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -76,12 +76,13 @@ private[deploy] object JsonProtocol {
}
def writeMasterState(obj: MasterStateResponse): JObject = {
+val alive_workers = obj.workers.filter(_.isAlive())
--- End diff --
BTW, I will change the variable name to camel case without any problem
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7038#discussion_r33882364
--- Diff:
core/src/main/scala/org/apache/spark/deploy/master/WorkerInfo.scala ---
@@ -107,4 +107,6 @@ private[spark] class WorkerInfo(
def setState(state: WorkerState.Value): Unit = {
this.state = state
}
+
+ def isAlive(): Boolean = this.state == WorkerState.ALIVE
--- End diff --
ah ok, that sounds fine then
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118431806 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118431800 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user thegiive commented on a diff in the pull request:
https://github.com/apache/spark/pull/7038#discussion_r33881442
--- Diff:
core/src/main/scala/org/apache/spark/deploy/master/WorkerInfo.scala ---
@@ -107,4 +107,6 @@ private[spark] class WorkerInfo(
def setState(state: WorkerState.Value): Unit = {
this.state = state
}
+
+ def isAlive(): Boolean = this.state == WorkerState.ALIVE
--- End diff --
Hi Andrew
Base on same reason on previous. I think best way to do that is add a one
line method on WorkerInfo
How do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user thegiive commented on a diff in the pull request:
https://github.com/apache/spark/pull/7038#discussion_r33881264
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -76,12 +76,13 @@ private[deploy] object JsonProtocol {
}
def writeMasterState(obj: MasterStateResponse): JObject = {
+val alive_workers = obj.workers.filter(_.isAlive())
--- End diff --
Hi Andrew
WorkerState is org.apache.spark.deploy.master's private object.
JsonProtocol cannot resolved WorkerState due to class visibility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user thegiive commented on a diff in the pull request:
https://github.com/apache/spark/pull/7038#discussion_r33881396
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -76,12 +76,13 @@ private[deploy] object JsonProtocol {
}
def writeMasterState(obj: MasterStateResponse): JObject = {
+val alive_workers = obj.workers.filter(_.isAlive())
--- End diff --
Hi Andrew
WorkerState is org.apache.spark.deploy.master's private object.
JsonProtocol cannot resolved WorkerState due to class visibility.This code
cannot work unless we change WorkerState code.
I think best way to do that is add a one line method on WorkerInfo
How do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118431965 [Test build #36515 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36515/consoleFull) for PR 7038 at commit [`9e54bf0`](https://github.com/apache/spark/commit/9e54bf09c75594941ff46a767af4d66d285fb63f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-117936713 ic - to me it is almost incorrect in the past, and we are fixing a bug here ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-117922942 what's the non-compatible part? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-117922427 Hm I just saw @srowen's comment. Actually this change is not backward compatible so I'm not sure if we can merge it. @rxin any thoughts? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-117936180 I think just that the response will have different information now -- are people relying on and properly expecting to get info for all workers in the JSON? then removing non-alive worker info is incompatible in that sense. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118214540 [Test build #36464 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36464/consoleFull) for PR 7038 at commit [`2c8ea89`](https://github.com/apache/spark/commit/2c8ea89b00e95b00b82092fda9d514336e96bac1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118214510 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user thegiive commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118214522 Hi all reviewer I modified the code base on the comment. Can you check if it is ok ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118214499 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118227376 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-118227357 [Test build #36464 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36464/console) for PR 7038 at commit [`2c8ea89`](https://github.com/apache/spark/commit/2c8ea89b00e95b00b82092fda9d514336e96bac1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-116832820 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-116833438 [Test build #36030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36030/consoleFull) for PR 7038 at commit [`431d2b0`](https://github.com/apache/spark/commit/431d2b04142d84dfd605c5f687fa0c4ddc39485e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-116834206 [Test build #36030 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36030/console) for PR 7038 at commit [`431d2b0`](https://github.com/apache/spark/commit/431d2b04142d84dfd605c5f687fa0c4ddc39485e). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-116832022 @thegiive have you verified that these are the only sources of mismatch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-116834403 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/7038#discussion_r33508453
--- Diff:
core/src/main/scala/org/apache/spark/deploy/master/WorkerInfo.scala ---
@@ -107,4 +107,8 @@ private[spark] class WorkerInfo(
def setState(state: WorkerState.Value): Unit = {
this.state = state
}
+
+ def isAlive(): Boolean={
--- End diff --
please add space around `=`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-116831567 OK TO TEST --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-116831720 I think we should have the JSON protocol do whatever the normal UI does. This change looks fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user thegiive commented on a diff in the pull request:
https://github.com/apache/spark/pull/7038#discussion_r33364777
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -77,11 +77,11 @@ private[deploy] object JsonProtocol {
def writeMasterState(obj: MasterStateResponse): JObject = {
(url - obj.uri) ~
-(workers - obj.workers.toList.map(writeWorkerInfo)) ~
-(cores - obj.workers.map(_.cores).sum) ~
-(coresused - obj.workers.map(_.coresUsed).sum) ~
-(memory - obj.workers.map(_.memory).sum) ~
-(memoryused - obj.workers.map(_.memoryUsed).sum) ~
+(workers -
obj.workers.filter(_.isAlive()).toList.map(writeWorkerInfo)) ~
--- End diff --
I can refactor this better later. Let me find the API author first. Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-115635388 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user thegiive commented on a diff in the pull request:
https://github.com/apache/spark/pull/7038#discussion_r33364667
--- Diff:
core/src/main/scala/org/apache/spark/deploy/master/WorkerInfo.scala ---
@@ -107,4 +107,8 @@ private[spark] class WorkerInfo(
def setState(state: WorkerState.Value): Unit = {
this.state = state
}
+
+ def isAlive(): Boolean={
--- End diff --
The reason why I write a method here is due to JsonProtocol.scala cannot
access WorkerState.ALIVE.
Changing the WorkerState from private to other permission is not good
approach. I think it is the best way to keep the current class/namespace struct
and get thing done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user thegiive commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-115725659 HI Sean, thanks for your opinion. My feeling is that WebUI and API is just different presentation format. But the content should be the same. Different meaning in the same column(workers,memory,core) will confuse the user. BTW, Let me talk to API author first :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user thegiive commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-115898969 Base on git blame, I think the JSON API creator is @rxin ( 9db1e503 ) Hi Reynold #6317 has already change some master UI's data's definition (total memory/memory in used , total core/core in used ) . Base on the same reason list in #6317 and the WebUI should sync the number with JSON API. I think it is reasonable to sync the JSON API's column definition. How do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/7038#issuecomment-115647605 This is changing the semantics of the response though. I don't think you can necessarily do this. It is not necessarily true that what is presented in the web UI for human consumption is identical to what's in the JSON response for machine consumption. You should ask the person who created this API to see if there is or should be a way to get just alive workers status. Is this info not in the response? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7038#discussion_r33346345
--- Diff: core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala ---
@@ -77,11 +77,11 @@ private[deploy] object JsonProtocol {
def writeMasterState(obj: MasterStateResponse): JObject = {
(url - obj.uri) ~
-(workers - obj.workers.toList.map(writeWorkerInfo)) ~
-(cores - obj.workers.map(_.cores).sum) ~
-(coresused - obj.workers.map(_.coresUsed).sum) ~
-(memory - obj.workers.map(_.memory).sum) ~
-(memoryused - obj.workers.map(_.memoryUsed).sum) ~
+(workers -
obj.workers.filter(_.isAlive()).toList.map(writeWorkerInfo)) ~
--- End diff --
Why do the filtering so many times?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7038#discussion_r33346369
--- Diff:
core/src/main/scala/org/apache/spark/deploy/master/WorkerInfo.scala ---
@@ -107,4 +107,8 @@ private[spark] class WorkerInfo(
def setState(state: WorkerState.Value): Unit = {
this.state = state
}
+
+ def isAlive(): Boolean={
--- End diff --
No parens, return type or braces are needed. In fact is it worth writing a
method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8656][WebUI] Fix the webUI and JSON API...
GitHub user thegiive opened a pull request: https://github.com/apache/spark/pull/7038 [SPARK-8656][WebUI] Fix the webUI and JSON API number is not synced Spark standalone master web UI show Alive workers worker number, Alive Workers total core, total used cores and Alive workers total memory, memory used. But the JSON API page http://MASTERURL:8088/json; shows all workers worker number, core, memory number. This webUI data is not sync with the JSON API. The proper way is to sync the number with webUI and JSON API. You can merge this pull request into a Git repository by running: $ git pull https://github.com/thegiive/spark SPARK-8656 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7038.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7038 commit 8b3b8e8f830d1148e43f050b6b0011817ea1262e Author: Wisely Chen wiselyc...@appier.com Date: 2015-06-26T10:21:06Z [SPARK-8656] Fix the webUI and JSON API number is not synced --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
