[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user sarutak closed the pull request at: https://github.com/apache/spark/pull/2333 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-75205810 OK. I close this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-75155273 @sarutak thanks a lot for working on this. There is another patch #4435 that provides a more elaborate implementation of the functionality here. Given that this PR has mostly gone stale at this point, would you mind closing this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-71532457 Hi @sarutak thanks for your work on this. Josh's other PR https://github.com/apache/spark/pull/2696 has been merged for a while now. I'm gonna take another crack at implementing this on top of the lastest changes and getting this integrated into the history server too. Can you please close this PR (unless you are planning on updating it in the very near future)? Hopefully that will let your other PR https://github.com/apache/spark/pull/2342 to get merged in -- looks really cool! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-56677764 Thank you for you work @JoshRosen ! I'll check it out. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-56419802 I've opened [SPARK-3644](https://issues.apache.org/jira/browse/SPARK-3644) as a forum for discussing the design of a REST API; sorry for the delay. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55966735 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55967426 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20493/consoleFull) for PR 2333 at commit [`7b4d6eb`](https://github.com/apache/spark/commit/7b4d6eb92fbbf758854bf322608ed3ae2e68ebed). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55974879 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20493/consoleFull) for PR 2333 at commit [`7b4d6eb`](https://github.com/apache/spark/commit/7b4d6eb92fbbf758854bf322608ed3ae2e68ebed). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55976178 @sarutak, I believe @JoshRosen is working on a more general framework for extracting the info displayed on the UI as JSON, so there's a chance that we won't go with the approach here. Just a heads up. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55976627 @andrewor14 Thank you for notification! Actually, I need JSON representation for #2342 . I'm planning to parse JSON to use D3. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55977729 @sarutak In the long run, I'd be interested in re-writing the UI in terms of a richer REST API that exposes data as JSON, exactly for the visualization use-case that you mentioned. This change (Javascript-based rendering, etc) is blocked by a couple of other features right now. I'm swamped with a bunch of other work, but I'll try to get a public design draft posted in the next couple of days. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55978533 Thanks @JoshRosen ! So, for now, I use self implementation for #2342 and I'll use the feature you'll try. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-7935 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20326/consoleFull) for PR 2333 at commit [`d41b3ca`](https://github.com/apache/spark/commit/d41b3caf1adb0c807aa6ce9d011e5e2553408fe2). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class JavaSparkContext(val sc: SparkContext)` * ` throw new IllegalStateException(The main method in the given main class must be static)` * `class TaskCompletionListenerException(errorMessages: Seq[String]) extends Exception ` * `class Dummy(object):` * `class RatingDeserializer(FramedSerializer):` * ` class Encoder[T : NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T] ` * ` class Encoder[T : NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T] ` * ` class Encoder[T : NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T] ` * ` class Encoder extends compression.Encoder[IntegerType.type] ` * ` class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[IntegerType.type])` * ` class Encoder extends compression.Encoder[LongType.type] ` * ` class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[LongType.type])` * `class JavaStreamingContext(val ssc: StreamingContext) extends Closeable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-4685 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-4859 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20326/consoleFull) for PR 2333 at commit [`d41b3ca`](https://github.com/apache/spark/commit/d41b3caf1adb0c807aa6ce9d011e5e2553408fe2). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55513182 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20289/consoleFull) for PR 2333 at commit [`7f51a4f`](https://github.com/apache/spark/commit/7f51a4f1a979221ba8685662a66170c42f0447e9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55514222 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20289/consoleFull) for PR 2333 at commit [`7f51a4f`](https://github.com/apache/spark/commit/7f51a4f1a979221ba8685662a66170c42f0447e9). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55454772 @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17442957
--- Diff: core/src/main/scala/org/apache/spark/ui/env/EnvironmentPage.scala
---
@@ -26,6 +29,23 @@ import org.apache.spark.ui.{UIUtils, WebUIPage}
private[ui] class EnvironmentPage(parent: EnvironmentTab) extends
WebUIPage() {
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+val jvmInfoJson =
+ (RUntime Informationf -
listener.jvmInformation.foldLeft(JObject())(_ ~ _))
--- End diff --
There are a lot of typos in this line...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443083
--- Diff: core/src/main/scala/org/apache/spark/ui/env/EnvironmentPage.scala
---
@@ -26,6 +29,23 @@ import org.apache.spark.ui.{UIUtils, WebUIPage}
private[ui] class EnvironmentPage(parent: EnvironmentTab) extends
WebUIPage() {
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+val jvmInfoJson =
+ (RUntime Informationf -
listener.jvmInformation.foldLeft(JObject())(_ ~ _))
--- End diff --
Can you show what the final JSON looks like? Since this is a map it might
be good to convert this faithfully, e.g.
```
{
Spark Properties: {
spark.master:local
spark.app.name:spark-shell
...
}
...
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443185
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
---
@@ -44,6 +47,30 @@ private case class ExecutorSummaryInfo(
private[ui] class ExecutorsPage(parent: ExecutorsTab) extends
WebUIPage() {
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+val storageStatusList = listener.storageStatusList
+
+val execInfoJsonList = for (statusId - 0 until
storageStatusList.size) yield {
+ val execInfo = getExecInfo(statusId)
+ (Executor ID - execInfo.id) ~
+ (Address - execInfo.hostPort) ~
+ (RDD Blocks - execInfo.rddBlocks) ~
+ (Memory Used - execInfo.memoryUsed) ~
+ (Disk Used - execInfo.diskUsed) ~
+ (Active Tasks - execInfo.activeTasks) ~
+ (Failed Tasks - execInfo.failedTasks) ~
+ (Complete Tasks - execInfo.completedTasks) ~
+ (TotalTasks - execInfo.totalTasks) ~
--- End diff --
space
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443226
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressPage.scala ---
@@ -31,6 +36,35 @@ private[ui] class JobProgressPage(parent:
JobProgressTab) extends WebUIPage()
private val listener = parent.listener
private lazy val isFairScheduler = parent.isFairScheduler
+ override def renderJson(request: HttpServletRequest): JValue = {
+listener.synchronized {
+
+ val activeStageList = listener.activeStages.values.map {
+case info: StageInfo =
+ JsonProtocol.stageInfoToJson(info)
+ }
--- End diff --
minor: just do
```
listener.activeStages.values.map { info =
JsonProtocol.stageInfoToJson(info) }
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443244
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressPage.scala ---
@@ -31,6 +36,35 @@ private[ui] class JobProgressPage(parent:
JobProgressTab) extends WebUIPage()
private val listener = parent.listener
private lazy val isFairScheduler = parent.isFairScheduler
+ override def renderJson(request: HttpServletRequest): JValue = {
+listener.synchronized {
+
+ val activeStageList = listener.activeStages.values.map {
+case info: StageInfo =
+ JsonProtocol.stageInfoToJson(info)
+ }
--- End diff --
here and other places
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443265
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressPage.scala ---
@@ -31,6 +36,35 @@ private[ui] class JobProgressPage(parent:
JobProgressTab) extends WebUIPage()
private val listener = parent.listener
private lazy val isFairScheduler = parent.isFairScheduler
+ override def renderJson(request: HttpServletRequest): JValue = {
+listener.synchronized {
+
+ val activeStageList = listener.activeStages.values.map {
+case info: StageInfo =
+ JsonProtocol.stageInfoToJson(info)
+ }
+ val activeStageJson = (Active Stages - activeStageList)
+
+ val completedStageList = listener.completedStages.reverse.map {
+case info: StageInfo =
+ JsonProtocol.stageInfoToJson(info)
+ }
+ val completedStageJson = (Completed Stages - completedStageList)
+
+ val failedStageList = listener.failedStages.reverse.map {
+case info: StageInfo =
+ JsonProtocol.stageInfoToJson(info)
+ }
+ val failedStageJson = (Failed Stages - failedStageList)
+
+ (Stages Info -
+(Scheduling Mode -
listener.schedulingMode.map(_.toString).getOrElse(Unknown)) ~
--- End diff --
Can you put this in a variable so this is more readable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443289
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/PoolPage.scala ---
@@ -30,6 +35,35 @@ private[ui] class PoolPage(parent: JobProgressTab)
extends WebUIPage(pool) {
private val sc = parent.sc
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+listener.synchronized {
+ val poolName = request.getParameter(poolname)
+ val poolToActiveStages = listener.poolToActiveStages
+ val activeStages = poolToActiveStages.get(poolName) match {
+case Some(s) = s.values.map {
+ case info: StageInfo =
+JsonProtocol.stageInfoToJson(info)
+}
+case None = Seq[JValue]()
+ }
+
+ val pools = if (live) Seq(sc.getPoolForName(poolName).get) else
Seq[Schedulable]()
+
+ val poolList = pools.map {
+case schedulable: Schedulable =
--- End diff --
same here, no need to do `case` and specify the type
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/2333#discussion_r17443318 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -20,17 +20,76 @@ package org.apache.spark.ui.jobs import java.util.Date import javax.servlet.http.HttpServletRequest +import org.json4s.JsonAST.JNothing --- End diff -- minor, group this import with other `org.json4s` imports --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17443332
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -20,17 +20,76 @@ package org.apache.spark.ui.jobs
import java.util.Date
import javax.servlet.http.HttpServletRequest
+import org.json4s.JsonAST.JNothing
+
import scala.xml.{Node, Unparsed}
+import org.json4s.{JObject, JValue}
+import org.json4s.JsonDSL._
+
import org.apache.spark.ui.{ToolTips, WebUIPage, UIUtils}
import org.apache.spark.ui.jobs.UIData._
-import org.apache.spark.util.{Utils, Distribution}
+import org.apache.spark.util.{JsonProtocol, Utils, Distribution}
import org.apache.spark.scheduler.AccumulableInfo
/** Page showing statistics and task list for a given stage */
private[ui] class StagePage(parent: JobProgressTab) extends
WebUIPage(stage) {
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+val stageId = request.getParameter(id).toInt
+val stageAttemptId = request.getParameter(attempt).toInt
+
+var stageSummary = (Stage ID - stageId) ~ (Stage Attempt ID -
stageAttemptId)
+
--- End diff --
extra new line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55316029 Hi @sarutak, thanks for working on this feature. There has been request of this on the mailing list and it's good to see this being done. There are a lot of style issues with the current state of the PR, however. Another thing is I don't think we need to prepend the category to the Json blob (e.g. Environment or RDDs), since the user should know what they asked for. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55356808 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20200/consoleFull) for PR 2333 at commit [`72c0644`](https://github.com/apache/spark/commit/72c06445211579a207b36de2d4072974fe7325de). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/2333#discussion_r17461531
--- Diff: core/src/main/scala/org/apache/spark/ui/env/EnvironmentPage.scala
---
@@ -26,6 +29,23 @@ import org.apache.spark.ui.{UIUtils, WebUIPage}
private[ui] class EnvironmentPage(parent: EnvironmentTab) extends
WebUIPage() {
private val listener = parent.listener
+ override def renderJson(request: HttpServletRequest): JValue = {
+val jvmInfoJson =
+ (RUntime Informationf -
listener.jvmInformation.foldLeft(JObject())(_ ~ _))
--- End diff --
This is the result of JSON for environment.
{
Runtime Information : {
Java Home : /usr/java/jdk1.7.0_65/jre,
Java Version : 1.7.0_65 (Oracle Corporation),
Scala Version : version 2.10.4
},
Spark Properties : {
spark.serializer : org.apache.spark.serializer.KryoSerializer,
...
spark.tachyonStore.folderName :
spark-a8ac134b-187d-4d4b-b9ef-140e626ee96c
},
System Properties : {
java.io.tmpdir : /tmp,
...
java.version : 1.7.0_65,
sun.io.unicode.encoding : UnicodeLittle
},
Classpath Entries : {
/home/sarutak/work/spark/sql/hive/target/scala-2.10/test-classes :
System Classpath,
...
/home/sarutak/work/spark/assembly/target/scala-2.10/spark-assembly-1.1.0-SNAPSHOT-hadoop2.2.0.jar
: System Classpath
}
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55357150 @andrewor14 Thank you for reviewing this PR. I agree with removing category from the JSON blob and have done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55359688 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20200/consoleFull) for PR 2333 at commit [`72c0644`](https://github.com/apache/spark/commit/72c06445211579a207b36de2d4072974fe7325de). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55248298 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20152/consoleFull) for PR 2333 at commit [`a349d0e`](https://github.com/apache/spark/commit/a349d0e26765174fd9147f6fced4693d5dc48c98). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55248376 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20146/consoleFull) for PR 2333 at commit [`c108200`](https://github.com/apache/spark/commit/c1082008ee54aef85a023cb2551c8ca12e33feea). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3454] Expose JSON representation of dat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2333#issuecomment-55253730 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20152/consoleFull) for PR 2333 at commit [`a349d0e`](https://github.com/apache/spark/commit/a349d0e26765174fd9147f6fced4693d5dc48c98). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
