[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/14414


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[mgummelt]

Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73949486
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
 ---
@@ -152,8 +152,13 @@ private[spark] class 
MesosCoarseGrainedSchedulerBackend(
   sc.sparkUser,
   sc.appName,
   sc.conf,
-  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress))
+  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress)),
+  None,
+  None,
+  sc.conf.getOption("spark.mesos.driver.frameworkId")
 )
+
+unsetFrameworkID(sc)
--- End diff --

> If no id is specified what will happen?

Mesos generates a unique framework ID when none is specified.  This PR only 
sets the framework ID on the first spark context created by the driver.

> What if you could use a unique id as aframeworkId plus an increment, so 
the scheduler can detect all frameworks even the ones coming from multiple 
contexts? 

We can't guarantee no collisions with that approach.  It would also be 
difficult for the dispatcher to determine how many frameworks have been 
started. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[mgummelt]

Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73948765
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
 ---
@@ -37,32 +37,36 @@ import org.apache.spark.util.Utils
 
 /**
  * Tracks the current state of a Mesos Task that runs a Spark driver.
+ *
  * @param driverDescription Submitted driver description from
  * [[org.apache.spark.deploy.rest.mesos.MesosRestServer]]
  * @param taskId Mesos TaskID generated for the task
  * @param slaveId Slave ID that the task is assigned to
  * @param mesosTaskStatus The last known task status update.
  * @param startDate The date the task was launched
--- End diff --

fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[skonto]

Github user skonto commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73874399
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
 ---
@@ -152,8 +152,13 @@ private[spark] class 
MesosCoarseGrainedSchedulerBackend(
   sc.sparkUser,
   sc.appName,
   sc.conf,
-  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress))
+  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress)),
+  None,
+  None,
+  sc.conf.getOption("spark.mesos.driver.frameworkId")
 )
+
+unsetFrameworkID(sc)
--- End diff --

If the driver creates two spark contexts shouldnt it use two different 
spark confs with that property set up differently?
If no id is specified what will happen? I guess the dispatcher will not 
know anything. What if you could use a unique id as aframeworkId plus an 
increment, so the scheduler can detect all frameworks even the ones coming from 
multiple contexts? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[skonto]

Github user skonto commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73872335
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
 ---
@@ -37,32 +37,36 @@ import org.apache.spark.util.Utils
 
 /**
  * Tracks the current state of a Mesos Task that runs a Spark driver.
+ *
  * @param driverDescription Submitted driver description from
  * [[org.apache.spark.deploy.rest.mesos.MesosRestServer]]
  * @param taskId Mesos TaskID generated for the task
  * @param slaveId Slave ID that the task is assigned to
  * @param mesosTaskStatus The last known task status update.
  * @param startDate The date the task was launched
--- End diff --

FinishDate is missing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[mgummelt]

Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73386654
  
--- Diff: docs/running-on-mesos.md ---
@@ -468,6 +468,17 @@ See the [configuration page](configuration.html) for 
information on Spark config
 If unset it will point to Spark's internal web UI.
   
 
+
+  spark.mesos.dispatcher.historyServer.url
+  (none)
+  
+Set the URL of the http://spark.apache.org/docs/latest/monitoring.html#viewing-after-the-fact;>history
+server>.  The dispatcher will then link each driver to its entry
--- End diff --

fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[mgummelt]

Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73386547
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
 ---
@@ -152,8 +152,13 @@ private[spark] class 
MesosCoarseGrainedSchedulerBackend(
   sc.sparkUser,
   sc.appName,
   sc.conf,
-  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress))
+  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress)),
+  Option.empty,
+  Option.empty,
+  sc.conf.getOption("spark.mesos.driver.frameworkId")
--- End diff --

see above


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[mgummelt]

Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73386516
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
 ---
@@ -152,8 +152,13 @@ private[spark] class 
MesosCoarseGrainedSchedulerBackend(
   sc.sparkUser,
   sc.appName,
   sc.conf,
-  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress))
+  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress)),
+  Option.empty,
--- End diff --

fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[mgummelt]

Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73386336
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
 ---
@@ -152,8 +152,13 @@ private[spark] class 
MesosCoarseGrainedSchedulerBackend(
   sc.sparkUser,
   sc.appName,
   sc.conf,
-  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress))
+  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress)),
--- End diff --

I agree, but I also want to make this change consistently, so let's make 
this part of https://issues.apache.org/jira/browse/SPARK-16881


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[mgummelt]

Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73386256
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
 ---
@@ -364,7 +377,12 @@ private[spark] class MesosClusterScheduler(
   val executorEnv = Map("SPARK_EXECUTOR_OPTS" -> executorOpts)
   val driverEnv = desc.conf.getAllWithPrefix("spark.mesos.driverEnv.")
 
-  driverEnv ++ executorEnv ++ desc.command.environment
+  var commandEnv = adjust(desc.command.environment, 
"SPARK_SUBMIT_OPTS", "")(
+v => s"$v 
-Dspark.mesos.driver.frameworkId=${getDriverFrameworkID(desc)}"
+  )
+
+
--- End diff --

fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[mgummelt]

Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73386197
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -28,10 +28,17 @@ import 
org.apache.spark.scheduler.cluster.mesos.MesosClusterSubmissionState
 import org.apache.spark.ui.{UIUtils, WebUIPage}
 
 private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends 
WebUIPage("") {
+  private val historyServerURL = 
parent.conf.getOption("spark.mesos.dispatcher.historyServer.url")
+
   def render(request: HttpServletRequest): Seq[Node] = {
 val state = parent.scheduler.getSchedulerState()
-val queuedHeaders = Seq("Driver ID", "Submit Date", "Main Class", 
"Driver Resources")
-val driverHeaders = queuedHeaders ++
+
+val driverHeader = Seq("Driver ID")
--- End diff --

Just so I can concatenate the sequences to produce a list of headers


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[mgummelt]

Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73385988
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -28,10 +28,17 @@ import 
org.apache.spark.scheduler.cluster.mesos.MesosClusterSubmissionState
 import org.apache.spark.ui.{UIUtils, WebUIPage}
 
 private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends 
WebUIPage("") {
+  private val historyServerURL = 
parent.conf.getOption("spark.mesos.dispatcher.historyServer.url")
+
   def render(request: HttpServletRequest): Seq[Node] = {
 val state = parent.scheduler.getSchedulerState()
-val queuedHeaders = Seq("Driver ID", "Submit Date", "Main Class", 
"Driver Resources")
-val driverHeaders = queuedHeaders ++
+
+val driverHeader = Seq("Driver ID")
+val historyHeader = if (historyServerURL.isDefined) Seq("History") 
else Nil
--- End diff --

fixed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[mgummelt]

Github user mgummelt commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73385194
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -28,10 +28,17 @@ import 
org.apache.spark.scheduler.cluster.mesos.MesosClusterSubmissionState
 import org.apache.spark.ui.{UIUtils, WebUIPage}
 
 private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends 
WebUIPage("") {
+  private val historyServerURL = 
parent.conf.getOption("spark.mesos.dispatcher.historyServer.url")
--- End diff --

Ah, I wasn't aware of that class.  We should definitely use it, but I'd 
like to make that change consistently.  I made a JIRA for the migration: 
https://issues.apache.org/jira/browse/SPARK-16881


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[jaceklaskowski]

Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73033311
  
--- Diff: docs/running-on-mesos.md ---
@@ -468,6 +468,17 @@ See the [configuration page](configuration.html) for 
information on Spark config
 If unset it will point to Spark's internal web UI.
   
 
+
+  spark.mesos.dispatcher.historyServer.url
+  (none)
+  
+Set the URL of the http://spark.apache.org/docs/latest/monitoring.html#viewing-after-the-fact;>history
+server>.  The dispatcher will then link each driver to its entry
--- End diff --

Looks like the ending tag is mistyped.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[jaceklaskowski]

Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73033008
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
 ---
@@ -152,8 +152,13 @@ private[spark] class 
MesosCoarseGrainedSchedulerBackend(
   sc.sparkUser,
   sc.appName,
   sc.conf,
-  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress))
+  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress)),
+  Option.empty,
+  Option.empty,
+  sc.conf.getOption("spark.mesos.driver.frameworkId")
--- End diff --

Mind introducing a constant for `spark.mesos.driver.frameworkId`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[jaceklaskowski]

Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73032952
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
 ---
@@ -152,8 +152,13 @@ private[spark] class 
MesosCoarseGrainedSchedulerBackend(
   sc.sparkUser,
   sc.appName,
   sc.conf,
-  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress))
+  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress)),
+  Option.empty,
--- End diff --

`None`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[jaceklaskowski]

Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73032912
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
 ---
@@ -152,8 +152,13 @@ private[spark] class 
MesosCoarseGrainedSchedulerBackend(
   sc.sparkUser,
   sc.appName,
   sc.conf,
-  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress))
+  
sc.conf.getOption("spark.mesos.driver.webui.url").orElse(sc.ui.map(_.appUIAddress)),
--- End diff --

Mind introducing a constant for `spark.mesos.driver.webui.url`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[jaceklaskowski]

Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73032820
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
 ---
@@ -364,7 +377,12 @@ private[spark] class MesosClusterScheduler(
   val executorEnv = Map("SPARK_EXECUTOR_OPTS" -> executorOpts)
   val driverEnv = desc.conf.getAllWithPrefix("spark.mesos.driverEnv.")
 
-  driverEnv ++ executorEnv ++ desc.command.environment
+  var commandEnv = adjust(desc.command.environment, 
"SPARK_SUBMIT_OPTS", "")(
+v => s"$v 
-Dspark.mesos.driver.frameworkId=${getDriverFrameworkID(desc)}"
+  )
+
+
--- End diff --

Remove the new line?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[jaceklaskowski]

Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73032511
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -28,10 +28,17 @@ import 
org.apache.spark.scheduler.cluster.mesos.MesosClusterSubmissionState
 import org.apache.spark.ui.{UIUtils, WebUIPage}
 
 private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends 
WebUIPage("") {
+  private val historyServerURL = 
parent.conf.getOption("spark.mesos.dispatcher.historyServer.url")
+
   def render(request: HttpServletRequest): Seq[Node] = {
 val state = parent.scheduler.getSchedulerState()
-val queuedHeaders = Seq("Driver ID", "Submit Date", "Main Class", 
"Driver Resources")
-val driverHeaders = queuedHeaders ++
+
+val driverHeader = Seq("Driver ID")
--- End diff --

Why is a header a Seq?!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[jaceklaskowski]

Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73032355
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -28,10 +28,17 @@ import 
org.apache.spark.scheduler.cluster.mesos.MesosClusterSubmissionState
 import org.apache.spark.ui.{UIUtils, WebUIPage}
 
 private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends 
WebUIPage("") {
+  private val historyServerURL = 
parent.conf.getOption("spark.mesos.dispatcher.historyServer.url")
+
   def render(request: HttpServletRequest): Seq[Node] = {
 val state = parent.scheduler.getSchedulerState()
-val queuedHeaders = Seq("Driver ID", "Submit Date", "Main Class", 
"Driver Resources")
-val driverHeaders = queuedHeaders ++
+
+val driverHeader = Seq("Driver ID")
+val historyHeader = if (historyServerURL.isDefined) Seq("History") 
else Nil
--- End diff --

`map.getOrElse`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[jaceklaskowski]

Github user jaceklaskowski commented on a diff in the pull request:

https://github.com/apache/spark/pull/14414#discussion_r73032190
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -28,10 +28,17 @@ import 
org.apache.spark.scheduler.cluster.mesos.MesosClusterSubmissionState
 import org.apache.spark.ui.{UIUtils, WebUIPage}
 
 private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends 
WebUIPage("") {
+  private val historyServerURL = 
parent.conf.getOption("spark.mesos.dispatcher.historyServer.url")
--- End diff --

Would it be possible to define a constant for 
"spark.mesos.dispatcher.historyServer.url" (as is in SQL and YARN modules)? See 
https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14414: [SPARK-16809] enable history server links in disp...

[mgummelt]

GitHub user mgummelt opened a pull request:

https://github.com/apache/spark/pull/14414

[SPARK-16809] enable history server links in dispatcher UI

## What changes were proposed in this pull request?

Links the Spark Mesos Dispatcher UI to the history server UI

- adds spark.mesos.dispatcher.historyServer.url
- explicitly generates frameworkIDs for the launched drivers, so the 
dispatcher knows how to correlate drivers and frameworkIDs 

## How was this patch tested?

manual testing



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mesosphere/spark history-server

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14414.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14414


commit 3c7f5aac18d908c90007bae888a22be0c1e00917
Author: Sergiusz Urbaniak 
Date:   2016-03-03T08:42:55Z

enable history server links in dispatcher UI




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org