Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/42#discussion_r10597261
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -634,6 +652,30 @@ private[spark] class Master(host: String, port: Int,
webUiPort: Int,
}
}
+ /**
+ * Start a new SparkUI rendered from persisted storage. If this is
unsuccessful for any reason,
+ * return None. Otherwise return the reconstructed UI.
+ */
+ def startPersistedSparkUI(app: ApplicationInfo): Option[SparkUI] = {
+ val appName = app.desc.name
+ val eventLogInfo = app.desc.eventLogInfo.getOrElse { return None }
+ val eventLogDir = eventLogInfo.logDir
+ val eventCompressionCodec = eventLogInfo.compressionCodec
+ val appConf = new SparkConf
+ eventCompressionCodec.foreach { codec =>
+ appConf.set("spark.eventLog.compress", "true")
+ appConf.set("spark.io.compression.codec", codec)
+ }
+ val ui = new SparkUI(appConf, "%s (finished)".format(appName),
"/history/%s".format(app.id))
--- End diff --
If you did the other changes suggested then the logic here would be a bit
different. You'd first create a ReplayBus then you'd pass it to the SparkUI
then you'd call the utility function and pass in the logdir to the bus. That
seems a bit more modular to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
