Probably relevant to people on this list: on Friday I released a clone of the Spark web UI built using Meteor <https://www.meteor.com/> so that everything updates in real-time, saving you from endlessly refreshing the page while jobs are running :) It can also serve as the UI for running as well as completed applications, so you don't have to mess with the separate history-server process if you don't want to.
*This blog post* <http://www.hammerlab.org/2015/07/25/spree-58-a-live-updating-web-ui-for-spark/> and *the github repo* <https://github.com/hammerlab/spree> have lots of information on how to use it. It has two sub-components, JsonRelay <https://github.com/hammerlab/spark-json-relay> and Slim <https://github.com/hammerlab/slim>; the former sends SparkListenerEvent <https://github.com/apache/spark/blob/v1.4.1/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala#L33>s out of Spark via a network socket, while the latter receives those events and writes various stats about them to Mongo (like an external JobProgressListener <https://github.com/apache/spark/blob/v1.4.1/core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala> that persists info to a database). You might find them to offer a better way of storing information about running and completed Spark applications than the event log files that Spark uses, and they can be used with or without the real-time web UI. Give them a try if they sound useful to you, and let me know if you have questions or comments! -Ryan