My primary concern was having too many watches for history. But I do not have an argument against stand-alone mode.
Also questions about automatic lifecycle management of history server - when to start / stop etc. in a cluster environment would be difficult when one is not the cluster manager. I agree that JSON files are not easily readable. Maybe a stand-alone history server which makes it implicit that it is only for that host would be fine. On Mon, Jan 13, 2014 at 1:32 PM, Patrick Wendell <[email protected]> wrote: > Pillis - I agree we need to decouple the representation from a > particular history server. But why not provide as simple history > server people can (optionally) run if they aren't using Yarn or Mesos? > For people running the standalone cluster scheduler this seems > important. Giving them only a JSON dump isn't super consumable for > most users. > > - Patrick > > On Mon, Jan 13, 2014 at 10:43 AM, Pillis Work <[email protected]> > wrote: > > The listeners in SparkUI which update the counters can trigger saves > along > > the way. > > The save can be on a 500ms delay after the last update, to batch changes. > > This solution would not require save on stop(). > > > > > > > > On Mon, Jan 13, 2014 at 6:15 AM, Tom Graves <[email protected]> > wrote: > > > >> So the downside to just saving stuff at the end is that if the app > crashes > >> or exits badly you don't have anything. Hadoop has taken the approach > of > >> saving events along the way. But Hadoop also uses that history file to > >> start where it left off at if something bad happens and it gets > restarted. > >> I don't think the latter really applies to spark though. > >> > >> Does mesos have a history server? > >> > >> Tom > >> > >> > >> > >> On Sunday, January 12, 2014 9:22 PM, Pillis Work <[email protected] > > > >> wrote: > >> > >> IMHO from a pure Spark standpoint, I don't know if having a dedicated > >> history service makes sense as of now - considering that cluster > managers > >> have their own history servers. Just showing UI of history runs might be > >> too thin a requirement for a full service. Spark should store history > >> information that can later be exposed in required ways. > >> > >> Since each SparkContext is the logical entry and exit point for doing > >> something useful in Spark, during its stop(), it should serialize that > >> run's statistics into a JSON file - like > "sc_run_[name]_[start-time].json". > >> When SparkUI.stop() is called, it in turn asks its UI objects (which > should > >> implement a trait) to provide either a flat or hierarchical Map of > String > >> key/value pairs. This map (flat, hierarchical) is then serialized to a > >> configured path (default being "var/history"). > >> > >> With regards to Mesos or YARN, their applications during shutdown can > have > >> API to import this Spark history into their history servers - by making > API > >> calls etc. > >> > >> This way Spark's history information is persisted independent of cluster > >> framework, and cluster frameworks can import the history when/as needed. > >> Hope this helps. > >> Regards, > >> pillis > >> > >> > >> > >> On Thu, Jan 9, 2014 at 6:13 AM, Tom Graves <[email protected]> > wrote: > >> > >> > Note that it looks like we are planning on adding support for > application > >> > specific frameworks to YARN sooner rather then later. There is an > initial > >> > design up here: https://issues.apache.org/jira/browse/YARN-1530. Note > >> > this has not been reviewed yet so changes are likely but gives an > idea of > >> > the general direction. If anyone has comments on how that might work > >> with > >> > SPARK I encourage you to post to the jira. > >> > > >> > As Sandy mentioned it would be very nice if the solution could be > >> > compatible with that. > >> > > >> > Tom > >> > > >> > > >> > > >> > On Wednesday, January 8, 2014 12:44 AM, Sandy Ryza < > >> > [email protected]> wrote: > >> > > >> > Hey, > >> > > >> > YARN-321 is targeted for the Hadoop 2.4. The minimum feature set > doesn't > >> > include application-specific data, so that probably won't be part of > 2.4 > >> > unless other things delay the release for a while. There are no APIs > for > >> > it yet and pluggable UIs have been discussed but not agreed upon. I > >> think > >> > requirements from Spark could be useful in helping shape what gets > done > >> > there. > >> > > >> > -Sandy > >> > > >> > > >> > > >> > On Tue, Jan 7, 2014 at 4:13 PM, Patrick Wendell <[email protected]> > >> > wrote: > >> > > >> > > Hey Sandy, > >> > > > >> > > Do you know what the status is for YARN-321 and what version of YARN > >> > > it's targeted for? Also, is there any kind of documentation or API > for > >> > > this? Does it control the presentation of the data itself (e.g. it > >> > > actually has its own UI)? > >> > > > >> > > @Tom - having an optional history server sounds like a good idea. > >> > > > >> > > One question is what format to use for storing the data and how the > >> > > persisted format relates to XML/HTML generation in the live UI. One > >> > > idea would be to add JSON as an intermediate format inside of the > >> > > current WebUI, and then any JSON page could be persisted and > rendered > >> > > by the history server using the same code. Once a SparkContext exits > >> > > it could dump a series of named paths each with a JSON file. Then > the > >> > > history server could load those paths and pass them through the > second > >> > > rendering stage (JSON => XML) to create each page. > >> > > > >> > > It would be good if SPARK-969 had a good design doc before anyone > >> > > starts working on it. > >> > > > >> > > - Patrick > >> > > > >> > > On Tue, Jan 7, 2014 at 3:18 PM, Sandy Ryza <[email protected] > > > >> > > wrote: > >> > > > As a sidenote, it would be nice to make sure that whatever done > here > >> > will > >> > > > work with the YARN Application History Server (YARN-321), a > generic > >> > > history > >> > > > server that functions similarly to MapReduce's JobHistoryServer. > It > >> > will > >> > > > eventually have the ability to store application-specific data. > >> > > > > >> > > > -Sandy > >> > > > > >> > > > > >> > > > On Tue, Jan 7, 2014 at 2:51 PM, Tom Graves <[email protected]> > >> > wrote: > >> > > > > >> > > >> I don't think you want to save the html/xml files. I would rather > >> see > >> > > the > >> > > >> info saved into a history file in like a json format that could > then > >> > be > >> > > >> re-read and the web ui display the info, hopefully without much > >> change > >> > > to > >> > > >> the UI parts. For instance perhaps the history server could read > >> the > >> > > file > >> > > >> and populate the appropriate Spark data structures that the web > ui > >> > > already > >> > > >> uses. > >> > > >> > >> > > >> I would suggest making it so the history server is an optional > >> server > >> > > and > >> > > >> could be run on any node. That way if the load on a particular > node > >> > > becomes > >> > > >> to much it could be moved, but you also could run it on the same > >> node > >> > as > >> > > >> the Master. All it really needs to know is where to get the > history > >> > > files > >> > > >> from and have access to that location. > >> > > >> > >> > > >> Hadoop actually has a history server for MapReduce which works > very > >> > > >> similar to what I mention above. One thing to keep in minds > here > >> is > >> > > >> security. You want to make sure that the history files can only > be > >> > > read by > >> > > >> users who have the appropriate permissions. The history server > >> itself > >> > > >> could run as a superuser who has permission to server up the > files > >> > > based > >> > > >> on the acls. > >> > > >> > >> > > >> > >> > > >> > >> > > >> On Tuesday, January 7, 2014 8:06 AM, "Xia, Junluan" < > >> > > [email protected]> > >> > > >> wrote: > >> > > >> > >> > > >> Hi all > >> > > >> Spark job web ui will not be available when job is over, > >> but > >> > it > >> > > >> is convenient for developer to debug with persisting job web ui. > I > >> > just > >> > > >> come up with draft for this issue. > >> > > >> > >> > > >> 1. We could simply save the web page with html/xml > >> > > >> format(stages/executors/storages/environment) to certain location > >> when > >> > > job > >> > > >> finished > >> > > >> > >> > > >> 2. But it is not easy for user to review the job info with > #1, > >> > we > >> > > >> could build extra job history service for developers > >> > > >> > >> > > >> 3. But where will we build this history service? In Driver > >> node > >> > or > >> > > >> Master node? > >> > > >> > >> > > >> Any suggestions about this improvement? > >> > > >> > >> > > >> regards, > >> > > >> Andrew > >> > > >> > >> > > > >> > > >> >
