I've had some odd behavior with jobs showing up in the history server in
1.0.0. Failed jobs do show up but it seems they can show up minutes or
hours later. I see in the history server logs messages about bad task ids.
But then eventually the jobs show up.
This might be your situation.
Anecdotally, if you click on the job in the Spark Master GUI after it is
done, this may help it show up in the history server faster. Haven't
reliably tested this though. May just be a coincidence of timing.
-Suren
On Wed, Jul 2, 2014 at 8:01 PM, Andrew Lee <alee...@hotmail.com> wrote:
> Hi All,
>
> I have HistoryServer up and running, and it is great.
>
> Is it possible to also enable HsitoryServer to parse failed jobs event by
> default as well?
>
> I get "No Completed Applications Found" if job fails.
>
> -
> -
> *========= Event Log Location: *hdfs:///user/test01/spark/logs/
>
> No Completed Applications Found
> =========
>
> The reason is that it is good to run the HistoryServer to keep track of
> performance and resource usage for each completed job,
> but I found it more useful when job fails. I can identify which stage did
> it fail, etc instead of sipping through the logs
> from the Resource Manager. The same event log is only available when the
> Application Master is still active, once the job fails,
> the Application Master is killed, and I lose the GUI access, even though I
> have the event log in JSON format, I can't open it with
> the HistoryServer.
>
> This is very helpful especially for long running jobs that last for 2-18
> hours that generates Gigabytes of logs.
>
> So I have 2 questions:
>
> 1. Any reason why we only render completed jobs? Why can't we bring in all
> jobs and choose from the GUI? Like a time machine to restore the status
> from the Application Master?
>
> ./core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala
>
> val logInfos = logDirs
>
> .sortBy { dir => getModificationTime(dir) }
>
> .map { dir => (dir,
> EventLoggingListener.parseLoggingInfo(dir.getPath, fileSystem)) }
>
> .filter { case (dir, info) => info.*applicationComplete* }
>
>
>
> 2. If I force to touch a file "APPLICATION_COMPLETE" in the failed job
> event log folder, will this cause any problem?
>
>
>
>
>
>
--
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v <suren.hira...@sociocast.com>elos.io
W: www.velos.io
