Probably your application has crashed or was terminated without invoking the stop method of spark context - in such cases it doesn't create the empty flag file which apparently tells the history server that it can safely show the log data - simpy go to some of the other dirs of the history server to see what the name of the flag file was and then create it manually in the dirs of the missing apps - then they will appear in the history server ui
From: Steve Loughran [mailto:ste...@hortonworks.com] Sent: Monday, June 22, 2015 7:22 PM To: Jonathon Cai Cc: user@spark.apache.org Subject: Re: Web UI vs History Server Bugs well, I'm afraid you've reached the limits of my knowledge ... hopefully someone else can answer On 22 Jun 2015, at 16:37, Jonathon Cai <jonathon....@yale.edu> wrote: No, what I'm seeing is that while the cluster is running, I can't see the app info after the app is completed. That is to say, when I click on the application name on master:8080, no info is shown. However, when I examine the same file on the History Server, the application information opens fine. On Sat, Jun 20, 2015 at 6:47 AM, Steve Loughran <ste...@hortonworks.com> wrote: > On 17 Jun 2015, at 19:10, jcai <jonathon....@yale.edu> wrote: > > Hi, > > I am running this on Spark stand-alone mode. I find that when I examine the > web UI, a couple bugs arise: > > 1. There is a discrepancy between the number denoting the duration of the > application when I run the history server and the number given by the web UI > (default address is master:8080). I checked more specific details, including > task and stage durations (when clicking on the application), and these > appear to be the same for both avenues. > > 2. Sometimes the web UI on master:8080 is unable to display more specific > information for an application that has finished (when clicking on the > application), even when there is a log file in the appropriate directory. > But when the history server is opened, it is able to read this file and > output information. > There's a JIRA open on the history server caching incomplete work...if you click on the link to a job while it's in progress, you don't get any updates later. does this sound like what you are seeing?
