Hi Andrew, thank you very much for your answer. Unfortunately it still doesn't work. I'm using Spark 1.0.0, and I start history server running sbin/start-history-server.sh <dir>, although I also set SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory in conf/spark-env.sh. I tried also other dir than /tmp/spark-events which have all possible permissions enabled. Also adding file: (and file://) didn't help - history server still shows: History Server Event Log Location: file:/tmp/spark-events/ No Completed Applications Found.
Best regards, Grzegorz On Thu, Sep 4, 2014 at 8:20 PM, Andrew Or <and...@databricks.com> wrote: > Hi Grzegorz, > > Sorry for the late response. Unfortunately, if the Master UI doesn't know > about your applications (they are "completed" with respect to a different > Master), then it can't regenerate the UIs even if the logs exist. You will > have to use the history server for that. > > How did you start the history server? If you are using Spark <=1.0, you > can pass the directory as an argument to the sbin/start-history-server.sh > script. Otherwise, you may need to set the following in your > conf/spark-env.sh to specify the log directory: > > export SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory=/tmp/spark-events > > It could also be a permissions thing. Make sure your logs in > /tmp/spark-events are accessible by the JVM that runs the history server. > Also, there's a chance that "/tmp/spark-events" is interpreted as an HDFS > path depending on which Spark version you're running. To resolve any > ambiguity, you may set the log path to "file:/tmp/spark-events" instead. > But first verify whether they actually exist. > > Let me know if you get it working, > -Andrew > > > > 2014-08-19 8:23 GMT-07:00 Grzegorz Białek <grzegorz.bia...@codilime.com>: > > Hi, >> Is there any way view history of applications statistics in master ui >> after restarting master server? I have all logs ing /tmp/spark-events/ but >> when I start history server in this directory it says "No Completed >> Applications Found". Maybe I could copy this logs to dir used by master >> server but I couldn't find any. Or maybe I'm doing something wrong >> launching history server. >> Do you have any idea how to solve it? >> >> Thanks, >> Grzegorz >> >> >> On Thu, Aug 14, 2014 at 10:53 AM, Grzegorz Białek < >> grzegorz.bia...@codilime.com> wrote: >> >>> Hi, >>> >>> Thank you both for your answers. Browsing using Master UI works fine. >>> Unfortunately History Server shows "No Completed Applications Found" even >>> if logs exists under given directory, but using Master UI is enough for me. >>> >>> Best regards, >>> Grzegorz >>> >>> >>> >>> On Wed, Aug 13, 2014 at 8:09 PM, Andrew Or <and...@databricks.com> >>> wrote: >>> >>>> The Spark UI isn't available through the same address; otherwise new >>>> applications won't be able to bind to it. Once the old application >>>> finishes, the standalone Master renders the after-the-fact application UI >>>> and exposes it under a different URL. To see this, go to the Master UI >>>> (<master-url>:8080) and click on your application in the "Completed >>>> Applications" table. >>>> >>>> >>>> 2014-08-13 10:56 GMT-07:00 Matei Zaharia <matei.zaha...@gmail.com>: >>>> >>>> Take a look at http://spark.apache.org/docs/latest/monitoring.html -- >>>>> you need to launch a history server to serve the logs. >>>>> >>>>> Matei >>>>> >>>>> On August 13, 2014 at 2:03:08 AM, grzegorz-bialek ( >>>>> grzegorz.bia...@codilime.com) wrote: >>>>> >>>>> Hi, >>>>> I wanted to access Spark web UI after application stops. I set >>>>> spark.eventLog.enabled to true and logs are availaible >>>>> in JSON format in /tmp/spark-event but web UI isn't available under >>>>> address >>>>> http://<driver-node>:4040 >>>>> I'm running Spark in standalone mode. >>>>> >>>>> What should I do to access web UI after application ends? >>>>> >>>>> Thanks, >>>>> Grzegorz >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Viewing-web-UI-after-fact-tp12023.html >>>>> Sent from the Apache Spark User List mailing list archive at >>>>> Nabble.com. >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>> >>>>> >>>> >>> >> >
