Re: Enabling Event Log
is it working now? On 1 May 2015 at 13:43, James King wrote: > Oops! well spotted. Many thanks Shixiong. > > On Fri, May 1, 2015 at 1:25 AM, Shixiong Zhu wrote: > >> "spark.history.fs.logDirectory" is for the history server. For Spark >> applications, they should use "spark.eventLog.dir". Since you commented out >> "spark.eventLog.dir", it will be "/tmp/spark-events". And this folder does >> not exits. >> >> Best Regards, >> Shixiong Zhu >> >> 2015-04-29 23:22 GMT-07:00 James King : >> >> I'm unclear why I'm getting this exception. >>> >>> It seems to have realized that I want to enable Event Logging but >>> ignoring where I want it to log to i.e. file:/opt/cb/tmp/spark-events which >>> does exist. >>> >>> spark-default.conf >>> >>> # Example: >>> spark.master spark://master1:7077,master2:7077 >>> spark.eventLog.enabled true >>> spark.history.fs.logDirectoryfile:/opt/cb/tmp/spark-events >>> # spark.eventLog.dir hdfs://namenode:8021/directory >>> # spark.serializer >>> org.apache.spark.serializer.KryoSerializer >>> # spark.driver.memory 5g >>> # spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value >>> -Dnumbers="one two three" >>> >>> Exception following job submission: >>> >>> spark.eventLog.enabled=true >>> spark.history.fs.logDirectory=file:/opt/cb/tmp/spark-events >>> >>> spark.jars=file:/opt/cb/scripts/spark-streamer/cb-spark-streamer-1.0-SNAPSHOT.jar >>> spark.master=spark://master1:7077,master2:7077 >>> Exception in thread "main" java.lang.IllegalArgumentException: Log >>> directory /tmp/spark-events does not exist. >>> at >>> org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:99) >>> at org.apache.spark.SparkContext.(SparkContext.scala:399) >>> at >>> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:642) >>> at >>> org.apache.spark.streaming.StreamingContext.(StreamingContext.scala:75) >>> at >>> org.apache.spark.streaming.api.java.JavaStreamingContext.(JavaStreamingContext.scala:132) >>> >>> >>> Many Thanks >>> jk >>> >> >> >
Re: Enabling Event Log
Oops! well spotted. Many thanks Shixiong. On Fri, May 1, 2015 at 1:25 AM, Shixiong Zhu wrote: > "spark.history.fs.logDirectory" is for the history server. For Spark > applications, they should use "spark.eventLog.dir". Since you commented out > "spark.eventLog.dir", it will be "/tmp/spark-events". And this folder does > not exits. > > Best Regards, > Shixiong Zhu > > 2015-04-29 23:22 GMT-07:00 James King : > > I'm unclear why I'm getting this exception. >> >> It seems to have realized that I want to enable Event Logging but >> ignoring where I want it to log to i.e. file:/opt/cb/tmp/spark-events which >> does exist. >> >> spark-default.conf >> >> # Example: >> spark.master spark://master1:7077,master2:7077 >> spark.eventLog.enabled true >> spark.history.fs.logDirectoryfile:/opt/cb/tmp/spark-events >> # spark.eventLog.dir hdfs://namenode:8021/directory >> # spark.serializer >> org.apache.spark.serializer.KryoSerializer >> # spark.driver.memory 5g >> # spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value >> -Dnumbers="one two three" >> >> Exception following job submission: >> >> spark.eventLog.enabled=true >> spark.history.fs.logDirectory=file:/opt/cb/tmp/spark-events >> >> spark.jars=file:/opt/cb/scripts/spark-streamer/cb-spark-streamer-1.0-SNAPSHOT.jar >> spark.master=spark://master1:7077,master2:7077 >> Exception in thread "main" java.lang.IllegalArgumentException: Log >> directory /tmp/spark-events does not exist. >> at >> org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:99) >> at org.apache.spark.SparkContext.(SparkContext.scala:399) >> at >> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:642) >> at >> org.apache.spark.streaming.StreamingContext.(StreamingContext.scala:75) >> at >> org.apache.spark.streaming.api.java.JavaStreamingContext.(JavaStreamingContext.scala:132) >> >> >> Many Thanks >> jk >> > >
Re: Enabling Event Log
"spark.history.fs.logDirectory" is for the history server. For Spark applications, they should use "spark.eventLog.dir". Since you commented out "spark.eventLog.dir", it will be "/tmp/spark-events". And this folder does not exits. Best Regards, Shixiong Zhu 2015-04-29 23:22 GMT-07:00 James King : > I'm unclear why I'm getting this exception. > > It seems to have realized that I want to enable Event Logging but > ignoring where I want it to log to i.e. file:/opt/cb/tmp/spark-events which > does exist. > > spark-default.conf > > # Example: > spark.master spark://master1:7077,master2:7077 > spark.eventLog.enabled true > spark.history.fs.logDirectoryfile:/opt/cb/tmp/spark-events > # spark.eventLog.dir hdfs://namenode:8021/directory > # spark.serializer > org.apache.spark.serializer.KryoSerializer > # spark.driver.memory 5g > # spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value > -Dnumbers="one two three" > > Exception following job submission: > > spark.eventLog.enabled=true > spark.history.fs.logDirectory=file:/opt/cb/tmp/spark-events > > spark.jars=file:/opt/cb/scripts/spark-streamer/cb-spark-streamer-1.0-SNAPSHOT.jar > spark.master=spark://master1:7077,master2:7077 > Exception in thread "main" java.lang.IllegalArgumentException: Log > directory /tmp/spark-events does not exist. > at > org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:99) > at org.apache.spark.SparkContext.(SparkContext.scala:399) > at > org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:642) > at > org.apache.spark.streaming.StreamingContext.(StreamingContext.scala:75) > at > org.apache.spark.streaming.api.java.JavaStreamingContext.(JavaStreamingContext.scala:132) > > > Many Thanks > jk >
Enabling Event Log
I'm unclear why I'm getting this exception. It seems to have realized that I want to enable Event Logging but ignoring where I want it to log to i.e. file:/opt/cb/tmp/spark-events which does exist. spark-default.conf # Example: spark.master spark://master1:7077,master2:7077 spark.eventLog.enabled true spark.history.fs.logDirectoryfile:/opt/cb/tmp/spark-events # spark.eventLog.dir hdfs://namenode:8021/directory # spark.serializer org.apache.spark.serializer.KryoSerializer # spark.driver.memory 5g # spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three" Exception following job submission: spark.eventLog.enabled=true spark.history.fs.logDirectory=file:/opt/cb/tmp/spark-events spark.jars=file:/opt/cb/scripts/spark-streamer/cb-spark-streamer-1.0-SNAPSHOT.jar spark.master=spark://master1:7077,master2:7077 Exception in thread "main" java.lang.IllegalArgumentException: Log directory /tmp/spark-events does not exist. at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:99) at org.apache.spark.SparkContext.(SparkContext.scala:399) at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:642) at org.apache.spark.streaming.StreamingContext.(StreamingContext.scala:75) at org.apache.spark.streaming.api.java.JavaStreamingContext.(JavaStreamingContext.scala:132) Many Thanks jk
