Are you running it on YARN? Might be a known YARN credential expiring issue.
Hari, any insights? On Thu, Jul 30, 2015 at 4:04 AM, Guillermo Ortiz <konstt2...@gmail.com> wrote: > I'm executing a job with Spark Streaming and got this error all times when > the job has been executing for a while (usually hours of days). > > I have no idea why it's happening. > > 15/07/30 13:02:14 ERROR LiveListenerBus: Listener EventLoggingListener > threw an exception > java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) > at > org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) > at scala.Option.foreach(Option.scala:236) > at > org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144) > at > org.apache.spark.scheduler.EventLoggingListener.onJobEnd(EventLoggingListener.scala:169) > at > org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:36) > at > org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) > at > org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) > at > org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53) > at > org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36) > at > org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76) > at > org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61) > at > org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61) > at > org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617) > at > org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60) > Caused by: java.io.IOException: Lease timeout of 0 seconds expired. > at > org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:2192) > at > org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:935) > at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:889) > at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) > at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) > at > org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) > at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) > at java.lang.Thread.run(Thread.java:745) > 15/07/30 13:02:14 INFO SparkContext: Starting job: foreachRDD at > MetricsSpark.scala:67 > 15/07/30 13:02:14 INFO DAGScheduler: Got job 5050 (foreachRDD at > MetricsSpark.scala:67) with 3 output partitions (allowLocal=false) > 15/07/30 13:02:14 INFO DAGScheduler: Final stage: Stage 5050(foreachRDD at > MetricsSpark.scala:67) > > Sometimes this error happens, but it doesn't mean that Spark stops working > forever. Because it looks like after this error Spark works correctly some > iterations, but most of time just fails after this error producing this > error all the time. > > The code just filters some records and index the record in > ElasticSearch.after adding some new fields. >
