These are my versions cdh version = 5.4.1 spark version, 1.3.0 kafka = KAFKA-0.8.2.0-1.kafka1.3.1.p0.9 hbase versions = 1.0.0
Regards, Nik. On Wed, Jan 6, 2016 at 3:50 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Which Spark / hadoop release are you using ? > > Thanks > > On Wed, Jan 6, 2016 at 12:16 PM, Nikhil Gs <gsnikhil1432...@gmail.com> > wrote: > >> Hello Team, >> >> >> Thank you for your time in advance. >> >> >> Below are the log lines of my spark job which is used for consuming the >> messages from Kafka Instance and its loading to Hbase. I have noticed the >> below Warn lines and later it resulted to errors. But I noticed that, >> exactly after 7 days the token is getting expired and its trying to renew >> the token but its not able to even after retrying it. Mine is a Kerberos >> cluster. Can you please look into it and guide me whats the issue. >> >> >> Your time and suggestions are very valuable. >> >> >> >> 15/12/29 11:33:50 INFO scheduler.JobScheduler: Finished job streaming job >> 1451410430000 ms.0 from job set of time 1451410430000 ms >> >> 15/12/29 11:33:50 INFO scheduler.JobScheduler: Starting job streaming job >> 1451410430000 ms.1 from job set of time 1451410430000 ms >> >> 15/12/29 11:33:50 INFO scheduler.JobScheduler: Finished job streaming job >> 1451410430000 ms.1 from job set of time 1451410430000 ms >> >> 15/12/29 11:33:50 INFO rdd.BlockRDD: Removing RDD 120956 from persistence >> list >> >> 15/12/29 11:33:50 INFO scheduler.JobScheduler: Total delay: 0.003 s for >> time 1451410430000 ms (execution: 0.000 s) >> >> 15/12/29 11:33:50 INFO storage.BlockManager: Removing RDD 120956 >> >> 15/12/29 11:33:50 INFO kafka.KafkaInputDStream: Removing blocks of RDD >> BlockRDD[120956] at createStream at SparkStreamingEngine.java:40 of time >> 1451410430000 ms >> >> 15/12/29 11:33:50 INFO rdd.MapPartitionsRDD: Removing RDD 120957 from >> persistence list >> >> 15/12/29 11:33:50 INFO storage.BlockManager: Removing RDD 120957 >> >> 15/12/29 11:33:50 INFO scheduler.ReceivedBlockTracker: Deleting batches >> ArrayBuffer(1451410410000 ms) >> >> 15/12/29 11:34:00 INFO scheduler.JobScheduler: Added jobs for time >> 1451410440000 ms >> >> 15/12/29 11:34:00 INFO scheduler.JobScheduler: Starting job streaming job >> 1451410440000 ms.0 from job set of time 1451410440000 ms >> >> 15/12/29 11:34:00 INFO scheduler.JobScheduler: Finished job streaming job >> 1451410440000 ms.0 from job set of time 1451410440000 ms >> >> 15/12/29 11:34:00 INFO scheduler.JobScheduler: Starting job streaming job >> 1451410440000 ms.1 from job set of time 1451410440000 ms >> >> 15/12/29 11:34:00 INFO scheduler.JobScheduler: Finished job streaming job >> 1451410440000 ms.1 from job set of time 1451410440000 ms >> >> 15/12/29 11:34:00 INFO rdd.BlockRDD: Removing RDD 120958 from persistence >> list >> >> 15/12/29 11:34:00 INFO scheduler.JobScheduler: Total delay: 0.003 s for >> time 1451410440000 ms (execution: 0.001 s) >> >> 15/12/29 11:34:00 INFO storage.BlockManager: Removing RDD 120958 >> >> 15/12/29 11:34:00 INFO kafka.KafkaInputDStream: Removing blocks of RDD >> BlockRDD[120958] at createStream at SparkStreamingEngine.java:40 of time >> 1451410440000 ms >> >> 15/12/29 11:34:00 INFO rdd.MapPartitionsRDD: Removing RDD 120959 from >> persistence list >> >> 15/12/29 11:34:00 INFO storage.BlockManager: Removing RDD 120959 >> >> 15/12/29 11:34:00 INFO scheduler.ReceivedBlockTracker: Deleting batches >> ArrayBuffer(1451410420000 ms) >> >> 15/12/29 11:34:10 INFO scheduler.JobScheduler: Added jobs for time >> 1451410450000 ms >> >> 15/12/29 11:34:10 INFO scheduler.JobScheduler: Starting job streaming job >> 1451410450000 ms.0 from job set of time 1451410450000 ms >> >> 15/12/29 11:34:10 INFO scheduler.JobScheduler: Finished job streaming job >> 1451410450000 ms.0 from job set of time 1451410450000 ms >> >> 15/12/29 11:34:10 INFO scheduler.JobScheduler: Starting job streaming job >> 1451410450000 ms.1 from job set of time 1451410450000 ms >> >> 15/12/29 11:34:10 INFO scheduler.JobScheduler: Finished job streaming job >> 1451410450000 ms.1 from job set of time 1451410450000 ms >> >> 15/12/29 11:34:10 INFO rdd.BlockRDD: Removing RDD 120960 from persistence >> list >> >> 15/12/29 11:34:10 INFO scheduler.JobScheduler: Total delay: 0.004 s for >> time 1451410450000 ms (execution: 0.001 s) >> >> 15/12/29 11:34:10 INFO storage.BlockManager: Removing RDD 120960 >> >> 15/12/29 11:34:10 INFO kafka.KafkaInputDStream: Removing blocks of RDD >> BlockRDD[120960] at createStream at SparkStreamingEngine.java:40 of time >> 1451410450000 ms >> >> 15/12/29 11:34:10 INFO rdd.MapPartitionsRDD: Removing RDD 120961 from >> persistence list >> >> 15/12/29 11:34:10 INFO storage.BlockManager: Removing RDD 120961 >> >> 15/12/29 11:34:10 INFO scheduler.ReceivedBlockTracker: Deleting batches >> ArrayBuffer(1451410430000 ms) >> >> 15/12/29 11:34:13 WARN security.UserGroupInformation: >> PriviledgedActionException as:sssssllllllll (auth:SIMPLE) >> cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): >> token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired >> >> 15/12/29 11:34:13 *WARN ipc.Client: Exception encountered while >> connecting to the server* : >> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): >> token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired >> >> 15/12/29 11:34:13 *WARN security.UserGroupInformation: >> PriviledgedActionException as:sssssllllllll (auth:SIMPLE) * >> cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):* >> token (HDFS_DELEGATION_TOKEN token 3104414 <3104414> for sssssllllllll) is >> expired* >> >> 15/12/29 11:34:13 *WARN hdfs.LeaseRenewer: Failed to renew lease for >> [DFSClient_NONMAPREDUCE_1297494905_1] for 30 seconds. Will retry shortly >> ...* >> >> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): >> token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1468) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1399) >> >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) >> >> at com.sun.proxy.$Proxy14.renewLease(Unknown Source) >> >> at >> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:571) >> >> at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:606) >> >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) >> >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) >> >> at com.sun.proxy.$Proxy15.renewLease(Unknown Source) >> >> at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:878) >> >> at >> org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) >> >> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) >> >> at >> org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) >> >> at >> org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) >> >> at java.lang.Thread.run(Thread.java:745) >> >> 15/12/29 11:34:14* WARN security.UserGroupInformation: >> PriviledgedActionException as:sssssllllllll (auth:SIMPLE) >> *cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): >> token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired >> >> 15/12/29 11:34:14 WARN ipc.Client: Exception encountered while connecting >> to the server : >> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): >> token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired >> >> 15/12/29 11:34:14 WARN security.UserGroupInformation: >> PriviledgedActionException as:sssssllllllll (auth:SIMPLE) >> cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): >> token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired >> >> 15/12/29 11:34:14 WARN hdfs.LeaseRenewer: Failed to renew lease for >> [DFSClient_NONMAPREDUCE_1297494905_1] for 31 seconds. Will retry shortly >> ... >> >> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): >> token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1468) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1399) >> >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) >> >> at com.sun.proxy.$Proxy14.renewLease(Unknown Source) >> >> at >> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:571) >> >> at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:606) >> >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) >> >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) >> >> at com.sun.proxy.$Proxy15.renewLease(Unknown Source) >> >> at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:878) >> >> at >> org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) >> >> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) >> >> at >> org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) >> >> at >> org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) >> >> at java.lang.Thread.run(Thread.java:745) >> >> 15/12/29 11:34:15 WARN security.UserGroupInformation: >> PriviledgedActionException as:sssssllllllll (auth:SIMPLE) >> cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): >> token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired >> >> 15/12/29 11:34:15 WARN ipc.Client: Exception encountered while connecting >> to the server : >> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): >> token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired >> >> 15/12/29 11:34:15 WARN security.UserGroupInformation: >> PriviledgedActionException as:sssssllllllll (auth:SIMPLE) >> cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): >> token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired >> >> 15/12/29 11:34:15 WARN hdfs.LeaseRenewer: Failed to renew lease for >> [DFSClient_NONMAPREDUCE_1297494905_1] for 32 seconds. Will retry shortly >> ... >> >> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): >> token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1468) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1399) >> >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) >> >> at com.sun.proxy.$Proxy14.renewLease(Unknown Source) >> >> at >> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:571) >> >> at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:606) >> >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) >> >> at >> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) >> >> at com.sun.proxy.$Proxy15.renewLease(Unknown Source) >> >> at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:878) >> >> at >> org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) >> >> >> >> *ERROR LINES:* >> >> >> >> 16/01/05 13:14:10 INFO scheduler.ReceivedBlockTracker: Deleting batches >> ArrayBuffer(1452021230000 ms)*16/01/05 13:14:10 ERROR >> scheduler.LiveListenerBus: Listener EventLoggingListener threw an exception* >> java.lang.reflect.InvocationTargetException >> at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) >> at >> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) >> at scala.Option.foreach(Option.scala:236) >> at >> org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144) >> at >> org.apache.spark.scheduler.EventLoggingListener.onUnpersistRDD(EventLoggingListener.scala:175) >> at >> org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:50) >> at >> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) >> at >> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) >> at >> org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53) >> at >> org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36) >> at >> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76) >> at >> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61) >> at >> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61) >> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617) >> at >> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)*Caused >> by: java.io.IOException: Lease timeout of 0 seconds expired.* >> at >> org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:2192) >> at >> org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:935) >> at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:889) >> at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) >> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) >> at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) >> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) >> at java.lang.Thread.run(Thread.java:745) >> >> >> >> Please Guide me. >> >> >> Regards, >> Nik. >> > >
