Hello Team,
Thank you for your time in advance.
Below are the log lines of my spark job which is used for consuming the
messages from Kafka Instance and its loading to Hbase. I have noticed the
below Warn lines and later it resulted to errors. But I noticed that,
exactly after 7 days the token is getting expired and its trying to renew
the token but its not able to even after retrying it. Mine is a Kerberos
cluster. Can you please look into it and guide me whats the issue.
Your time and suggestions are very valuable.
15/12/29 11:33:50 INFO scheduler.JobScheduler: Finished job streaming job
1451410430000 ms.0 from job set of time 1451410430000 ms
15/12/29 11:33:50 INFO scheduler.JobScheduler: Starting job streaming job
1451410430000 ms.1 from job set of time 1451410430000 ms
15/12/29 11:33:50 INFO scheduler.JobScheduler: Finished job streaming job
1451410430000 ms.1 from job set of time 1451410430000 ms
15/12/29 11:33:50 INFO rdd.BlockRDD: Removing RDD 120956 from persistence
list
15/12/29 11:33:50 INFO scheduler.JobScheduler: Total delay: 0.003 s for
time 1451410430000 ms (execution: 0.000 s)
15/12/29 11:33:50 INFO storage.BlockManager: Removing RDD 120956
15/12/29 11:33:50 INFO kafka.KafkaInputDStream: Removing blocks of RDD
BlockRDD[120956] at createStream at SparkStreamingEngine.java:40 of time
1451410430000 ms
15/12/29 11:33:50 INFO rdd.MapPartitionsRDD: Removing RDD 120957 from
persistence list
15/12/29 11:33:50 INFO storage.BlockManager: Removing RDD 120957
15/12/29 11:33:50 INFO scheduler.ReceivedBlockTracker: Deleting batches
ArrayBuffer(1451410410000 ms)
15/12/29 11:34:00 INFO scheduler.JobScheduler: Added jobs for time
1451410440000 ms
15/12/29 11:34:00 INFO scheduler.JobScheduler: Starting job streaming job
1451410440000 ms.0 from job set of time 1451410440000 ms
15/12/29 11:34:00 INFO scheduler.JobScheduler: Finished job streaming job
1451410440000 ms.0 from job set of time 1451410440000 ms
15/12/29 11:34:00 INFO scheduler.JobScheduler: Starting job streaming job
1451410440000 ms.1 from job set of time 1451410440000 ms
15/12/29 11:34:00 INFO scheduler.JobScheduler: Finished job streaming job
1451410440000 ms.1 from job set of time 1451410440000 ms
15/12/29 11:34:00 INFO rdd.BlockRDD: Removing RDD 120958 from persistence
list
15/12/29 11:34:00 INFO scheduler.JobScheduler: Total delay: 0.003 s for
time 1451410440000 ms (execution: 0.001 s)
15/12/29 11:34:00 INFO storage.BlockManager: Removing RDD 120958
15/12/29 11:34:00 INFO kafka.KafkaInputDStream: Removing blocks of RDD
BlockRDD[120958] at createStream at SparkStreamingEngine.java:40 of time
1451410440000 ms
15/12/29 11:34:00 INFO rdd.MapPartitionsRDD: Removing RDD 120959 from
persistence list
15/12/29 11:34:00 INFO storage.BlockManager: Removing RDD 120959
15/12/29 11:34:00 INFO scheduler.ReceivedBlockTracker: Deleting batches
ArrayBuffer(1451410420000 ms)
15/12/29 11:34:10 INFO scheduler.JobScheduler: Added jobs for time
1451410450000 ms
15/12/29 11:34:10 INFO scheduler.JobScheduler: Starting job streaming job
1451410450000 ms.0 from job set of time 1451410450000 ms
15/12/29 11:34:10 INFO scheduler.JobScheduler: Finished job streaming job
1451410450000 ms.0 from job set of time 1451410450000 ms
15/12/29 11:34:10 INFO scheduler.JobScheduler: Starting job streaming job
1451410450000 ms.1 from job set of time 1451410450000 ms
15/12/29 11:34:10 INFO scheduler.JobScheduler: Finished job streaming job
1451410450000 ms.1 from job set of time 1451410450000 ms
15/12/29 11:34:10 INFO rdd.BlockRDD: Removing RDD 120960 from persistence
list
15/12/29 11:34:10 INFO scheduler.JobScheduler: Total delay: 0.004 s for
time 1451410450000 ms (execution: 0.001 s)
15/12/29 11:34:10 INFO storage.BlockManager: Removing RDD 120960
15/12/29 11:34:10 INFO kafka.KafkaInputDStream: Removing blocks of RDD
BlockRDD[120960] at createStream at SparkStreamingEngine.java:40 of time
1451410450000 ms
15/12/29 11:34:10 INFO rdd.MapPartitionsRDD: Removing RDD 120961 from
persistence list
15/12/29 11:34:10 INFO storage.BlockManager: Removing RDD 120961
15/12/29 11:34:10 INFO scheduler.ReceivedBlockTracker: Deleting batches
ArrayBuffer(1451410430000 ms)
15/12/29 11:34:13 WARN security.UserGroupInformation:
PriviledgedActionException as:sssssllllllll (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired
15/12/29 11:34:13 *WARN ipc.Client: Exception encountered while connecting
to the server* :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired
15/12/29 11:34:13 *WARN security.UserGroupInformation:
PriviledgedActionException as:sssssllllllll (auth:SIMPLE) *
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):*
token (HDFS_DELEGATION_TOKEN token 3104414 <3104414> for sssssllllllll) is
expired*
15/12/29 11:34:13 *WARN hdfs.LeaseRenewer: Failed to renew lease for
[DFSClient_NONMAPREDUCE_1297494905_1] for 30 seconds. Will retry shortly
...*
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy14.renewLease(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:571)
at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy15.renewLease(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:878)
at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442)
at
org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
at java.lang.Thread.run(Thread.java:745)
15/12/29 11:34:14* WARN security.UserGroupInformation:
PriviledgedActionException as:sssssllllllll (auth:SIMPLE)
*cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired
15/12/29 11:34:14 WARN ipc.Client: Exception encountered while connecting
to the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired
15/12/29 11:34:14 WARN security.UserGroupInformation:
PriviledgedActionException as:sssssllllllll (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired
15/12/29 11:34:14 WARN hdfs.LeaseRenewer: Failed to renew lease for
[DFSClient_NONMAPREDUCE_1297494905_1] for 31 seconds. Will retry shortly
...
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy14.renewLease(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:571)
at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy15.renewLease(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:878)
at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442)
at
org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
at java.lang.Thread.run(Thread.java:745)
15/12/29 11:34:15 WARN security.UserGroupInformation:
PriviledgedActionException as:sssssllllllll (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired
15/12/29 11:34:15 WARN ipc.Client: Exception encountered while connecting
to the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired
15/12/29 11:34:15 WARN security.UserGroupInformation:
PriviledgedActionException as:sssssllllllll (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired
15/12/29 11:34:15 WARN hdfs.LeaseRenewer: Failed to renew lease for
[DFSClient_NONMAPREDUCE_1297494905_1] for 32 seconds. Will retry shortly
...
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
token (HDFS_DELEGATION_TOKEN token 3104414 for sssssllllllll) is expired
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy14.renewLease(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:571)
at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy15.renewLease(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:878)
at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
*ERROR LINES:*
16/01/05 13:14:10 INFO scheduler.ReceivedBlockTracker: Deleting
batches ArrayBuffer(1452021230000 ms)*16/01/05 13:14:10 ERROR
scheduler.LiveListenerBus: Listener EventLoggingListener threw an
exception*
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
at
org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144)
at
org.apache.spark.scheduler.EventLoggingListener.onUnpersistRDD(EventLoggingListener.scala:175)
at
org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:50)
at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
at
org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53)
at
org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)*Caused
by: java.io.IOException: Lease timeout of 0 seconds expired.*
at
org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:2192)
at
org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:935)
at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:889)
at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442)
at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
at java.lang.Thread.run(Thread.java:745)
Please Guide me.
Regards,
Nik.
