Hello:
The following error occurred while running the shell task using Oozie:
W] ACTION[0000001-200717100610211-oozie-root-W@shell-node] Adding
default credentials for action: hdfs, yarn and jhs
2020-07-17 10:12:58,520 INFO ShellActionExecutor:520 - SERVER[master]
USER[oozie] GROUP[-] TOKEN[] APP[shell-wf]
JOB[0000001-200717100610211-oozie-root-W]
ACTION[0000001-200717100610211-oozie-root-W@shell-node] Adding default
credentials for action: hdfs, yarn and jhs
2020-07-17 10:12:58,522 INFO deprecation:1395 - SERVER[master]
fs.default.name is deprecated. Instead, use fs.defaultFS
2020-07-17 10:12:58,522 INFO deprecation:1395 - SERVER[master]
fs.default.name is deprecated. Instead, use fs.defaultFS
2020-07-17 10:12:58,523 WARN HadoopTokenHelper:523 - SERVER[master]
USER[oozie] GROUP[-] TOKEN[] APP[shell-wf]
JOB[0000001-200717100610211-oozie-root-W]
ACTION[0000001-200717100610211-oozie-root-W@shell-node] An error happened while
trying to get server principal. Getting it from service principal anyway.
2020-07-17 10:12:58,523 WARN HadoopTokenHelper:523 - SERVER[master]
USER[oozie] GROUP[-] TOKEN[] APP[shell-wf]
JOB[0000001-200717100610211-oozie-root-W]
ACTION[0000001-200717100610211-oozie-root-W@shell-node] An error happened while
trying to get server principal. Getting it from service principal anyway.
2020-07-17 10:12:58,523 INFO HadoopTokenHelper:520 - SERVER[master]
USER[oozie] GROUP[-] TOKEN[] APP[shell-wf]
JOB[0000001-200717100610211-oozie-root-W]
ACTION[0000001-200717100610211-oozie-root-W@shell-node] Delegation Token
Renewer for yarncluster is yarn
2020-07-17 10:12:58,523 INFO HadoopTokenHelper:520 - SERVER[master]
USER[oozie] GROUP[-] TOKEN[] APP[shell-wf]
JOB[0000001-200717100610211-oozie-root-W]
ACTION[0000001-200717100610211-oozie-root-W@shell-node] Delegation Token
Renewer for yarncluster is yarn
2020-07-17 10:12:58,525 WARN Client:765 - SERVER[master] Exception
encountered while connecting to the server :
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
2020-07-17 10:12:58,525 WARN Client:765 - SERVER[master] Exception
encountered while connecting to the server :
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
2020-07-17 10:12:58,525 WARN ActionStartXCommand:523 - SERVER[master]
USER[oozie] GROUP[-] TOKEN[] APP[shell-wf]
JOB[0000001-200717100610211-oozie-root-W]
ACTION[0000001-200717100610211-oozie-root-W@shell-node] Error starting action
[shell-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009:
DestHost:destPort master:10020 , LocalHost:localPort master/28.11.1.14:0.
Failed on local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]]
org.apache.oozie.action.ActionExecutorException: JA009: DestHost:destPort
master:10020 , LocalHost:localPort master/28.11.1.14:0. Failed on local
exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
at
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463)
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:443)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1134)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1644)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:243)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:68)
at
org.apache.oozie.command.XCommand.call(XCommand.java:290)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:363)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:292)
at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: DestHost:destPort master:10020 ,
LocalHost:localPort master/28.11.1.14:0. Failed on local exception:
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[TOKEN, KERBEROS]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at
org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:808)
at
org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at
org.apache.hadoop.ipc.Client.call(Client.java:1491)
at
org.apache.hadoop.ipc.Client.call(Client.java:1388)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at
com.sun.proxy.$Proxy42.getDelegationToken(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:211)
at
org.apache.oozie.action.hadoop.JHSCredentials.getDelegationTokenFromJHS(JHSCredentials.java:88)
at
org.apache.oozie.action.hadoop.JHSCredentials.updateCredentials(JHSCredentials.java:65)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1546)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1082)
... 11 more
Caused by: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
at
org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:770)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:733)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:827)
at
org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
at
org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
at
org.apache.hadoop.ipc.Client.call(Client.java:1435)
... 20 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
at
org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:627)
at
org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:421)
at
org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:814)
at
org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:810)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:810)
... 23 more
2020-07-17 10:12:58,525 WARN ActionStartXCommand:523 - SERVER[master]
USER[oozie] GROUP[-] TOKEN[] APP[shell-wf]
JOB[0000001-200717100610211-oozie-root-W]
ACTION[0000001-200717100610211-oozie-root-W@shell-node] Error starting action
[shell-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009:
DestHost:destPort master:10020 , LocalHost:localPort master/28.11.1.14:0.
Failed on local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]]
org.apache.oozie.action.ActionExecutorException: JA009: DestHost:destPort
master:10020 , LocalHost:localPort master/28.11.1.14:0. Failed on local
exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
at
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463)
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:443)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1134)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1644)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:243)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:68)
at
org.apache.oozie.command.XCommand.call(XCommand.java:290)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:363)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:292)
at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: DestHost:destPort master:10020 ,
LocalHost:localPort master/28.11.1.14:0. Failed on local exception:
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[TOKEN, KERBEROS]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:833)
at
org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:808)
at
org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at
org.apache.hadoop.ipc.Client.call(Client.java:1491)
at
org.apache.hadoop.ipc.Client.call(Client.java:1388)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at
com.sun.proxy.$Proxy42.getDelegationToken(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getDelegationToken(MRClientProtocolPBClientImpl.java:211)
at
org.apache.oozie.action.hadoop.JHSCredentials.getDelegationTokenFromJHS(JHSCredentials.java:88)
at
org.apache.oozie.action.hadoop.JHSCredentials.updateCredentials(JHSCredentials.java:65)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1546)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1082)
... 11 more
Caused by: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
at
org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:770)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:733)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:827)
at
org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
at
org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
at
org.apache.hadoop.ipc.Client.call(Client.java:1435)
... 20 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
at
org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:627)
at
org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:421)
at
org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:814)
at
org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:810)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:810)
... 23 more
2020-07-17 10:12:58,527 INFO ActionStartXCommand:520 - SERVER[master]
USER[oozie] GROUP[-] TOKEN[] APP[shell-wf]
JOB[0000001-200717100610211-oozie-root-W]
ACTION[0000001-200717100610211-oozie-root-W@shell-node] Next Retry, Attempt
Number [1] in [10,000] milliseconds
2020-07-17 10:12:58,527 INFO ActionStartXCommand:520 - SERVER[master]
USER[oozie] GROUP[-] TOKEN[] APP[shell-wf]
JOB[0000001-200717100610211-oozie-root-W]
ACTION[0000001-200717100610211-oozie-root-W@shell-node] Next Retry, Attempt
Number [1] in [10,000] milliseconds
The cluster environment information is as follows:
oozie version is 5.2.0
hadoop version is 3.2.1
the authentication mode is kerberos.
This problem also occurs when running other types of tasks,
such as MapReduce.
Can anyone tell me other ways to solve this problem?
thinks.
I am looking forward to your reply??
