[jira] [Updated] (METRON-1178) kinit + authorize on all nodes running topologies for multi-node metron deployments
[
https://issues.apache.org/jira/browse/METRON-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian updated METRON-1178:
--
Description:
In a 12-node deployment, it is seen post-kerberization that some of the
topologies fail to start with exceptions thrown like one pasted below. This is
because the node does not have the proper credentials, since kinit has not
occurred on this host.
{code}
2017-09-11 16:04:46.923 o.a.h.i.Client [WARN] Exception encountered while
connecting to the server : javax.security.sasl.SaslException: GSS ini
tiate failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Failed to find any Kerberos tgt)]
2017-09-11 16:04:46.930 o.a.m.p.GrokParser [ERROR] Failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initi
ate failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Failed to find any Kerberos tgt)]; Host Details : local hos
t is: "metron-2/xx.xx.xx.xx; destination host is:
"metron-2.openstacklocal":8020;
java.io.IOException: Failed on local exception: java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSExcep
tion: No valid credentials provided (Mechanism level: Failed to find any
Kerberos tgt)]; Host Details : local host is: "nat-r7-lqys-metron-2/17
2.22.104.43"; destination host is: "nat-r7-lqys-metron-2.openstacklocal":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
~[stormjar.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1480) ~[stormjar.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1407) ~[stormjar.jar:?]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
~[stormjar.jar:?]
at com.sun.proxy.$Proxy45.getFileInfo(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
~[stor
mjar.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_141]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_141]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_141]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
~[stormjar.jar:?]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
~[stormjar.jar:?]
at com.sun.proxy.$Proxy46.getFileInfo(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
~[stormjar.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
~[stormjar.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
~[stormjar.jar:?]
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
~[stormjar.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
~[stormjar.jar:?]
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
~[stormjar.jar:?]
at
org.apache.metron.parsers.GrokParser.openInputStream(GrokParser.java:83)
~[stormjar.jar:?]
at org.apache.metron.parsers.GrokParser.init(GrokParser.java:94)
[stormjar.jar:?]
at
org.apache.metron.parsers.bolt.ParserBolt.prepare(ParserBolt.java:108)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__6573$fn__6586.invoke(executor.clj:798)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:482)
[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate
failed [Caused by GSSException: No valid credentials provided (
Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:682)
~[stormjar.jar:?]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_141]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_141]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
~[stormjar.jar:?]
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:645)
~[stormjar.jar:?]
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:732)
~[stormjar.jar:?]
at
[jira] [Updated] (METRON-1178) kinit + authorize on all nodes running topologies for multi-node metron deployments
[
https://issues.apache.org/jira/browse/METRON-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian updated METRON-1178:
--
Summary: kinit + authorize on all nodes running topologies for multi-node
metron deployments (was: kinit on all nodes running topologies for multi-node
metron deployments )
> kinit + authorize on all nodes running topologies for multi-node metron
> deployments
>
>
> Key: METRON-1178
> URL: https://issues.apache.org/jira/browse/METRON-1178
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>
> In a 12-node deployment, it is seen post-kerberization that some of the
> topologies fail to start with exceptions thrown like one pasted below. This
> is because the node does not have the proper credentials, since kinit has not
> occurred on this host.
> {code}
> 2017-09-11 16:04:46.923 o.a.h.i.Client [WARN] Exception encountered while
> connecting to the server : javax.security.sasl.SaslException: GSS ini
> tiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> 2017-09-11 16:04:46.930 o.a.m.p.GrokParser [ERROR] Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initi
> ate failed [Caused by GSSException: No valid credentials provided (Mechanism
> level: Failed to find any Kerberos tgt)]; Host Details : local hos
> t is: "metron-2/xx.xx.xx.xx; destination host is:
> "metron-2.openstacklocal":8020;
> java.io.IOException: Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSExcep
> tion: No valid credentials provided (Mechanism level: Failed to find any
> Kerberos tgt)]; Host Details : local host is: "nat-r7-lqys-metron-2/17
> 2.22.104.43"; destination host is: "nat-r7-lqys-metron-2.openstacklocal":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> ~[stormjar.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1480) ~[stormjar.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1407) ~[stormjar.jar:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> ~[stormjar.jar:?]
> at com.sun.proxy.$Proxy45.getFileInfo(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
> ~[stor
> mjar.jar:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_141]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_141]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> ~[stormjar.jar:?]
> at com.sun.proxy.$Proxy46.getFileInfo(Unknown Source) ~[?:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> ~[stormjar.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
> ~[stormjar.jar:?]
> at
> org.apache.metron.parsers.GrokParser.openInputStream(GrokParser.java:83)
> ~[stormjar.jar:?]
> at org.apache.metron.parsers.GrokParser.init(GrokParser.java:94)
> [stormjar.jar:?]
> at
> org.apache.metron.parsers.bolt.ParserBolt.prepare(ParserBolt.java:108)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__6573$fn__6586.invoke(executor.clj:798)
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:482)
> [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
> Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS
> initiate failed [Caused by GSSException: No valid credentials provided (
> Mechanism level: Failed to find any Kerberos
