Hi,
I am using Storm to write HBase with Simple authentication. HBase's version is
1.2.0-cdh5.7.0. But the connection somehow can not be obtained. The user
submitting the topology and running the Storm cluster is the same as the HBase
user, which has write and read privileges on the HBase. Below is the code
snippet where a connection is attempted to obtain.
public HBaseClient(Map<String, Object> map, final Configuration configuration,
final String tableName) {
try{
UserProvider provider = HBaseSecurityUtil.login(map, configuration);
final User user = provider.getCurrent();
System.out.println("Communicate with HBase as " + user.getName());
table = user.getUGI().doAs(new PrivilegedExceptionAction<Table>() {
@Override
public Table run() throws IOException {
System.out.println("Obtain a connection to HBase");
Connection conn =
ConnectionFactory.createConnection(configuration, user);
System.out.println("Connection obtained.");
return conn.getTable(TableName.valueOf(tableName));
}
});
}catch(Exception e){
throw new RuntimeException("HBase bolt preparation failed: " +
e.getMessage(), e);
}
}
And the exception log is as follow:
java.lang.RuntimeException: HBase bolt preparation failed:
java.lang.reflect.InvocationTargetException
at
org.apache.storm.hbase.common.HBaseClient.<init>(HBaseClient.java:58)
~[stormjar.jar:?]
at
org.apache.storm.hbase.bolt.AbstractHBaseBolt.prepare(AbstractHBaseBolt.java:75)
~[stormjar.jar:?]
at org.apache.storm.hbase.bolt.HBaseBolt.prepare(HBaseBolt.java:108)
~[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__5044$fn__5057.invoke(executor.clj:791)
~[storm-core-1.1.0.jar:1.1.0]
at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:482)
[storm-core-1.1.0.jar:1.1.0]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:181)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at org.apache.storm.hbase.common.HBaseClient$1.run(HBaseClient.java:52)
~[stormjar.jar:?]
at org.apache.storm.hbase.common.HBaseClient$1.run(HBaseClient.java:48)
~[stormjar.jar:?]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_141]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_141]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
~[hadoop-common-2.6.0-cdh5.7.0.jar:?]
at
org.apache.storm.hbase.common.HBaseClient.<init>(HBaseClient.java:48)
~[stormjar.jar:?]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[?:1.8.0_141]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
~[?:1.8.0_141]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[?:1.8.0_141]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
~[?:1.8.0_141]
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:181)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at org.apache.storm.hbase.common.HBaseClient$1.run(HBaseClient.java:52)
~[stormjar.jar:?]
at org.apache.storm.hbase.common.HBaseClient$1.run(HBaseClient.java:48)
~[stormjar.jar:?]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_141]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_141]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
~[hadoop-common-2.6.0-cdh5.7.0.jar:?]
at
org.apache.storm.hbase.common.HBaseClient.<init>(HBaseClient.java:48)
~[stormjar.jar:?]
... 6 more
Caused by: java.lang.ExceptionInInitializerError
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2742)
~[hadoop-common-2.6.0-cdh5.7.0.jar:?]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2715)
~[hadoop-common-2.6.0-cdh5.7.0.jar:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:382)
~[hadoop-common-2.6.0-cdh5.7.0.jar:?]
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
~[hadoop-common-2.6.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:120)
~[hbase-common-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.util.DynamicClassLoader.<init>(DynamicClassLoader.java:98)
~[hbase-common-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.<clinit>(ProtobufUtil.java:241)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:919)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:657)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[?:1.8.0_141]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
~[?:1.8.0_141]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[?:1.8.0_141]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
~[?:1.8.0_141]
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:181)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at org.apache.storm.hbase.common.HBaseClient$1.run(HBaseClient.java:52)
~[stormjar.jar:?]
at org.apache.storm.hbase.common.HBaseClient$1.run(HBaseClient.java:48)
~[stormjar.jar:?]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_141]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_141]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
~[hadoop-common-2.6.0-cdh5.7.0.jar:?]
at
org.apache.storm.hbase.common.HBaseClient.<init>(HBaseClient.java:48)
~[stormjar.jar:?]
... 6 more
Caused by: java.lang.IllegalStateException: Shutdown in progress
at
java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66)
~[?:1.8.0_141]
at java.lang.Runtime.addShutdownHook(Runtime.java:211) ~[?:1.8.0_141]
at
org.apache.hadoop.util.ShutdownHookManager.<clinit>(ShutdownHookManager.java:47)
~[hadoop-common-2.6.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2742)
~[hadoop-common-2.6.0-cdh5.7.0.jar:?]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2715)
~[hadoop-common-2.6.0-cdh5.7.0.jar:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:382)
~[hadoop-common-2.6.0-cdh5.7.0.jar:?]
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
~[hadoop-common-2.6.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:120)
~[hbase-common-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.util.DynamicClassLoader.<init>(DynamicClassLoader.java:98)
~[hbase-common-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.<clinit>(ProtobufUtil.java:241)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:919)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:657)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[?:1.8.0_141]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
~[?:1.8.0_141]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[?:1.8.0_141]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
~[?:1.8.0_141]
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:181)
~[hbase-client-1.2.0-cdh5.7.0.jar:?]
at org.apache.storm.hbase.common.HBaseClient$1.run(HBaseClient.java:52)
~[stormjar.jar:?]
at org.apache.storm.hbase.common.HBaseClient$1.run(HBaseClient.java:48)
~[stormjar.jar:?]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_141]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_141]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
~[hadoop-common-2.6.0-cdh5.7.0.jar:?]
at
org.apache.storm.hbase.common.HBaseClient.<init>(HBaseClient.java:48)
~[stormjar.jar:?]
Curiously, I find the connection is obtained successfully by examining the same
log file.
2017-12-01 15:22:49.146 o.a.s.d.executor Thread-49-MigrateCounter-executor[59
59] [INFO] Prepared bolt MigrateCounter:(59)
2017-12-01 15:22:49.149 o.a.s.d.executor Thread-19-MigrateCounter-executor[65
65] [INFO] Prepared bolt MigrateCounter:(65)
2017-12-01 15:22:49.154 o.a.s.d.executor Thread-45-MigrateCounter-executor[89
89] [INFO] Prepared bolt MigrateCounter:(89)
2017-12-01 15:22:49.157 o.a.s.d.executor
Thread-23-IsdnHBaseMapperBolt-executor[35 35] [INFO] Prepared bolt
IsdnHBaseMapperBolt:(35)
2017-12-01 15:22:49.158 o.a.s.d.executor
Thread-37-AppHBaseMapperBolt-executor[5 5] [INFO] Prepared bolt
AppHBaseMapperBolt:(5)
2017-12-01 15:22:49.158 o.a.s.d.executor
Thread-43-TopNAppWindowedBolt-executor[143 143] [INFO] Prepared bolt
TopNAppWindowedBolt:(143)
2017-12-01 15:22:49.159 o.a.s.d.executor Thread-15-MigrateCounter-executor[137
137] [INFO] Prepared bolt MigrateCounter:(137)
2017-12-01 15:22:49.160 o.a.s.d.executor Thread-17-MigrateCounter-executor[53
53] [INFO] Prepared bolt MigrateCounter:(53)
2017-12-01 15:22:49.161 o.a.s.d.executor Thread-47-MigrateCounter-executor[83
83] [INFO] Prepared bolt MigrateCounter:(83)
2017-12-01 15:22:49.161 o.a.s.d.executor Thread-29-DPIParseBolt-executor[23 23]
[INFO] Prepared bolt DPIParseBolt:(23)
2017-12-01 15:22:49.162 o.a.s.d.executor Thread-9-MigrateCounter-executor[107
107] [INFO] Prepared bolt MigrateCounter:(107)
2017-12-01 15:22:49.162 o.a.s.d.executor Thread-25-MigrateCounter-executor[101
101] [INFO] Prepared bolt MigrateCounter:(101)
2017-12-01 15:22:49.638 o.a.h.h.s.DomainSocketFactory
Thread-57-AppMetricBolt-executor[17 17] [WARN] The short-circuit local reads
feature cannot be used because libhadoop cannot be loaded.
2017-12-01 15:22:49.697 STDIO Thread-57-AppMetricBolt-executor[17 17] [INFO]
Connection obtained.
2017-12-01 15:22:49.707 o.a.s.d.executor Thread-57-AppMetricBolt-executor[17
17] [INFO] Prepared bolt AppMetricBolt:(17)
2017-12-01 15:22:49.891 o.a.s.util Thread-31-AppHBaseMapperBolt-executor[11 11]
[ERROR] Async loop died!
Could anybody offer some suggestions?
________________________________
[email protected]
