Of course, I could create a connection in
val result = rdd.map(line = {
val conf = HBaseConfiguration.create
val connection = HConnectionManager.createConnection(conf)
val table = connection.getTable(user)
...
table.close()
connection.close()
}
but that would be too slow, which is
Thanks, Ted.
Util.Connection.close() should be called only once, so it can NOT be in a
map function
val result = rdd.map(line = {
val table = Util.Connection.getTable(user)
...
Util.Connection.close()
}
As you mentioned:
Calling table.close() is the recommended approach.
I may have misunderstood your point.
val result = rdd.map(line = {
val table = Util.Connection.getTable(user)
...
table.close()
}
Did you mean this is enough, and there’s no need to call
Util.Connection.close(),
or HConnectionManager.deleteAllConnections()?
Where is the documentation that
Which hbase release are you using ?
Let me refer to 0.94 code hbase.
Take a look at the following method
in src/main/java/org/apache/hadoop/hbase/client/HTable.java :
public void close() throws IOException {
...
if (cleanupConnectionOnClose) {
if (this.connection != null) {
Thanks, Ted,
We use CDH 5.1 and the HBase version is 0.98.1-cdh5.1.0, in which the
javadoc of HConnectionManager.java still recommends shutdown hook.
I look into val table = Util.Connection.getTable(user), and find it
didn't invoke
public HTable(Configuration conf, final byte[] tableName, final
Looking at Apache 0.98 code, you can follow the example in the class
javadoc (line 144 of HConnectionManager.java):
* HTableInterface table = connection.getTable(table1);
* try {
* // Use the table as needed, for a single operation and a single thread
* } finally {
* table.close();
*
In order to share an HBase connection pool, we create an object
Object Util {
val HBaseConf = HBaseConfiguration.create
val Connection= HConnectionManager.createConnection(HBaseConf)
}
which would be shared among tasks on the same executor. e.g.
val result = rdd.map(line = {
val table
Pardon me - there was typo in previous email.
Calling table.close() is the recommended approach.
HConnectionManager does reference counting. When all references to the
underlying connection are gone, connection would be released.
Cheers
On Wed, Oct 15, 2014 at 7:13 AM, Ted Yu
