Need info about hbase connection pool
Hi, I am using hbase connection pooling using HConnection hConnection=HConnectionManager.createConnection(conf); I see most of method of HConnectionManager goes to depreciated. Please provide to me best way to create Hbase connection pool and what your future plan for this topic. thanks OmParkash
Re: Need info about hbase connection pool
HConnection is also deprecated. It would be better to do: Connection connection = ConnectionFactory.createConnection(conf); On Wed, Mar 18, 2015 at 7:55 AM, OM PARKASH Nain mr.omparkashn...@gmail.com wrote: Hi, I am using hbase connection pooling using HConnection hConnection=HConnectionManager.createConnection(conf); I see most of method of HConnectionManager goes to depreciated. Please provide to me best way to create Hbase connection pool and what your future plan for this topic. thanks OmParkash
Re: Need info about hbase connection pool
In 1.0+ API, there's no more automatic pooling. You application uses the ConnectionFactory (as Solomon says) to get a connection instance for the application, and retrieve accessor objects (Table, Admin, c) from that singleton. Have a look at https://github.com/ndimiduk/hbase-1.0-api-examples for some examples. On Wed, Mar 18, 2015 at 7:56 AM, Solomon Duskis sdus...@gmail.com wrote: HConnection is also deprecated. It would be better to do: Connection connection = ConnectionFactory.createConnection(conf); On Wed, Mar 18, 2015 at 7:55 AM, OM PARKASH Nain mr.omparkashn...@gmail.com wrote: Hi, I am using hbase connection pooling using HConnection hConnection=HConnectionManager.createConnection(conf); I see most of method of HConnectionManager goes to depreciated. Please provide to me best way to create Hbase connection pool and what your future plan for this topic. thanks OmParkash
Re: HBase connection pool
Did you check how many open connections each ZK server has?
I my hypothesis is that you have ZK connection leaking and ZK server starts
to drop connection to prevent DDoS attack since you hit limit for opened
connections.
2015-02-26 22:15 GMT+03:00 Nick Dimiduk ndimi...@gmail.com:
Can you tell when these WARN messages are produced? Is it related to the
creation of the connection object or one of the HTable instances?
On Thu, Feb 26, 2015 at 7:27 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
Nick,
I tried what you suggested, 1 HConnection and 1 Configuration for the
entire app:
this.config = HBaseConfiguration.create();
this.connection = HConnectionManager.createConnection(config);
And Threaded pooled HTableInterfaces:
final HConnection lconnection = this.connection;
this.tlTable = new ThreadLocalHTableInterface() {
@Override
protected HTableInterface initialValue() {
try {
return lconnection.getTable(HBaseSerialWritesPOC);
// return new HTable(tlConfig.get(),
// HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I started getting this error in my application:
2015-02-26 10:23:17,833 INFO [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:logStartConnect(966)) - Opening socket connection to
server xxx. Will not attempt to authenticate using SASL (unknown error)
2015-02-26 10:23:17,834 INFO [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:primeConnection(849)) - Socket connection established to
xxx, initiating session
2015-02-26 10:23:17,836 WARN [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:run(1089)) - Session 0x0 for server xxx, unexpected
error,
closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
-Marcelo
From: ndimi...@gmail.com
Subject: Re: HBase connection pool
Okay, looks like you're using a implicitly managed connection. It should
be fine to share a single config instance across all threads. The
advantage
of HTablePool over this approach is that the number of HTables would be
managed independently from the number of Threads. This may or not be a
concern for you, based on your memory requirements, c. In your case,
you're not specifying an ExecutorService per HTable, so the HTable
instances will be relatively light weight. Each table will manage it's
own
write buffer, which can be shared by multiple threads when autoFlush is
disabled and HTablePool is used. This may or may not be desirable,
depending on your use-case.
For what it's worth, HTablePool is marked deprecated in 1.0, will likely
be removed in 2.0. To future proof this code, I would move to a single
shared HConnection for the whole application, and a thread-local HTable
created from/with that connection.
-n
On Wed, Feb 25, 2015 at 10:53 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
Hi Nick,
I am using HBase version 0.96, I sent the link from version 0.94 because
I haven't found the java API docs for 0.96, sorry about that.
I have created the HTable directly from the config object, as follows:
this.tlConfig = new ThreadLocalConfiguration() {
@Override
protected Configuration initialValue() {
return HBaseConfiguration.create();
}
};
this.tlTable = new ThreadLocalHTable() {
@Override
protected HTable initialValue() {
try {
return new HTable(tlConfig.get(), HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I am now sure if the Configuration object should be 1 per thread as
well,
maybe I could share this one?
So, just to clarify, would I get any advantage using HTablePool object
instead of ThreadLocalHTable as I did?
-Marcelo
From: ndimi...@gmail.com
Subject: Re: HBase connection pool
Hi Marcelo,
First thing, to be clear, you're working with a 0.94 release? The reason
I ask is we've been doing some work in this area to improve things, so
semantics may be slightly different between 0.94, 0.98, and 1.0.
How are you managing the HConnection object (or are you)? How are you
creating your HTable instances? These will determine how the connection
is
obtained and used in relation to HTables.
In general, multiple HTable instances connected to tables in the same
cluster should be sharing
Re: HBase connection pool
But then wouldn't it happen when I had 1 Configuration per thread? I had more
connections before start using 1 HConnection for the whole app, and it use to
work fine.
From: user@hbase.apache.org
Subject: Re: HBase connection pool
Did you check how many open connections each ZK server has?
I my hypothesis is that you have ZK connection leaking and ZK server starts
to drop connection to prevent DDoS attack since you hit limit for opened
connections.
2015-02-26 22:15 GMT+03:00 Nick Dimiduk ndimi...@gmail.com:
Can you tell when these WARN messages are produced? Is it related to the
creation of the connection object or one of the HTable instances?
On Thu, Feb 26, 2015 at 7:27 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
Nick,
I tried what you suggested, 1 HConnection and 1 Configuration for the
entire app:
this.config = HBaseConfiguration.create();
this.connection = HConnectionManager.createConnection(config);
And Threaded pooled HTableInterfaces:
final HConnection lconnection = this.connection;
this.tlTable = new ThreadLocalHTableInterface() {
@Override
protected HTableInterface initialValue() {
try {
return lconnection.getTable(HBaseSerialWritesPOC);
// return new HTable(tlConfig.get(),
// HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I started getting this error in my application:
2015-02-26 10:23:17,833 INFO [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:logStartConnect(966)) - Opening socket connection to
server xxx. Will not attempt to authenticate using SASL (unknown error)
2015-02-26 10:23:17,834 INFO [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:primeConnection(849)) - Socket connection established to
xxx, initiating session
2015-02-26 10:23:17,836 WARN [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:run(1089)) - Session 0x0 for server xxx, unexpected
error,
closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
-Marcelo
From: ndimi...@gmail.com
Subject: Re: HBase connection pool
Okay, looks like you're using a implicitly managed connection. It should
be fine to share a single config instance across all threads. The
advantage
of HTablePool over this approach is that the number of HTables would be
managed independently from the number of Threads. This may or not be a
concern for you, based on your memory requirements, c. In your case,
you're not specifying an ExecutorService per HTable, so the HTable
instances will be relatively light weight. Each table will manage it's
own
write buffer, which can be shared by multiple threads when autoFlush is
disabled and HTablePool is used. This may or may not be desirable,
depending on your use-case.
For what it's worth, HTablePool is marked deprecated in 1.0, will likely
be removed in 2.0. To future proof this code, I would move to a single
shared HConnection for the whole application, and a thread-local HTable
created from/with that connection.
-n
On Wed, Feb 25, 2015 at 10:53 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
Hi Nick,
I am using HBase version 0.96, I sent the link from version 0.94 because
I haven't found the java API docs for 0.96, sorry about that.
I have created the HTable directly from the config object, as follows:
this.tlConfig = new ThreadLocalConfiguration() {
@Override
protected Configuration initialValue() {
return HBaseConfiguration.create();
}
};
this.tlTable = new ThreadLocalHTable() {
@Override
protected HTable initialValue() {
try {
return new HTable(tlConfig.get(), HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I am now sure if the Configuration object should be 1 per thread as
well,
maybe I could share this one?
So, just to clarify, would I get any advantage using HTablePool object
instead of ThreadLocalHTable as I did?
-Marcelo
From: ndimi...@gmail.com
Subject: Re: HBase connection pool
Hi Marcelo,
First thing, to be clear, you're working with a 0.94 release? The reason
I ask is we've been doing some work in this area to improve things, so
semantics may be slightly different between 0.94, 0.98, and 1.0.
How are you managing the HConnection object (or are you
Re: HBase connection pool
I can't know for sure, but I think it's related to the HTable instances...
I create just 1 HConnection and 1 HTable object per thread. I have run this
with 32 threads.
From: user@hbase.apache.org
Subject: Re: HBase connection pool
Can you tell when these WARN messages are produced? Is it related to the
creation of the connection object or one of the HTable instances?
On Thu, Feb 26, 2015 at 7:27 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
Nick,
I tried what you suggested, 1 HConnection and 1 Configuration for the
entire app:
this.config = HBaseConfiguration.create();
this.connection = HConnectionManager.createConnection(config);
And Threaded pooled HTableInterfaces:
final HConnection lconnection = this.connection;
this.tlTable = new ThreadLocalHTableInterface() {
@Override
protected HTableInterface initialValue() {
try {
return lconnection.getTable(HBaseSerialWritesPOC);
// return new HTable(tlConfig.get(),
// HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I started getting this error in my application:
2015-02-26 10:23:17,833 INFO [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:logStartConnect(966)) - Opening socket connection to
server xxx. Will not attempt to authenticate using SASL (unknown error)
2015-02-26 10:23:17,834 INFO [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:primeConnection(849)) - Socket connection established to
xxx, initiating session
2015-02-26 10:23:17,836 WARN [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:run(1089)) - Session 0x0 for server xxx, unexpected error,
closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
-Marcelo
From: ndimi...@gmail.com
Subject: Re: HBase connection pool
Okay, looks like you're using a implicitly managed connection. It should
be fine to share a single config instance across all threads. The advantage
of HTablePool over this approach is that the number of HTables would be
managed independently from the number of Threads. This may or not be a
concern for you, based on your memory requirements, c. In your case,
you're not specifying an ExecutorService per HTable, so the HTable
instances will be relatively light weight. Each table will manage it's own
write buffer, which can be shared by multiple threads when autoFlush is
disabled and HTablePool is used. This may or may not be desirable,
depending on your use-case.
For what it's worth, HTablePool is marked deprecated in 1.0, will likely
be removed in 2.0. To future proof this code, I would move to a single
shared HConnection for the whole application, and a thread-local HTable
created from/with that connection.
-n
On Wed, Feb 25, 2015 at 10:53 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
Hi Nick,
I am using HBase version 0.96, I sent the link from version 0.94 because
I haven't found the java API docs for 0.96, sorry about that.
I have created the HTable directly from the config object, as follows:
this.tlConfig = new ThreadLocalConfiguration() {
@Override
protected Configuration initialValue() {
return HBaseConfiguration.create();
}
};
this.tlTable = new ThreadLocalHTable() {
@Override
protected HTable initialValue() {
try {
return new HTable(tlConfig.get(), HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I am now sure if the Configuration object should be 1 per thread as well,
maybe I could share this one?
So, just to clarify, would I get any advantage using HTablePool object
instead of ThreadLocalHTable as I did?
-Marcelo
From: ndimi...@gmail.com
Subject: Re: HBase connection pool
Hi Marcelo,
First thing, to be clear, you're working with a 0.94 release? The reason
I ask is we've been doing some work in this area to improve things, so
semantics may be slightly different between 0.94, 0.98, and 1.0.
How are you managing the HConnection object (or are you)? How are you
creating your HTable instances? These will determine how the connection is
obtained and used in relation to HTables.
In general, multiple HTable instances connected to tables in the same
cluster should be sharing the same HConnection instance. This is handled
explicitly when you manage your own HConnection and HTables (i.e.,
HConnection conn = ... ; HTable t = new HTable(TABLE_NAME, conn
Re: HBase connection pool
Create one HConnection for all threads and then share it.
Create HTable in each tread using HConnection.
Do stuff.
Close HTable, DO NOT close HConnection.
It works 100% I did have pretty the same problem. Group helped me to
resolve it the way I suggest you.
2015-02-27 13:23 GMT+03:00 Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net:
But then wouldn't it happen when I had 1 Configuration per thread? I had
more connections before start using 1 HConnection for the whole app, and it
use to work fine.
From: user@hbase.apache.org
Subject: Re: HBase connection pool
Did you check how many open connections each ZK server has?
I my hypothesis is that you have ZK connection leaking and ZK server starts
to drop connection to prevent DDoS attack since you hit limit for opened
connections.
2015-02-26 22:15 GMT+03:00 Nick Dimiduk ndimi...@gmail.com:
Can you tell when these WARN messages are produced? Is it related to the
creation of the connection object or one of the HTable instances?
On Thu, Feb 26, 2015 at 7:27 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
Nick,
I tried what you suggested, 1 HConnection and 1 Configuration for the
entire app:
this.config = HBaseConfiguration.create();
this.connection = HConnectionManager.createConnection(config);
And Threaded pooled HTableInterfaces:
final HConnection lconnection = this.connection;
this.tlTable = new ThreadLocalHTableInterface() {
@Override
protected HTableInterface initialValue() {
try {
return lconnection.getTable(HBaseSerialWritesPOC);
// return new HTable(tlConfig.get(),
// HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I started getting this error in my application:
2015-02-26 10:23:17,833 INFO [main-SendThread(xxx)]
zookeeper.ClientCnxn
(ClientCnxn.java:logStartConnect(966)) - Opening socket connection to
server xxx. Will not attempt to authenticate using SASL (unknown error)
2015-02-26 10:23:17,834 INFO [main-SendThread(xxx)]
zookeeper.ClientCnxn
(ClientCnxn.java:primeConnection(849)) - Socket connection established
to
xxx, initiating session
2015-02-26 10:23:17,836 WARN [main-SendThread(xxx)]
zookeeper.ClientCnxn
(ClientCnxn.java:run(1089)) - Session 0x0 for server xxx, unexpected
error,
closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
-Marcelo
From: ndimi...@gmail.com
Subject: Re: HBase connection pool
Okay, looks like you're using a implicitly managed connection. It
should
be fine to share a single config instance across all threads. The
advantage
of HTablePool over this approach is that the number of HTables would be
managed independently from the number of Threads. This may or not be a
concern for you, based on your memory requirements, c. In your case,
you're not specifying an ExecutorService per HTable, so the HTable
instances will be relatively light weight. Each table will manage it's
own
write buffer, which can be shared by multiple threads when autoFlush is
disabled and HTablePool is used. This may or may not be desirable,
depending on your use-case.
For what it's worth, HTablePool is marked deprecated in 1.0, will
likely
be removed in 2.0. To future proof this code, I would move to a
single
shared HConnection for the whole application, and a thread-local HTable
created from/with that connection.
-n
On Wed, Feb 25, 2015 at 10:53 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
Hi Nick,
I am using HBase version 0.96, I sent the link from version 0.94
because
I haven't found the java API docs for 0.96, sorry about that.
I have created the HTable directly from the config object, as follows:
this.tlConfig = new ThreadLocalConfiguration() {
@Override
protected Configuration initialValue() {
return HBaseConfiguration.create();
}
};
this.tlTable = new ThreadLocalHTable() {
@Override
protected HTable initialValue() {
try {
return new HTable(tlConfig.get(), HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I am now sure if the Configuration object should be 1 per thread as
well,
maybe I could share this one?
So
Re: HBase connection pool
Nick,
I tried what you suggested, 1 HConnection and 1 Configuration for the entire
app:
this.config = HBaseConfiguration.create();
this.connection = HConnectionManager.createConnection(config);
And Threaded pooled HTableInterfaces:
final HConnection lconnection = this.connection;
this.tlTable = new ThreadLocalHTableInterface() {
@Override
protected HTableInterface initialValue() {
try {
return lconnection.getTable(HBaseSerialWritesPOC);
// return new HTable(tlConfig.get(),
// HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I started getting this error in my application:
2015-02-26 10:23:17,833 INFO [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:logStartConnect(966)) - Opening socket connection to server
xxx. Will not attempt to authenticate using SASL (unknown error)
2015-02-26 10:23:17,834 INFO [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:primeConnection(849)) - Socket connection established to xxx,
initiating session
2015-02-26 10:23:17,836 WARN [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:run(1089)) - Session 0x0 for server xxx, unexpected error,
closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
-Marcelo
From: ndimi...@gmail.com
Subject: Re: HBase connection pool
Okay, looks like you're using a implicitly managed connection. It should be
fine to share a single config instance across all threads. The advantage of
HTablePool over this approach is that the number of HTables would be managed
independently from the number of Threads. This may or not be a concern for you,
based on your memory requirements, c. In your case, you're not specifying an
ExecutorService per HTable, so the HTable instances will be relatively light
weight. Each table will manage it's own write buffer, which can be shared by
multiple threads when autoFlush is disabled and HTablePool is used. This may or
may not be desirable, depending on your use-case.
For what it's worth, HTablePool is marked deprecated in 1.0, will likely be
removed in 2.0. To future proof this code, I would move to a single shared
HConnection for the whole application, and a thread-local HTable created
from/with that connection.
-n
On Wed, Feb 25, 2015 at 10:53 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
Hi Nick,
I am using HBase version 0.96, I sent the link from version 0.94 because I
haven't found the java API docs for 0.96, sorry about that.
I have created the HTable directly from the config object, as follows:
this.tlConfig = new ThreadLocalConfiguration() {
@Override
protected Configuration initialValue() {
return HBaseConfiguration.create();
}
};
this.tlTable = new ThreadLocalHTable() {
@Override
protected HTable initialValue() {
try {
return new HTable(tlConfig.get(), HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I am now sure if the Configuration object should be 1 per thread as well, maybe
I could share this one?
So, just to clarify, would I get any advantage using HTablePool object instead
of ThreadLocalHTable as I did?
-Marcelo
From: ndimi...@gmail.com
Subject: Re: HBase connection pool
Hi Marcelo,
First thing, to be clear, you're working with a 0.94 release? The reason I ask
is we've been doing some work in this area to improve things, so semantics may
be slightly different between 0.94, 0.98, and 1.0.
How are you managing the HConnection object (or are you)? How are you creating
your HTable instances? These will determine how the connection is obtained and
used in relation to HTables.
In general, multiple HTable instances connected to tables in the same cluster
should be sharing the same HConnection instance. This is handled explicitly
when you manage your own HConnection and HTables (i.e., HConnection conn = ...
; HTable t = new HTable(TABLE_NAME, conn); ) It's handled implicitly when you
construct via Configuration objects (HTable t = new HTable(conf, TABLE_NAME); )
This implicit option is going away in future versions.
HTable is not safe for concurrent access because of how the write path is
implemented (at least; there may be other portions that I'm not as familiar
with). You should be perfectly fine to have an HTable per thread in a
ThreadLocal.
-n
On Wed, Feb 25, 2015 at 9:41 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
In HBase API, does 1
Re: HBase connection pool
Can you tell when these WARN messages are produced? Is it related to the
creation of the connection object or one of the HTable instances?
On Thu, Feb 26, 2015 at 7:27 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
Nick,
I tried what you suggested, 1 HConnection and 1 Configuration for the
entire app:
this.config = HBaseConfiguration.create();
this.connection = HConnectionManager.createConnection(config);
And Threaded pooled HTableInterfaces:
final HConnection lconnection = this.connection;
this.tlTable = new ThreadLocalHTableInterface() {
@Override
protected HTableInterface initialValue() {
try {
return lconnection.getTable(HBaseSerialWritesPOC);
// return new HTable(tlConfig.get(),
// HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I started getting this error in my application:
2015-02-26 10:23:17,833 INFO [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:logStartConnect(966)) - Opening socket connection to
server xxx. Will not attempt to authenticate using SASL (unknown error)
2015-02-26 10:23:17,834 INFO [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:primeConnection(849)) - Socket connection established to
xxx, initiating session
2015-02-26 10:23:17,836 WARN [main-SendThread(xxx)] zookeeper.ClientCnxn
(ClientCnxn.java:run(1089)) - Session 0x0 for server xxx, unexpected error,
closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
-Marcelo
From: ndimi...@gmail.com
Subject: Re: HBase connection pool
Okay, looks like you're using a implicitly managed connection. It should
be fine to share a single config instance across all threads. The advantage
of HTablePool over this approach is that the number of HTables would be
managed independently from the number of Threads. This may or not be a
concern for you, based on your memory requirements, c. In your case,
you're not specifying an ExecutorService per HTable, so the HTable
instances will be relatively light weight. Each table will manage it's own
write buffer, which can be shared by multiple threads when autoFlush is
disabled and HTablePool is used. This may or may not be desirable,
depending on your use-case.
For what it's worth, HTablePool is marked deprecated in 1.0, will likely
be removed in 2.0. To future proof this code, I would move to a single
shared HConnection for the whole application, and a thread-local HTable
created from/with that connection.
-n
On Wed, Feb 25, 2015 at 10:53 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
Hi Nick,
I am using HBase version 0.96, I sent the link from version 0.94 because
I haven't found the java API docs for 0.96, sorry about that.
I have created the HTable directly from the config object, as follows:
this.tlConfig = new ThreadLocalConfiguration() {
@Override
protected Configuration initialValue() {
return HBaseConfiguration.create();
}
};
this.tlTable = new ThreadLocalHTable() {
@Override
protected HTable initialValue() {
try {
return new HTable(tlConfig.get(), HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I am now sure if the Configuration object should be 1 per thread as well,
maybe I could share this one?
So, just to clarify, would I get any advantage using HTablePool object
instead of ThreadLocalHTable as I did?
-Marcelo
From: ndimi...@gmail.com
Subject: Re: HBase connection pool
Hi Marcelo,
First thing, to be clear, you're working with a 0.94 release? The reason
I ask is we've been doing some work in this area to improve things, so
semantics may be slightly different between 0.94, 0.98, and 1.0.
How are you managing the HConnection object (or are you)? How are you
creating your HTable instances? These will determine how the connection is
obtained and used in relation to HTables.
In general, multiple HTable instances connected to tables in the same
cluster should be sharing the same HConnection instance. This is handled
explicitly when you manage your own HConnection and HTables (i.e.,
HConnection conn = ... ; HTable t = new HTable(TABLE_NAME, conn); ) It's
handled implicitly when you construct via Configuration objects (HTable t =
new HTable(conf, TABLE_NAME); ) This implicit option is going away in
future versions.
HTable is not safe for concurrent access because of how
HBase connection pool
In HBase API, does 1 HTable object means 1 connection to each region server (just for 1 table)? The docs say (http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HTable.html): This class is not thread safe for reads nor write. I got confused, as I saw there is a HTablePool class, but it's only for a table as well, can't connections be reused for more than 1 table? In my java application, I used ThreadLocal variables (ThreadLocalHTable) to create an HTable variable per thread. If I do several operations on each thread, I should still use the same connection, right? []s
Re: HBase connection pool
Okay, looks like you're using a implicitly managed connection. It should be
fine to share a single config instance across all threads. The advantage of
HTablePool over this approach is that the number of HTables would be
managed independently from the number of Threads. This may or not be a
concern for you, based on your memory requirements, c. In your case,
you're not specifying an ExecutorService per HTable, so the HTable
instances will be relatively light weight. Each table will manage it's own
write buffer, which can be shared by multiple threads when autoFlush is
disabled and HTablePool is used. This may or may not be desirable,
depending on your use-case.
For what it's worth, HTablePool is marked deprecated in 1.0, will likely be
removed in 2.0. To future proof this code, I would move to a single
shared HConnection for the whole application, and a thread-local HTable
created from/with that connection.
-n
On Wed, Feb 25, 2015 at 10:53 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
Hi Nick,
I am using HBase version 0.96, I sent the link from version 0.94 because I
haven't found the java API docs for 0.96, sorry about that.
I have created the HTable directly from the config object, as follows:
this.tlConfig = new ThreadLocalConfiguration() {
@Override
protected Configuration initialValue() {
return HBaseConfiguration.create();
}
};
this.tlTable = new ThreadLocalHTable() {
@Override
protected HTable initialValue() {
try {
return new HTable(tlConfig.get(), HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I am now sure if the Configuration object should be 1 per thread as well,
maybe I could share this one?
So, just to clarify, would I get any advantage using HTablePool object
instead of ThreadLocalHTable as I did?
-Marcelo
From: ndimi...@gmail.com
Subject: Re: HBase connection pool
Hi Marcelo,
First thing, to be clear, you're working with a 0.94 release? The reason I
ask is we've been doing some work in this area to improve things, so
semantics may be slightly different between 0.94, 0.98, and 1.0.
How are you managing the HConnection object (or are you)? How are you
creating your HTable instances? These will determine how the connection is
obtained and used in relation to HTables.
In general, multiple HTable instances connected to tables in the same
cluster should be sharing the same HConnection instance. This is handled
explicitly when you manage your own HConnection and HTables (i.e.,
HConnection conn = ... ; HTable t = new HTable(TABLE_NAME, conn); ) It's
handled implicitly when you construct via Configuration objects (HTable t =
new HTable(conf, TABLE_NAME); ) This implicit option is going away in
future versions.
HTable is not safe for concurrent access because of how the write path is
implemented (at least; there may be other portions that I'm not as familiar
with). You should be perfectly fine to have an HTable per thread in a
ThreadLocal.
-n
On Wed, Feb 25, 2015 at 9:41 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
In HBase API, does 1 HTable object means 1 connection to each region
server (just for 1 table)?
The docs say (
http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HTable.html
):
This class is not thread safe for reads nor write.
I got confused, as I saw there is a HTablePool class, but it's only for a
table as well, can't connections be reused for more than 1 table?
In my java application, I used ThreadLocal variables
(ThreadLocalHTable) to create an HTable variable per thread. If I do
several operations on each thread, I should still use the same connection,
right?
[]s
Re: HBase connection pool
Hi Nick,
I am using HBase version 0.96, I sent the link from version 0.94 because I
haven't found the java API docs for 0.96, sorry about that.
I have created the HTable directly from the config object, as follows:
this.tlConfig = new ThreadLocalConfiguration() {
@Override
protected Configuration initialValue() {
return HBaseConfiguration.create();
}
};
this.tlTable = new ThreadLocalHTable() {
@Override
protected HTable initialValue() {
try {
return new HTable(tlConfig.get(), HBaseSerialWritesPOC);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
};
I am now sure if the Configuration object should be 1 per thread as well, maybe
I could share this one?
So, just to clarify, would I get any advantage using HTablePool object instead
of ThreadLocalHTable as I did?
-Marcelo
From: ndimi...@gmail.com
Subject: Re: HBase connection pool
Hi Marcelo,
First thing, to be clear, you're working with a 0.94 release? The reason I ask
is we've been doing some work in this area to improve things, so semantics may
be slightly different between 0.94, 0.98, and 1.0.
How are you managing the HConnection object (or are you)? How are you creating
your HTable instances? These will determine how the connection is obtained and
used in relation to HTables.
In general, multiple HTable instances connected to tables in the same cluster
should be sharing the same HConnection instance. This is handled explicitly
when you manage your own HConnection and HTables (i.e., HConnection conn = ...
; HTable t = new HTable(TABLE_NAME, conn); ) It's handled implicitly when you
construct via Configuration objects (HTable t = new HTable(conf, TABLE_NAME); )
This implicit option is going away in future versions.
HTable is not safe for concurrent access because of how the write path is
implemented (at least; there may be other portions that I'm not as familiar
with). You should be perfectly fine to have an HTable per thread in a
ThreadLocal.
-n
On Wed, Feb 25, 2015 at 9:41 AM, Marcelo Valle (BLOOMBERG/ LONDON)
mvallemil...@bloomberg.net wrote:
In HBase API, does 1 HTable object means 1 connection to each region server
(just for 1 table)?
The docs say
(http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HTable.html):
This class is not thread safe for reads nor write.
I got confused, as I saw there is a HTablePool class, but it's only for a table
as well, can't connections be reused for more than 1 table?
In my java application, I used ThreadLocal variables (ThreadLocalHTable) to
create an HTable variable per thread. If I do several operations on each
thread, I should still use the same connection, right?
[]s
Re: HBase connection pool
Hi Marcelo, First thing, to be clear, you're working with a 0.94 release? The reason I ask is we've been doing some work in this area to improve things, so semantics may be slightly different between 0.94, 0.98, and 1.0. How are you managing the HConnection object (or are you)? How are you creating your HTable instances? These will determine how the connection is obtained and used in relation to HTables. In general, multiple HTable instances connected to tables in the same cluster should be sharing the same HConnection instance. This is handled explicitly when you manage your own HConnection and HTables (i.e., HConnection conn = ... ; HTable t = new HTable(TABLE_NAME, conn); ) It's handled implicitly when you construct via Configuration objects (HTable t = new HTable(conf, TABLE_NAME); ) This implicit option is going away in future versions. HTable is not safe for concurrent access because of how the write path is implemented (at least; there may be other portions that I'm not as familiar with). You should be perfectly fine to have an HTable per thread in a ThreadLocal. -n On Wed, Feb 25, 2015 at 9:41 AM, Marcelo Valle (BLOOMBERG/ LONDON) mvallemil...@bloomberg.net wrote: In HBase API, does 1 HTable object means 1 connection to each region server (just for 1 table)? The docs say ( http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HTable.html ): This class is not thread safe for reads nor write. I got confused, as I saw there is a HTablePool class, but it's only for a table as well, can't connections be reused for more than 1 table? In my java application, I used ThreadLocal variables (ThreadLocalHTable) to create an HTable variable per thread. If I do several operations on each thread, I should still use the same connection, right? []s
Re: HBase connection pool
Hey people thanks a lot for your help :) :) aadish wrote: Hey people, I am very new to HBase, and I would like if someone gave me guidance regarding connection pooling. Thanks a lot, in advance. Regards, Aadish -- View this message in context: http://old.nabble.com/HBase-connection-pool-tp32080864p32162608.html Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase connection pool
Hey people thanks a lot for your help :) :) aadish wrote: Hey people, I am very new to HBase, and I would like if someone gave me guidance regarding connection pooling. Thanks a lot, in advance. Regards, Aadish -- View this message in context: http://old.nabble.com/HBase-connection-pool-tp32080864p32162609.html Sent from the HBase User mailing list archive at Nabble.com.
HBase connection pool
Hey people, I am very new to HBase, and I would like if someone gave me guidance regarding connection pooling. Thanks a lot, in advance. Regards, Aadish -- View this message in context: http://old.nabble.com/HBase-connection-pool-tp32080864p32080864.html Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase connection pool
Hi, This has a way to pool through HTablePool http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/10646 Regards, Raja Nagendra Kumar, C.T.O www.tejasoft.com aadish wrote: Hey people, I am very new to HBase, and I would like if someone gave me guidance regarding connection pooling. Thanks a lot, in advance. Regards, Aadish -- View this message in context: http://old.nabble.com/HBase-connection-pool-tp32080864p32080951.html Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase connection pool
Hi there, you probably want to start with this first. http://hbase.apache.org/book.html#client On 7/18/11 2:16 AM, aadish kotwal.aad...@gmail.com wrote: Hey people, I am very new to HBase, and I would like if someone gave me guidance regarding connection pooling. Thanks a lot, in advance. Regards, Aadish -- View this message in context: http://old.nabble.com/HBase-connection-pool-tp32080864p32080864.html Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase connection pool
here is connection pool implementation that works for us: http://pastebin.com/KBv9t7S8 -Jack On Sun, Jul 17, 2011 at 11:16 PM, aadish kotwal.aad...@gmail.com wrote: Hey people, I am very new to HBase, and I would like if someone gave me guidance regarding connection pooling. Thanks a lot, in advance. Regards, Aadish -- View this message in context: http://old.nabble.com/HBase-connection-pool-tp32080864p32080864.html Sent from the HBase User mailing list archive at Nabble.com.
