So the idea is
1. instantiate HConnection using HConnectionManager once
2. Create HTable instance for each Servlet.doPost and close after operation
is done.
Is that correct?
Do region locations cached in this case?
Are ZK connections to region/ZK reused?
Can I "harm" put/get data because of Servlet concurrency? Each connection
could be used in many threads fro different HTable instances for the same
HBase table?
2014-12-13 22:21 GMT+03:00 lars hofhansl <la...@apache.org>:
>
> Note also that the createConnection part is somewhat expensive (creates a
> new thread pool for use with Puts, also does a ZK lookup, etc).If possible
> create the connection ahead of time and only get/close an HTable per
> request/thread.
> -- Lars
> From: Serega Sheypak <serega.shey...@gmail.com>
> To: user <user@hbase.apache.org>
> Sent: Friday, December 12, 2014 11:45 AM
> Subject: Re: HConnectionManager leaks with zookeeper conection oo many
> connections from /my.tomcat.server.com - max is 60
>
> i have 10K doPost/doGet requests per second.
> Servlet is NOT single-threaded. each doPost/doGet
> invokes these lines (encapsulated in DAO):
>
> 16 HConnection connection =
> HConnectionManager.createConnection(config);
> 17 HTableInterface table =
> connection.getTable(TableName.valueOf("table1"));
>
> and
>
> 24 } finally {
> 25 table.close();
> 26 connection.close();
> 27 }
>
> I assumed that this static construction
>
> 16 HConnection connection =
> HConnectionManager.createConnection(config);
>
> correctly handles multi-threaded access somewhere deep inside.
> Right now I don't understand what do I do wrong.
>
> Try to wrap each your request in Runnable to emulate multi-threaded
> pressure on ZK. Your code is linear and mine is not, it's concurrent.
>
> Thanks....
>
>
>
>
>
>
> 2014-12-12 22:28 GMT+03:00 Stack <st...@duboce.net>:
> >
> > I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted your
> > code adding in a put for each cycle. I ran loop 1000 times and no
> > complaint from zk.
> >
> > Tell me more (Is servlet doing single-threaded model? A single
> > Configuration is being used or new ones are being created per servlet
> > invocation?
> >
> > Below is code and output.
> >
> > (For better perf, cache the connection)
> > St.Ack
> >
> >
> > 1 package org.apache.hadoop.hbase;
> > 2
> > 3 import java.io.IOException;
> > 4
> > 5 import org.apache.hadoop.conf.Configuration;
> > 6 import org.apache.hadoop.hbase.client.HConnection;
> > 7 import org.apache.hadoop.hbase.client.HConnectionManager;
> > 8 import org.apache.hadoop.hbase.client.HTableInterface;
> > 9 import org.apache.hadoop.hbase.client.Put;
> > 10 import org.apache.hadoop.hbase.util.Bytes;
> > 11
> > 12 public class TestConnnection {
> > 13 public static void main(String[] args) throws IOException {
> > 14 Configuration config = HBaseConfiguration.create();
> > 15 for (int i = 0; i < 1000; i++) {
> > 16 HConnection connection =
> > HConnectionManager.createConnection(config);
> > 17 HTableInterface table =
> > connection.getTable(TableName.valueOf("table1"));
> > 18 byte [] cf = Bytes.toBytes("t");
> > 19 try {
> > 20 byte [] bytes = Bytes.toBytes(i);
> > 21 Put p = new Put(bytes);
> > 22 p.add(cf, cf, bytes);
> > 23 table.put(p);
> > 24 } finally {
> > 25 table.close();
> > 26 connection.close();
> > 27 }
> > 28 System.out.println("i=" + i);
> > 29 }
> > 30 }
> > 31 }
> >
> >
> > ....
> > 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
> > ensemble=localhost:2181
> > 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=localhost:2181 sessionTimeout=90000
> > watcher=hconnection-0x70dfa475, quorum=localhost:2181, baseZNode=/hbase
> > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > error)
> > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Socket connection established to localhost/
> > 127.0.0.1:2181, initiating session
> > 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server localhost/
> > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout =
> 40000
> > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner: Finished
> > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > STARTKEY => '', ENDKEY => ''}
> > 2014-12-12 11:26:10,406 INFO [main]
> > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x14a3fe0664b083a
> > 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
> > 0x14a3fe0664b083a closed
> > 2014-12-12 11:26:10,407 INFO [main-EventThread] zookeeper.ClientCnxn:
> > EventThread shut down
> > i=997
> > 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
> > ensemble=localhost:2181
> > 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=localhost:2181 sessionTimeout=90000
> > watcher=hconnection-0x22a7106, quorum=localhost:2181, baseZNode=/hbase
> > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server
> > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using
> SASL
> > (unknown error)
> > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Socket connection established to
> > localhost/0:0:0:0:0:0:0:1:2181, initiating session
> > 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server
> > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b, negotiated
> > timeout = 40000
> > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner: Finished
> > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > STARTKEY => '', ENDKEY => ''}
> > 2014-12-12 11:26:10,521 INFO [main]
> > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x14a3fe0664b083b
> > 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
> > 0x14a3fe0664b083b closed
> > 2014-12-12 11:26:10,521 INFO [main-EventThread] zookeeper.ClientCnxn:
> > EventThread shut down
> > i=998
> > 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
> > ensemble=localhost:2181
> > 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=localhost:2181 sessionTimeout=90000
> > watcher=hconnection-0x3822f407, quorum=localhost:2181, baseZNode=/hbase
> > 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > error)
> > 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Socket connection established to localhost/
> > 127.0.0.1:2181, initiating session
> > 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server localhost/
> > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout =
> 40000
> > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner: Finished
> > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > STARTKEY => '', ENDKEY => ''}
> > 2014-12-12 11:26:10,638 INFO [main]
> > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x14a3fe0664b083c
> > 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
> > 0x14a3fe0664b083c closed
> > 2014-12-12 11:26:10,639 INFO [main-EventThread] zookeeper.ClientCnxn:
> > EventThread shut down
> > i=999
> >
> >
> >
> >
> > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <
> serega.shey...@gmail.com
> > >
> > wrote:
> >
> > > Hi, I'm using CDH 5.2, 0.98
> > > I don't know how to use it correctly. I've just used this sample:
> > >
> > >
> >
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> > >
> > > HConnection connection = HConnectionManager.createConnection(config);
> > > HTableInterface table =
> > connection.getTable(TableName.valueOf("table1"));
> > > try {
> > > // Use the table as needed, for a single operation and a single
> thread
> > > } finally {
> > > table.close();
> > > connection.close();
> > > }
> > >
> > >
> > > Functional testing OK, performance fails :((((
> > >
> > >
> > >
> > > 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>:
> > > >
> > > > Does zk count go up on each request to the servlet? Which version of
> > > hbase
> > > > so can try on this end? Do you have some client-side log? Better if
> > you
> > > > cache the connection rather than make it each time since setup is
> > costly
> > > > but lets fix first problem first.
> > > > St.Ack
> > > >
> > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
> > > serega.shey...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi, I'm using HConnectionManager from java servlet
> > > > > Looks like it's leaking, all my zookeepers complains that there is
> > too
> > > > many
> > > > > connections from servlet hosts.
> > > > > Typical line from lZK log:
> > > > >
> > > > > oo many connections from /my.tomcat.server.com - max is 60
> > > > >
> > > > >
> > > > > Here is code sample
> > > > >
> > > > > public class BaseConnection {
> > > > >
> > > > > private static final Logger LOG =
> > > > > LoggerFactory.getLogger(BaseConnection.class);
> > > > >
> > > > > protected void close(HConnection connection){
> > > > > try{
> > > > > if(connection == null){
> > > > > return;
> > > > > }
> > > > > connection.close();
> > > > > }
> > > > > catch (Exception e){
> > > > > LOG.warn("Error while closing HConnection", e);
> > > > > }
> > > > > }
> > > > >
> > > > > protected void close(HTableInterface hTable){
> > > > > try{
> > > > > if(hTable == null){
> > > > > return;
> > > > > }
> > > > > hTable.close();
> > > > > }
> > > > > catch (Exception e){
> > > > > LOG.warn("Error while closing HTable", e);
> > > > > }
> > > > > }
> > > > > }
> > > > >
> > > > > sample PUT code from subclass:
> > > > >
> > > > > public void put(List<MyBean> entries){
> > > > > HConnection hConnection = null;
> > > > > HTableInterface hTable = null;
> > > > > try {
> > > > > List<Put> puts = new ArrayList<Put>(entries.size());
> > > > > for(MyBean myBean : entries){
> > > > > puts.add(new MyBeanSerDe().createPut(myBean));
> > > > > }
> > > > > hConnection =
> > > HConnectionManager.createConnection(configuration);
> > > > > hTable = hConnection.getTable(NAME_B);
> > > > > hTable.put(puts);
> > > > >
> > > > > }catch (Exception e){
> > > > > LOG.error("Error while doing bulk put", e);
> > > > > }finally{
> > > > > close(hTable);
> > > > > close(hConnection);
> > > > > }
> > > > > }
> > > > >
> > > >
> > >
> >
>
>
>
>
