Hm... confusing, So here is the code path: 1. Servlet has doPost method 2. HistoryController instantiated during each doPost request. HistoryController is Logic wrapper around low-level HBase persistence stuff Each doPost invokes:
final HistoryController getHistoryController(){ return new HistoryController(configuration: getHBaseConfiguration()) } getHistoryController().updateHistory(historyEntryVisitorId) 3. HistoryController has method: private HistoryEntryConnection getConnection(){ new HistoryEntryConnection(configuration) } so each updateHistory(historyEntryVisitorId) gets new low-level HistoryEntryConnection instance. HistoryEntryConnection is dead simple, I've shown it previously public void put(List<MyBean> entries){ HConnection hConnection = null; HTableInterface hTable = null; try { List<Put> puts = new ArrayList<Put>(entries.size()); for(MyBean myBean : entries){ puts.add(new MyBeanSerDe().createPut(myBean)); } hConnection = HConnectionManager.createConnection(configuration); hTable = hConnection.getTable(NAME_B); hTable.put(puts); }catch (Exception e){ LOG.error("Error while doing bulk put", e); }finally{ close(hTable); close(hConnection); } } Manager, Connection, HTable usage copy-pasted from javadoc. Nothing new. BTW, is there any possibility to limit quantity of opened connections in HConnectionManager internals ? I have several web-apps using HBase as backends. Maybe their cumulative pressure makes ZK go down... 2014-12-13 4:03 GMT+03:00 Stack <st...@duboce.net>: > > On Fri, Dec 12, 2014 at 11:45 AM, Serega Sheypak <serega.shey...@gmail.com > > > wrote: > > > > i have 10K doPost/doGet requests per second. > > > > How many concurrent threads going on in your tomcat? Is the Connection > shared amongst all threads? Perhaps you have > 60 concurrent connections > running at a time and each Connection has its own zk instance. Thread > dump? > > Post all the code? > > I tried multithreaded with 100 threads but still can't repro (see code > below). Next would be to get zk to dump concurrent connection count but > lets see what your answers above are first. > > St.Ack > > > public class TestConnnection { > > static class Putter extends Thread { > > Configuration c; > > Putter(final Configuration c, final String i) { > > super(i); > > this.c = c; > > } > > > @Override > > public void run() { > > try { > > for (int i = 0; i < 100; i++) { > > HConnection connection = HConnectionManager.createConnection(this.c > ); > > HTableInterface table = connection.getTable(TableName.valueOf( > "table1")); > > byte [] cf = Bytes.toBytes("t"); > > try { > > byte [] bytes = Bytes.toBytes(i); > > Put p = new Put(bytes); > > p.add(cf, cf, bytes); > > table.put(p); > > } finally { > > table.close(); > > connection.close(); > > } > > System.out.println(getName() + ", i=" + i); > > } > > } catch (Throwable t) { > > throw new RuntimeException(t); > > } > > } > > }; > > > public static void main(String[] args) throws IOException { > > Configuration config = HBaseConfiguration.create(); > > for (int i = 0; i < 100; i++) { > > Thread t = new Putter(config, "" + i); > > t.start(); > > } > > } > > } > > > > > > > Servlet is NOT single-threaded. each doPost/doGet > > invokes these lines (encapsulated in DAO): > > > > 16 HConnection connection = > > HConnectionManager.createConnection(config); > > 17 HTableInterface table = > > connection.getTable(TableName.valueOf("table1")); > > > > and > > > > 24 } finally { > > 25 table.close(); > > 26 connection.close(); > > 27 } > > > > I assumed that this static construction > > > > 16 HConnection connection = > > HConnectionManager.createConnection(config); > > > > correctly handles multi-threaded access somewhere deep inside. > > Right now I don't understand what do I do wrong. > > > > Try to wrap each your request in Runnable to emulate multi-threaded > > pressure on ZK. Your code is linear and mine is not, it's concurrent. > > > > Thanks.... > > > > > > > > > > 2014-12-12 22:28 GMT+03:00 Stack <st...@duboce.net>: > > > > > > I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted > your > > > code adding in a put for each cycle. I ran loop 1000 times and no > > > complaint from zk. > > > > > > Tell me more (Is servlet doing single-threaded model? A single > > > Configuration is being used or new ones are being created per servlet > > > invocation? > > > > > > Below is code and output. > > > > > > (For better perf, cache the connection) > > > St.Ack > > > > > > > > > 1 package org.apache.hadoop.hbase; > > > 2 > > > 3 import java.io.IOException; > > > 4 > > > 5 import org.apache.hadoop.conf.Configuration; > > > 6 import org.apache.hadoop.hbase.client.HConnection; > > > 7 import org.apache.hadoop.hbase.client.HConnectionManager; > > > 8 import org.apache.hadoop.hbase.client.HTableInterface; > > > 9 import org.apache.hadoop.hbase.client.Put; > > > 10 import org.apache.hadoop.hbase.util.Bytes; > > > 11 > > > 12 public class TestConnnection { > > > 13 public static void main(String[] args) throws IOException { > > > 14 Configuration config = HBaseConfiguration.create(); > > > 15 for (int i = 0; i < 1000; i++) { > > > 16 HConnection connection = > > > HConnectionManager.createConnection(config); > > > 17 HTableInterface table = > > > connection.getTable(TableName.valueOf("table1")); > > > 18 byte [] cf = Bytes.toBytes("t"); > > > 19 try { > > > 20 byte [] bytes = Bytes.toBytes(i); > > > 21 Put p = new Put(bytes); > > > 22 p.add(cf, cf, bytes); > > > 23 table.put(p); > > > 24 } finally { > > > 25 table.close(); > > > 26 connection.close(); > > > 27 } > > > 28 System.out.println("i=" + i); > > > 29 } > > > 30 } > > > 31 } > > > > > > > > > .... > > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper: > > > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper > > > ensemble=localhost:2181 > > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating > > client > > > connection, connectString=localhost:2181 sessionTimeout=90000 > > > watcher=hconnection-0x70dfa475, quorum=localhost:2181, baseZNode=/hbase > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)] > > > zookeeper.ClientCnxn: Opening socket connection to server localhost/ > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown > > > error) > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)] > > > zookeeper.ClientCnxn: Socket connection established to localhost/ > > > 127.0.0.1:2181, initiating session > > > 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)] > > > zookeeper.ClientCnxn: Session establishment complete on server > localhost/ > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout = > > 40000 > > > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner: > Finished > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1', > > > STARTKEY => '', ENDKEY => ''} > > > 2014-12-12 11:26:10,406 INFO [main] > > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper > > > sessionid=0x14a3fe0664b083a > > > 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session: > > > 0x14a3fe0664b083a closed > > > 2014-12-12 11:26:10,407 INFO [main-EventThread] zookeeper.ClientCnxn: > > > EventThread shut down > > > i=997 > > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper: > > > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper > > > ensemble=localhost:2181 > > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating > > client > > > connection, connectString=localhost:2181 sessionTimeout=90000 > > > watcher=hconnection-0x22a7106, quorum=localhost:2181, baseZNode=/hbase > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)] > > > zookeeper.ClientCnxn: Opening socket connection to server > > > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using > > SASL > > > (unknown error) > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)] > > > zookeeper.ClientCnxn: Socket connection established to > > > localhost/0:0:0:0:0:0:0:1:2181, initiating session > > > 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)] > > > zookeeper.ClientCnxn: Session establishment complete on server > > > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b, > negotiated > > > timeout = 40000 > > > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner: > Finished > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1', > > > STARTKEY => '', ENDKEY => ''} > > > 2014-12-12 11:26:10,521 INFO [main] > > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper > > > sessionid=0x14a3fe0664b083b > > > 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session: > > > 0x14a3fe0664b083b closed > > > 2014-12-12 11:26:10,521 INFO [main-EventThread] zookeeper.ClientCnxn: > > > EventThread shut down > > > i=998 > > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper: > > > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper > > > ensemble=localhost:2181 > > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating > > client > > > connection, connectString=localhost:2181 sessionTimeout=90000 > > > watcher=hconnection-0x3822f407, quorum=localhost:2181, baseZNode=/hbase > > > 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)] > > > zookeeper.ClientCnxn: Opening socket connection to server localhost/ > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown > > > error) > > > 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)] > > > zookeeper.ClientCnxn: Socket connection established to localhost/ > > > 127.0.0.1:2181, initiating session > > > 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)] > > > zookeeper.ClientCnxn: Session establishment complete on server > localhost/ > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout = > > 40000 > > > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner: > Finished > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1', > > > STARTKEY => '', ENDKEY => ''} > > > 2014-12-12 11:26:10,638 INFO [main] > > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper > > > sessionid=0x14a3fe0664b083c > > > 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session: > > > 0x14a3fe0664b083c closed > > > 2014-12-12 11:26:10,639 INFO [main-EventThread] zookeeper.ClientCnxn: > > > EventThread shut down > > > i=999 > > > > > > > > > > > > > > > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak < > > serega.shey...@gmail.com > > > > > > > wrote: > > > > > > > Hi, I'm using CDH 5.2, 0.98 > > > > I don't know how to use it correctly. I've just used this sample: > > > > > > > > > > > > > > https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html > > > > > > > > HConnection connection = > HConnectionManager.createConnection(config); > > > > HTableInterface table = > > > connection.getTable(TableName.valueOf("table1")); > > > > try { > > > > // Use the table as needed, for a single operation and a single > > thread > > > > } finally { > > > > table.close(); > > > > connection.close(); > > > > } > > > > > > > > > > > > Functional testing OK, performance fails :(((( > > > > > > > > > > > > > > > > 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>: > > > > > > > > > > Does zk count go up on each request to the servlet? Which version > of > > > > hbase > > > > > so can try on this end? Do you have some client-side log? Better > if > > > you > > > > > cache the connection rather than make it each time since setup is > > > costly > > > > > but lets fix first problem first. > > > > > St.Ack > > > > > > > > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak < > > > > serega.shey...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi, I'm using HConnectionManager from java servlet > > > > > > Looks like it's leaking, all my zookeepers complains that there > is > > > too > > > > > many > > > > > > connections from servlet hosts. > > > > > > Typical line from lZK log: > > > > > > > > > > > > oo many connections from /my.tomcat.server.com - max is 60 > > > > > > > > > > > > > > > > > > Here is code sample > > > > > > > > > > > > public class BaseConnection { > > > > > > > > > > > > private static final Logger LOG = > > > > > > LoggerFactory.getLogger(BaseConnection.class); > > > > > > > > > > > > protected void close(HConnection connection){ > > > > > > try{ > > > > > > if(connection == null){ > > > > > > return; > > > > > > } > > > > > > connection.close(); > > > > > > } > > > > > > catch (Exception e){ > > > > > > LOG.warn("Error while closing HConnection", e); > > > > > > } > > > > > > } > > > > > > > > > > > > protected void close(HTableInterface hTable){ > > > > > > try{ > > > > > > if(hTable == null){ > > > > > > return; > > > > > > } > > > > > > hTable.close(); > > > > > > } > > > > > > catch (Exception e){ > > > > > > LOG.warn("Error while closing HTable", e); > > > > > > } > > > > > > } > > > > > > } > > > > > > > > > > > > sample PUT code from subclass: > > > > > > > > > > > > public void put(List<MyBean> entries){ > > > > > > HConnection hConnection = null; > > > > > > HTableInterface hTable = null; > > > > > > try { > > > > > > List<Put> puts = new ArrayList<Put>(entries.size()); > > > > > > for(MyBean myBean : entries){ > > > > > > puts.add(new MyBeanSerDe().createPut(myBean)); > > > > > > } > > > > > > hConnection = > > > > HConnectionManager.createConnection(configuration); > > > > > > hTable = hConnection.getTable(NAME_B); > > > > > > hTable.put(puts); > > > > > > > > > > > > }catch (Exception e){ > > > > > > LOG.error("Error while doing bulk put", e); > > > > > > }finally{ > > > > > > close(hTable); > > > > > > close(hConnection); > > > > > > } > > > > > > } > > > > > > > > > > > > > > > > > > > > >