Did you check how many open connections each ZK server has? I my hypothesis is that you have ZK connection leaking and ZK server starts to drop connection to prevent DDoS attack since you hit limit for opened connections.
2015-02-26 22:15 GMT+03:00 Nick Dimiduk <ndimi...@gmail.com>: > Can you tell when these WARN messages are produced? Is it related to the > creation of the connection object or one of the HTable instances? > > On Thu, Feb 26, 2015 at 7:27 AM, Marcelo Valle (BLOOMBERG/ LONDON) < > mvallemil...@bloomberg.net> wrote: > > > Nick, > > > > I tried what you suggested, 1 HConnection and 1 Configuration for the > > entire app: > > > > this.config = HBaseConfiguration.create(); > > this.connection = HConnectionManager.createConnection(config); > > > > And Threaded pooled HTableInterfaces: > > > > final HConnection lconnection = this.connection; > > this.tlTable = new ThreadLocal<HTableInterface>() { > > @Override > > protected HTableInterface initialValue() { > > try { > > return lconnection.getTable("HBaseSerialWritesPOC"); > > // return new HTable(tlConfig.get(), > > // "HBaseSerialWritesPOC"); > > } catch (IOException e) { > > throw new RuntimeException(e); > > } > > } > > }; > > > > I started getting this error in my application: > > > > 2015-02-26 10:23:17,833 INFO [main-SendThread(xxx)] zookeeper.ClientCnxn > > (ClientCnxn.java:logStartConnect(966)) - Opening socket connection to > > server xxx. Will not attempt to authenticate using SASL (unknown error) > > 2015-02-26 10:23:17,834 INFO [main-SendThread(xxx)] zookeeper.ClientCnxn > > (ClientCnxn.java:primeConnection(849)) - Socket connection established to > > xxx, initiating session > > 2015-02-26 10:23:17,836 WARN [main-SendThread(xxx)] zookeeper.ClientCnxn > > (ClientCnxn.java:run(1089)) - Session 0x0 for server xxx, unexpected > error, > > closing socket connection and attempting reconnect > > java.io.IOException: Connection reset by peer > > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > > at sun.nio.ch.IOUtil.read(IOUtil.java:192) > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > > at > > > org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68) > > at > > > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355) > > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) > > > > > > -Marcelo > > > > From: ndimi...@gmail.com > > Subject: Re: HBase connection pool > > > > Okay, looks like you're using a implicitly managed connection. It should > > be fine to share a single config instance across all threads. The > advantage > > of HTablePool over this approach is that the number of HTables would be > > managed independently from the number of Threads. This may or not be a > > concern for you, based on your memory requirements, &c. In your case, > > you're not specifying an ExecutorService per HTable, so the HTable > > instances will be relatively light weight. Each table will manage it's > own > > write buffer, which can be shared by multiple threads when autoFlush is > > disabled and HTablePool is used. This may or may not be desirable, > > depending on your use-case. > > > > For what it's worth, HTablePool is marked deprecated in 1.0, will likely > > be removed in 2.0. To "future proof" this code, I would move to a single > > shared HConnection for the whole application, and a thread-local HTable > > created from/with that connection. > > > > -n > > > > On Wed, Feb 25, 2015 at 10:53 AM, Marcelo Valle (BLOOMBERG/ LONDON) < > > mvallemil...@bloomberg.net> wrote: > > > >> Hi Nick, > >> > >> I am using HBase version 0.96, I sent the link from version 0.94 because > >> I haven't found the java API docs for 0.96, sorry about that. > >> I have created the HTable directly from the config object, as follows: > >> > >> this.tlConfig = new ThreadLocal<Configuration>() { > >> > >> @Override > >> protected Configuration initialValue() { > >> return HBaseConfiguration.create(); > >> } > >> }; > >> this.tlTable = new ThreadLocal<HTable>() { > >> @Override > >> protected HTable initialValue() { > >> try { > >> return new HTable(tlConfig.get(), "HBaseSerialWritesPOC"); > >> } catch (IOException e) { > >> throw new RuntimeException(e); > >> } > >> } > >> }; > >> > >> I am now sure if the Configuration object should be 1 per thread as > well, > >> maybe I could share this one? > >> > >> So, just to clarify, would I get any advantage using HTablePool object > >> instead of ThreadLocal<HTable> as I did? > >> > >> -Marcelo > >> > >> From: ndimi...@gmail.com > >> Subject: Re: HBase connection pool > >> > >> Hi Marcelo, > >> > >> First thing, to be clear, you're working with a 0.94 release? The reason > >> I ask is we've been doing some work in this area to improve things, so > >> semantics may be slightly different between 0.94, 0.98, and 1.0. > >> > >> How are you managing the HConnection object (or are you)? How are you > >> creating your HTable instances? These will determine how the connection > is > >> obtained and used in relation to HTables. > >> > >> In general, multiple HTable instances connected to tables in the same > >> cluster should be sharing the same HConnection instance. This is handled > >> explicitly when you manage your own HConnection and HTables (i.e., > >> HConnection conn = ... ; HTable t = new HTable(TABLE_NAME, conn); ) It's > >> handled implicitly when you construct via Configuration objects (HTable > t = > >> new HTable(conf, TABLE_NAME); ) This implicit option is going away in > >> future versions. > >> > >> HTable is not safe for concurrent access because of how the write path > is > >> implemented (at least; there may be other portions that I'm not as > familiar > >> with). You should be perfectly fine to have an HTable per thread in a > >> ThreadLocal. > >> > >> -n > >> > >> On Wed, Feb 25, 2015 at 9:41 AM, Marcelo Valle (BLOOMBERG/ LONDON) < > >> mvallemil...@bloomberg.net> wrote: > >> > >>> In HBase API, does 1 HTable object means 1 connection to each region > >>> server (just for 1 table)? > >>> > >>> The docs say ( > >>> > http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HTable.html > >>> ): > >>> "This class is not thread safe for reads nor write." > >>> > >>> I got confused, as I saw there is a HTablePool class, but it's only for > >>> a table as well, can't connections be reused for more than 1 table? > >>> > >>> In my java application, I used ThreadLocal variables > >>> (ThreadLocal<HTable>) to create an HTable variable per thread. If I do > >>> several operations on each thread, I should still use the same > connection, > >>> right? > >>> > >>> []s > >> > >> > >> > >> > > > > >