Hi Sorry if my reply mislead you. I meant to see the GC logs that should give you an idea of if Full GC happened.
Regards Ram > -----Original Message----- > From: Xiang Hua [mailto:bea...@gmail.com] > Sent: Monday, October 15, 2012 12:42 PM > To: user@hbase.apache.org > Subject: Re: hmaster and regionserver died > > We will check the zk log. > > On Monday, October 15, 2012, Ramkrishna.S.Vasudevan wrote: > > > Check your GC configurations. Seems to that a Full GC has happened > and the > > Zookeeper thought that to be session expiry. > > > > Regards > > Ram > > > > > -----Original Message----- > > > From: Xiang Hua [mailto:bea...@gmail.com] > > > Sent: Saturday, October 13, 2012 6:20 PM > > > To: user@hbase.apache.org > > > Subject: hmaster and regionserver died > > > > > > Hi, > > > the HMaster died as well as regionservers, below is hmaster's > log. > > > could > > > you please find what's problem? > > > > > > > > > 2012-10-12 00:14:19,444 INFO org.apache.zookeeper.ClientCnxn: > Socket > > > connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/ > > > 10.20.16.34:2181, initiating session > > > 2012-10-12 00:14:19,520 INFO org.apache.zookeeper.ClientCnxn: > Session > > > establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase- > stor- > > > 3/ > > > 10.20.16.34:2181, sessionid = 0x139c539bc090002, negotiated timeout > = > > > 40000 > > > 2012-10-12 00:14:23,738 INFO org.apache.zookeeper.ClientCnxn: > Client > > > session timed out, have not heard from server in 15046ms for > sessionid > > > 0x239c539ba630001, closing socket connection and attempting > reconnect > > > 2012-10-12 00:14:24,246 INFO org.apache.zookeeper.ClientCnxn: > Opening > > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/ > > > 10.20.16.33:2181 > > > 2012-10-12 00:14:25,173 INFO org.apache.zookeeper.ClientCnxn: > Client > > > session timed out, have not heard from server in 15245ms for > sessionid > > > 0x139c539bc090003, closing socket connection and attempting > reconnect > > > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: > Opening > > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/ > > > 10.20.16.33:2181 > > > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: > Socket > > > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/ > > > 10.20.16.33:2181, initiating session > > > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: > > > EventThread > > > shut down > > > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: > Unable to > > > reconnect to ZooKeeper service, session 0x139c539bc090003 has > expired, > > > closing socket connection > > > 2012-10-12 00:14:27,247 INFO org.apache.zookeeper.ClientCnxn: > Socket > > > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/ > > > 10.20.16.33:2181, initiating session > > > 2012-10-12 00:14:27,248 WARN org.apache.zookeeper.ClientCnxn: > Session > > > 0x239c539ba630001 for server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor- > 2/ > > > 10.20.16.33:2181, unexpected error, closing socket connection and > > > attempting reconnect > > > java.io.IOException: Connection reset by peer > > > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218) > > > at sun.nio.ch.IOUtil.read(IOUtil.java:186) > > > at > sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359) > > > at > > > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:859) > > > at > > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157) > > > 2012-10-12 00:14:28,026 INFO org.apache.zookeeper.ClientCnxn: > Opening > > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/ > > > 10.20.16.34:2181 > > > 2012-10-12 00:14:41,359 INFO org.apache.zookeeper.ClientCnxn: > Client > > > session timed out, have not heard from server in 14007ms for > sessionid > > > 0x239c539ba630001, closing socket connection and attempting > reconnect > > > 2012-10-12 00:14:41,592 INFO org.apache.zookeeper.ClientCnxn: > Opening > > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/ > > > 10.20.16.32:2181 > > > 2012-10-12 00:14:46,186 INFO org.apache.zookeeper.ClientCnxn: > Client > > > session timed out, have not heard from server in 26666ms for > sessionid > >
