Hi Ted, As mentioned in that HBASE-14926 it is fixed in HBase 0.98.17. I have googled it for both its source code and binary but I didn't find one. can you guide me where I can find Hbase 0.98.17 source code or binary package
Thanks On Tue, Jan 12, 2016 at 12:57 AM, Ted Yu <yuzhih...@gmail.com> wrote: > I took a look at the stack trace you posted. > > Some thrift worker was in the state shown in HBASE-14926 > > I wonder if the fix for HBASE-14926 would help in your case. > > Cheers > > On Mon, Jan 11, 2016 at 5:11 AM, Rajeshkumar J < > rajeshkumarit8...@gmail.com> > wrote: > > > Hi Ted, > > > > Plz ignore my previous mail as the log is taken when hbase is working > > fine. Please find the logs below when Hbase didn't return any records > > > > http://pastebin.com/APYjiGSP > > > > Thanks > > > > On Mon, Jan 11, 2016 at 11:19 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > > When you search for 'java jstack tutorial' using google, you would > find a > > > lot of tutorials. > > > > > > You can use 'ps aux' command to search for the following class: > > > org.apache.hadoop.hbase.thrift.ThriftServer > > > > > > Apply jstack on the corresponding process id. > > > > > > Cheers > > > > > > On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J < > > > rajeshkumarit8...@gmail.com> > > > wrote: > > > > > > > Hi Ted, > > > > > > > > I don't know how how to take stacktrace using jstack. But i tried > > > using > > > > some commands but it fails. can you help me in this? > > > > > > > > Thanks > > > > > > > > On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu <yuzhih...@gmail.com> > wrote: > > > > > > > > > What I meant was to capture stack trace of the thrift server (using > > > > jstack > > > > > command) when your query is running. > > > > > > > > > > This would give us some more clue as to the state thrift server was > > in > > > > when > > > > > the problem happened. > > > > > > > > > > Cheers > > > > > > > > > > On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J < > > > > > rajeshkumarit8...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Ted, > > > > > > > > > > > > This is what I found in the Hbase thrift server logs > > > > > > > > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:java.library.path=;C:\Hadoop\bin > > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:java.io.tmpdir=C:\Windows\TEMP\ > > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:java.compiler=<NA> > > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:os.name=Windows Server 2012 > > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:os.arch=amd64 > > > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:os.version=6.2 > > > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:user.name=xxxx > > > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:user.home=C:\ > > > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > Client > > > > > > environment:user.dir=C:\HBase\bin > > > > > > 2016-01-09 06:27:43,142 INFO [thrift-worker-0] > > zookeeper.ZooKeeper: > > > > > > Initiating client connection, > > > > > > connectString=hostname:2181,hostname:2181,hostname:2181 > > > > > > sessionTimeout=90000 watcher=hconnection-0x45e35460x0, > > > > > > quorum=hostname:2181,hostname:2181,hostname:2181, > baseZNode=/hbase > > > > > > 2016-01-09 06:27:43,339 INFO > > > > [thrift-worker-0-SendThread(hostname:2181)] > > > > > > zookeeper.ClientCnxn: Opening socket connection to server > > > > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using > > SASL > > > > > > (unknown error) > > > > > > 2016-01-09 06:27:43,342 INFO > > > > [thrift-worker-0-SendThread(hostname:2181)] > > > > > > zookeeper.ClientCnxn: Socket connection established to > > > > > > hostname/xx.xx.x.xx:2181, initiating session > > > > > > 2016-01-09 06:27:43,365 INFO > > > > [thrift-worker-0-SendThread(hostname:2181)] > > > > > > zookeeper.ClientCnxn: Session establishment complete on server > > > > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, > negotiated > > > > > timeout > > > > > > = 40000 > > > > > > 2016-01-09 07:14:10,607 INFO > > > > > > [hostname,60000,1452337815006_splitLogManager__ChoreService_1] > > > > > > master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager > > Timeout > > > > > > Monitor missed its start time > > > > > > 2016-01-09 07:16:43,511 INFO > > > > > > [B.defaultRpcServer.handler=8,queue=2,port=60000] > master.HMaster: > > > > > > Client=Administrator/null create 'chk2', {NAME => 'fa', > > > > > DATA_BLOCK_ENCODING > > > > > > => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', > VERSIONS > > > => > > > > > '1', > > > > > > COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', > > > > > > KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => > > > > > 'false', > > > > > > BLOCKCACHE => 'true'} > > > > > > 2016-01-09 07:16:44,121 INFO [ProcedureExecutorThread-1] > > > > > > procedure2.ProcedureExecutor: Rolledback procedure > > > CreateTableProcedure > > > > > > (table=chk2) user=Administrator (auth:SIMPLE) id=2 > state=ROLLEDBACK > > > > > > exec-time=462msec > > > > exception=org.apache.hadoop.hbase.TableExistsException: > > > > > > chk2 > > > > > > 2016-01-09 07:49:55,574 INFO [ConnectionCache_ChoreService_1] > > > > > > client.ConnectionManager$HConnectionImplementation: Closing > > zookeeper > > > > > > sessionid=0x1522613893e0004 > > > > > > 2016-01-09 07:49:55,629 INFO [ConnectionCache_ChoreService_1] > > > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed > > > > > > 2016-01-09 07:49:55,661 INFO [thrift-worker-0-EventThread] > > > > > > zookeeper.ClientCnxn: EventThread shut down > > > > > > 2016-01-09 18:40:29,469 INFO [thrift-worker-1] > > > > > > zookeeper.RecoverableZooKeeper: Process > > > > identifier=hconnection-0x591287f8 > > > > > > connecting to ZooKeeper > > > > > ensemble=hostname:2181,hostname:2181,hostname:2181 > > > > > > 2016-01-09 18:40:29,482 INFO [thrift-worker-1] > > zookeeper.ZooKeeper: > > > > > > Initiating client connection, > > > > > > connectString=hostname:2181,hostname:2181,hostname:2181 > > > > > > sessionTimeout=90000 watcher=hconnection-0x591287f80x0, > > > > > > quorum=hostname:2181,hostname:2181,hostname:2181, > baseZNode=/hbase > > > > > > 2016-01-09 18:40:29,553 INFO > > > > [thrift-worker-1-SendThread(hostname:2181)] > > > > > > zookeeper.ClientCnxn: Opening socket connection to server > > > > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using > > SASL > > > > > > (unknown error) > > > > > > 2016-01-09 18:40:29,559 INFO > > > > [thrift-worker-1-SendThread(hostname:2181)] > > > > > > zookeeper.ClientCnxn: Socket connection established to > > > > > > hostname/xx.xx.x.xx:2181, initiating session > > > > > > 2016-01-09 18:40:29,569 INFO > > > > [thrift-worker-1-SendThread(hostname:2181)] > > > > > > zookeeper.ClientCnxn: Session establishment complete on server > > > > > > hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008, > negotiated > > > > > timeout > > > > > > = 40000 > > > > > > 2016-01-09 18:54:25,558 INFO [ConnectionCache_ChoreService_1] > > > > > > client.ConnectionManager$HConnectionImplementation: Closing > > zookeeper > > > > > > sessionid=0x35226137e300008 > > > > > > 2016-01-09 18:54:25,566 INFO [ConnectionCache_ChoreService_1] > > > > > > zookeeper.ZooKeeper: Session: 0x35226137e300008 closed > > > > > > 2016-01-09 18:54:25,578 INFO [thrift-worker-1-EventThread] > > > > > > zookeeper.ClientCnxn: EventThread shut down > > > > > > 2016-01-09 20:05:42,469 INFO [thrift-worker-13] > > > > > > zookeeper.RecoverableZooKeeper: Process > > > > identifier=hconnection-0x6305dbeb > > > > > > connecting to ZooKeeper > > > > > ensemble=hostname:2181,hostname:2181,hostname:2181 > > > > > > 2016-01-09 20:05:42,476 INFO [thrift-worker-13] > > zookeeper.ZooKeeper: > > > > > > Initiating client connection, > > > > > > connectString=hostname:2181,hostname:2181,hostname:2181 > > > > > > sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0, > > > > > > quorum=hostname:2181,hostname:2181,hostname:2181, > baseZNode=/hbase > > > > > > 2016-01-09 20:05:42,507 INFO > > > > > [thrift-worker-13-SendThread(hostname:2181)] > > > > > > zookeeper.ClientCnxn: Opening socket connection to server > > > > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using > > SASL > > > > > > (unknown error) > > > > > > 2016-01-09 20:05:42,510 INFO > > > > > [thrift-worker-13-SendThread(hostname:2181)] > > > > > > zookeeper.ClientCnxn: Socket connection established to > > > > > > hostname/xx.xx.x.xx:2181, initiating session > > > > > > 2016-01-09 20:05:42,708 INFO > > > > > [thrift-worker-13-SendThread(hostname:2181)] > > > > > > zookeeper.ClientCnxn: Session establishment complete on server > > > > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, > negotiated > > > > > timeout > > > > > > = 40000 > > > > > > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1] > > > > > > client.ConnectionManager$HConnectionImplementation: Closing > > zookeeper > > > > > > sessionid=0x1522613893e0005 > > > > > > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1] > > > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Rajeshkumar J > > > > > > > > > > > > On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yuzhih...@gmail.com> > > > wrote: > > > > > > > > > > > > > Can you take a stack trace of the thrift server and pastebin > the > > > > trace > > > > > ? > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J < > > > > > > > rajeshkumarit8...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > I am connecting via Hbase thrift server to access records in > > > Hbase > > > > > and > > > > > > I > > > > > > > am > > > > > > > > doing this in C# and i am using range scan to get records > from > > > the > > > > > > table. > > > > > > > > This seems to be working for few hours but after few hours it > > > > didn't > > > > > > > > returned any records and then i searched hbase logs i found > > this > > > > > > > > > > > > > > > > 2016-01-04 00:44:54,880 INFO [ConnectionCleaner] > > > > > > > > client.HConnectionManager$HConnectionImplementation: Closing > > > > > zookeeper > > > > > > > > sessionid=0x3520b1721650006 > > > > > > > > 2016-01-04 00:44:54,891 INFO [ConnectionCleaner] > > > > > zookeeper.ZooKeeper: > > > > > > > > Session: 0x3520b1721650006 closed > > > > > > > > 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread] > > > > > > > > zookeeper.ClientCnxn: EventThread shut down > > > > > > > > > > > > > > > > And I am using hbase 0.98.13 > > > > > > > > > > > > > > > > I have asked this question before and some user suggested me > to > > > > > upgrade > > > > > > > to > > > > > > > > new version so i have upgraded HBase to 1.1.2. But now it > also > > it > > > > > > didn't > > > > > > > > return any records but different log generated as shown below > > > > > > > > > > > > > > > > 2016-01-09 20:05:42,708 INFO > > > > > [thrift-worker-13-SendThread(xxxx:2181)] > > > > > > > > zookeeper.ClientCnxn: Session establishment complete on > server > > > > > > > > xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, > negotiated > > > > > > timeout = > > > > > > > > 40000 > > > > > > > > 2016-01-09 20:15:45,541 INFO > [ConnectionCache_ChoreService_1] > > > > > > > > client.ConnectionManager$HConnectionImplementation: Closing > > > > zookeeper > > > > > > > > sessionid=0x1522613893e0005 > > > > > > > > 2016-01-09 20:15:45,567 INFO > [ConnectionCache_ChoreService_1] > > > > > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed > > > > > > > > 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread] > > > > > > > > zookeeper.ClientCnxn: EventThread shut down > > > > > > > > > > > > > > > > Can any one help me in this? > > > > > > > > > > > > > > > > Regards, > > > > > > > > Rajeshkumar J > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
