Re: Splitting causes HBase to crash

2016-05-13 Thread Gunnar Tapper
2016-05-13 11:56:00,000 WARN org.apache.hadoop.hbase.wal.WALSplitter: Found old edits file. It could be the result of a previous failed split attempt. Deleting hdfs://ip-172-31-50-109.ec2.internal:8020/hbase/data/default/TRAFODION.PLACEHOLDER.COMPANYID_10027749/287b1c6e726e9feaa84a5ab07490f9ff/reco

Re: Splitting causes HBase to crash

2016-05-13 Thread Ted Yu
bq. 2016-05-13 11:56:52,763 WARN org.apache.hadoop.hbase.master.SplitLogManager: error while splitting logs in [hdfs://ip-172-31-50-109.ec2.internal:8020/hbase/WALs/ip- 172-31-54-241.ec2.internal,60020,1463123941413-splitting] installed = 1 but only 0 done Looks like WAL splitting was slow or stal

Re: Splitting causes HBase to crash

2016-05-13 Thread Gunnar Tapper
Some more info. I remove /hbase using hbase zkcli rmr /hbaase. The log messages I provided occurred after that. This is a HA configuration with two HMasters. After sitting in an initializing state for a long time, I end up with: hbase(main):001:0> list TABLE ERROR: Can't get master address fro

Re: Splitting causes HBase to crash

2016-05-13 Thread Gunnar Tapper
Hi Ted, Two of the three logs just logs successful connection attempts. The third log shows: 2016-05-13 11:37:00,372 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x154a8fa6dbc06eb type:setData cxid:0x1be8 zxid:0xe0004d100 txntype:

Re: Splitting causes HBase to crash

2016-05-13 Thread Ted Yu
bq. Unable to list children of znode /hbase/region-in-transition Looks like there might be some problem with zookeeper quorum. Can you check zookeeper server logs ? Cheers On Fri, May 13, 2016 at 12:17 AM, Gunnar Tapper wrote: > Hi, > > I'm doing some development testing with Apache Trafodion

Re: Splitting causes HBase to crash

2016-05-13 Thread Samir Ahmic
In most cases like can try following: 1. stop hbase 2. clean hbase zookeeper data (hbase zkcli --> rmr /hbase) 3. start hbase Regards Samir On Fri, May 13, 2016 at 9:17 AM, Gunnar Tapper wrote: > Hi, > > I'm doing some development testing with Apache Trafodion running > HBase Version 1.0.0-cdh

Splitting causes HBase to crash

2016-05-13 Thread Gunnar Tapper
Hi, I'm doing some development testing with Apache Trafodion running HBase Version 1.0.0-cdh5.4.5. All of a sudden, HBase has started to crash. First, it could not be recovered until I changed hbase_master_distributed_log_splitting to false. At that point, HBase restarted and sat happily idling f