HBase strange behaviour

2015-07-06 Thread Akmal Abbasov
Hi all, I have a strange behaviour in my HBase cluster. I have 5 rs and 2 masters. One of the rs stopped working, restart didn’t worked, and I had to delete and recreate it. But when this rs have stopped, the cluster also stopped functioning. There were a lot of inconsistencies. When I recreated

Re: HBase strange behaviour

2015-07-06 Thread Ted Yu
bq. I had to delete and recreate it What error(s) did you get when trying to restart the region server ? Have you checked its log files ? bq. start balancer manually, but it returned false Can you check master log around this time ? If there was region in transition, balancer wouldn't balance.

Re: HBase strange behaviour

2015-07-06 Thread Akmal Abbasov
> What error(s) did you get when trying to restart the region server ? Have > you checked its log files ? it was a VM, and I was not able to access it any more, I can’t login to it. Restarting several times didn’t helped. > Can you check master log around this time ? If there was region in > tra

Re: HBase strange behaviour

2015-07-06 Thread Ted Yu
Do you see in the master log something similar to the following ? master.HMaster: Not running balancer because 1 region(s) in transition You can search backwards for balancer / assignment related logs. Cheers On Mon, Jul 6, 2015 at 8:49 AM, Akmal Abbasov wrote: > > What error(s) did you get w

Unable to start Hive Cli

2015-07-06 Thread Anil Bhat
Hi, I am getting following error while trying to get into hive cli. Logging initialized using configuration in jar:file:/usr/local/Cellar/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties [ERROR] Terminal initialization failed; falling back to unsupported java.lang.Incompatib

Re: Unable to start Hive Cli

2015-07-06 Thread Anil Bhat
Please Ignore, wrong mailing list On Tuesday, July 7, 2015 12:28 AM, Anil Bhat wrote: Hi, I am getting following error while trying to get into hive cli. Logging initialized using configuration in jar:file:/usr/local/Cellar/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log

Re: Ask for troubleshooting advice: RegionServer is aborted when flushing memstore.

2015-07-06 Thread Ted Yu
Have you looked at the recently fixed bug HBASE-13329 ? >From stack trace, there is big similarity. The fix would be in the upcoming 1.2.0 release. FYI On Fri, Jul 3, 2015 at 11:30 PM, Dinh Duong Mai wrote: > Dear HBase team, > > I reported an issue to HBase issue tracking system (HBASE-14018

Re: HBase strange behaviour

2015-07-06 Thread Akmal Abbasov
> Do you see in the master log something similar to the following ? > > master.HMaster: Not running balancer because 1 region(s) in transition yes, I have several of them, but all of them were 3 days ago. I check the ‘ritCount’ metric, and it is 0, also I checked the /hbase/region-in-transition

Re: HBase strange behaviour

2015-07-06 Thread Ted Yu
Have you run the following command in hbase shell ? balance_switch true Cheers On Mon, Jul 6, 2015 at 12:16 PM, Akmal Abbasov wrote: > > Do you see in the master log something similar to the following ? > > > > master.HMaster: Not running balancer because 1 region(s) in transition > yes, I have

Re: RegionServer 60030 Show All RPC Handler Task is empty

2015-07-06 Thread Louis Hust
Any idea 2015-07-03 14:17 GMT+08:00 Louis Hust : > Hi, stack, > > It seems that after one day, the " Show All RPC Handler Tasks" tab go > back to empty. > Confused!! > > > 2015-07-02 14:50 GMT+08:00 Stack : > >> On Wed, Jul 1, 2015 at 9:18 PM, Louis Hust wrote: >> >> > Hi, all >> > >> > I restar

Re: RegionServer 60030 Show All RPC Handler Task is empty

2015-07-06 Thread anil gupta
Did you check logs of those RS? Anything unusual over there? Maybe, you can share logs? On Mon, Jul 6, 2015 at 8:51 PM, Louis Hust wrote: > Any idea > > 2015-07-03 14:17 GMT+08:00 Louis Hust : > > > Hi, stack, > > > > It seems that after one day, the " Show All RPC Handler Tasks" tab go > > back

Re: RegionServer 60030 Show All RPC Handler Task is empty

2015-07-06 Thread Esteban Gutierrez
Louis, before you see this problem, how many tasks are reported by the debug dump servlet? e.g what do you see when you run "curl -s http://rs:port/dump | grep Task: | wc -l" ? Have you tried to dump directly the JSON object "curl -s http://rs:port/rs-status?format=json&filter=handler"; ? thanks,