Re: Dead loop for batch put when get WrongRegionException

2015-07-23 Thread Louis Hust
log output as myself when the the bug happens. However, this only occurs when you are using 0.98.X version. 1.0 and above do not have this problem. On Thu, Jul 23, 2015 at 2:55 PM Louis Hust louis.h...@gmail.com wrote: It seems that the HBASE-13896 https://issues.apache.org/jira/browse/HBASE

Re: Dead loop for batch put when get WrongRegionException

2015-07-23 Thread Louis Hust
://issues.apache.org/jira/browse/HBASE-13896 On Thu, Jul 23, 2015 at 11:17 AM Louis Hust louis.h...@gmail.com wrote: Hi ,all We are using batch put to insert rows, and sometimes get the following WARN in the region server log: log 2015-07-23 10:08:49,684 WARN

Dead loop for batch put when get WrongRegionException

2015-07-22 Thread Louis Hust
Hi ,all We are using batch put to insert rows, and sometimes get the following WARN in the region server log: log 2015-07-23 10:08:49,684 WARN [B.defaultRpcServer.handler=5,queue=5,port=60020] regionserver.HRegion: Failed getting lock in batch put, row=BHXYHZFIHHR3ECON10100215072399

Re: Scan got exception

2015-07-09 Thread Louis Hust
Yes, i will report as a bug 2015-07-09 0:48 GMT+08:00 Vladimir Rodionov vladrodio...@gmail.com: Is this issue reproducible? If - yes, then please submit a bug. -Vlad On Wed, Jul 8, 2015 at 2:32 AM, Louis Hust louis.h...@gmail.com wrote: Any idea? 2015-07-01 9:50 GMT+08:00 Louis Hust

Re: Scan got exception

2015-07-09 Thread Louis Hust
I report a bug: https://issues.apache.org/jira/browse/HBASE-14046 2015-07-09 16:27 GMT+08:00 Louis Hust louis.h...@gmail.com: Yes, i will report as a bug 2015-07-09 0:48 GMT+08:00 Vladimir Rodionov vladrodio...@gmail.com: Is this issue reproducible? If - yes, then please submit a bug

Re: RegionServer 60030 Show All RPC Handler Task is empty

2015-07-09 Thread Louis Hust
:32 AM, Louis Hust louis.h...@gmail.com wrote: Hi esteban, For the dump url get the following out: Tasks: === Executors: For the json format, got the following output: Shell curl http://:60030/rs-status

Re: RegionServer 60030 Show All RPC Handler Task is empty

2015-07-08 Thread Louis Hust
://rs:port/rs-status?format=jsonfilter=handler; ? thanks, esteban. -- Cloudera, Inc. On Mon, Jul 6, 2015 at 8:51 PM, Louis Hust louis.h...@gmail.com wrote: Any idea 2015-07-03 14:17 GMT+08:00 Louis Hust louis.h...@gmail.com: Hi, stack, It seems that after one day, the Show All

Re: Scan got exception

2015-07-08 Thread Louis Hust
Any idea? 2015-07-01 9:50 GMT+08:00 Louis Hust louis.h...@gmail.com: So the cdh5.2.0 is patched with HBASE-11678 ? 2015-07-01 6:43 GMT+08:00 Stack st...@duboce.net: I checked Vladimir and 5.2.0 is the first release with the necessary HBASE-11678 BucketCache ramCache fills heap after

Re: RegionServer 60030 Show All RPC Handler Task is empty

2015-07-06 Thread Louis Hust
Any idea 2015-07-03 14:17 GMT+08:00 Louis Hust louis.h...@gmail.com: Hi, stack, It seems that after one day, the Show All RPC Handler Tasks tab go back to empty. Confused!! 2015-07-02 14:50 GMT+08:00 Stack st...@duboce.net: On Wed, Jul 1, 2015 at 9:18 PM, Louis Hust louis.h

Re: RegionServer 60030 Show All RPC Handler Task is empty

2015-07-03 Thread Louis Hust
Hi, stack, It seems that after one day, the Show All RPC Handler Tasks tab go back to empty. Confused!! 2015-07-02 14:50 GMT+08:00 Stack st...@duboce.net: On Wed, Jul 1, 2015 at 9:18 PM, Louis Hust louis.h...@gmail.com wrote: Hi, all I restart a regionserver and do a balanceswitch

Re: RegionServer 60030 Show All RPC Handler Task is empty

2015-07-01 Thread Louis Hust
Hi, all I restart a regionserver and do a balanceswitch manually, the task seems ok 2015-06-30 15:38 GMT+08:00 Louis Hust louis.h...@gmail.com: Any ideas? 2015-06-30 9:53 GMT+08:00 Louis Hust louis.h...@gmail.com: Anybody can help? 2015-06-26 23:05 GMT+08:00 Louis Hust louis.h

Re: RegionServer 60030 Show All RPC Handler Task is empty

2015-06-30 Thread Louis Hust
Any ideas? 2015-06-30 9:53 GMT+08:00 Louis Hust louis.h...@gmail.com: Anybody can help? 2015-06-26 23:05 GMT+08:00 Louis Hust louis.h...@gmail.com: Hi, all We are using 0.98.6-cdh5.2.0, rUnknown And when we visit the web regionserverhost:60030 to see Show All RPC Handler Tasks

Re: Scan got exception

2015-06-30 Thread Louis Hust
sourcehbase-site.xml/source /property 2015-06-30 12:22 GMT+08:00 Ted Yu yuzhih...@gmail.com: How do you configure BucketCache ? Thanks On Mon, Jun 29, 2015 at 8:35 PM, Louis Hust louis.h...@gmail.com wrote: BTW, the hbase is hbase0.98.6 CHD5.2.0 2015-06-30 11:31 GMT+08:00 Louis Hust

Re: Scan got exception

2015-06-30 Thread Louis Hust
, Vladimir Rodionov vladrodio...@gmail.com wrote: I believe CDH 5.2.0 does not contain all BucketCache critical patches, but I may be wrong. -Vlad On Tue, Jun 30, 2015 at 12:25 AM, Louis Hust louis.h...@gmail.com wrote: property namehbase.bucketcache.size/name

Re: RegionServer 60030 Show All RPC Handler Task is empty

2015-06-29 Thread Louis Hust
Anybody can help? 2015-06-26 23:05 GMT+08:00 Louis Hust louis.h...@gmail.com: Hi, all We are using 0.98.6-cdh5.2.0, rUnknown And when we visit the web regionserverhost:60030 to see Show All RPC Handler Tasks, found No tasks currently running on this node. But we configure property

Re: Scan got exception

2015-06-29 Thread Louis Hust
BTW, the hbase is hbase0.98.6 CHD5.2.0 2015-06-30 11:31 GMT+08:00 Louis Hust louis.h...@gmail.com: Hi, all When I scan a table using hbase shell, got the following message: hbase(main):001:0 scan 'atpco:ttf_record6' ROW COLUMN+CELL ERROR

Scan got exception

2015-06-29 Thread Louis Hust
Hi, all When I scan a table using hbase shell, got the following message: hbase(main):001:0 scan 'atpco:ttf_record6' ROW COLUMN+CELL ERROR: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected nextCallSeq: 1 But the nextCallSeq

RegionServer 60030 Show All RPC Handler Task is empty

2015-06-26 Thread Louis Hust
Hi, all We are using 0.98.6-cdh5.2.0, rUnknown And when we visit the web regionserverhost:60030 to see Show All RPC Handler Tasks, found No tasks currently running on this node. But we configure property namehbase.regionserver.handler.count/name value150/value sourcehbase-site.xml/source

Re: How to monitor and control heavy network traffic for region server

2015-06-10 Thread Louis Hust
as an example, something similar you will need to implement on top of ResultScanner - ThrottledResultScanner. Good idea for improvement, actually. -Vlad On Wed, Jun 10, 2015 at 7:30 PM, Louis Hust louis.h...@gmail.com wrote: hi, Dave, For now we will not upgrade the version, so

Re: How to monitor and control heavy network traffic for region server

2015-06-10 Thread Louis Hust
hi, Dave, For now we will not upgrade the version, so if there is something we can monitor the long scan for 0.96? 2015-06-11 2:00 GMT+08:00 Dave Latham lat...@davelink.net: I'm not aware of anything in version 0.96 that will limit the scan for you - you may have to do it in your client

Re: HBase Block locality always 0

2015-06-08 Thread Louis Hust
the region server, another region server rs2 got the same problem. 2015-05-19 10:41 GMT+08:00 Louis Hust louis.h...@gmail.com: Hi, Alex, May be the Block locality display wrong? cause I checked some region file and found some replica on the same machine! 2015-05-19 7:18 GMT+08:00 Alex Baranau

Re: Frequent flush and small compactions, region server load is very high

2015-05-21 Thread Louis Hust
, May 19, 2015 at 11:50 PM, Louis Hust louis.h...@gmail.com wrote: Hi, ted, today i counter the problem again, and I find the metrics of GCCOUNT G1 YOUNG GENERATION is related to the CPU LOAD, so this means it's GC problem? The options for regionserver like below: /home/q/java/default

Re: Frequent flush and small compactions, region server load is very high

2015-05-20 Thread Louis Hust
,RFAS org.apache.hadoop.hbase.regionserver.HRegionServer start Any idea? 2015-05-20 11:07 GMT+08:00 Louis Hust louis.h...@gmail.com: Hi, @ted, It seems not only affect the table ttsi_order:ttsi_order_process, some other table's region also flushed and compact 2015-05-20 11:04 GMT+08:00

Re: Frequent flush and small compactions, region server load is very high

2015-05-20 Thread Louis Hust
/YGbbupEDoKTrDo Cheers On Tue, May 19, 2015 at 11:50 PM, Louis Hust louis.h...@gmail.com wrote: Hi, ted, today i counter the problem again, and I find the metrics of GCCOUNT G1 YOUNG GENERATION is related to the CPU LOAD, so this means it's GC problem? The options for regionserver like below

Frequent flush and small compactions, region server load is very high

2015-05-19 Thread Louis Hust
Hi, all My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region server load is very high, about 50, and the CPU usage is about 2400%(with 24 core). I check the log in region server, and found frequent memstore flush and compactions, is this the reason for the high load? {log}

Re: Frequent flush and small compactions, region server load is very high

2015-05-19 Thread Louis Hust
'ttsi_order:ttsi_order_process' ? Thanks On Tue, May 19, 2015 at 4:56 AM, Louis Hust louis.h...@gmail.com wrote: Hi, all My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region server load is very high, about 50, and the CPU usage is about 2400%(with 24 core). I check the log

Re: Frequent flush and small compactions, region server load is very high

2015-05-19 Thread Louis Hust
any ideas? 2015-05-19 19:56 GMT+08:00 Louis Hust louis.h...@gmail.com: Hi, all My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region server load is very high, about 50, and the CPU usage is about 2400%(with 24 core). I check the log in region server, and found frequent

Re: Frequent flush and small compactions, region server load is very high

2015-05-19 Thread Louis Hust
Hi, @ted, It seems not only affect the table ttsi_order:ttsi_order_process, some other table's region also flushed and compact 2015-05-20 11:04 GMT+08:00 Louis Hust louis.h...@gmail.com: Hi, ted hbase(main):001:0 describe 'ttsi_order:ttsi_order_process' DESCRIPTION ENABLED

Re: HMaster restart with error

2015-05-18 Thread Louis Hust
to process the request - it needed some more time. Cheers On Sun, May 17, 2015 at 11:03 PM, Louis Hust louis.h...@gmail.com wrote: Yes, ted, can you tell me what the following excpetion means in l-namenode1.log? 2015-05-15 12:16:40,522 INFO [MASTER_SERVER_OPERATIONS-l-namenode1:6-0

Re: HBase Block locality always 0

2015-05-18 Thread Louis Hust
- open source framework to build and run data applications on Hadoop HBase On Fri, May 15, 2015 at 8:12 PM, Louis Hust louis.h...@gmail.com wrote: Hi, Esteban, Hadoop Version 2.2.0, r1537062. So i do not know why it always write other datanode instead of local datanode

Re: HMaster restart with error

2015-05-18 Thread Louis Hust
count to settle) after it became active master. I tried to download from http://pan.baidu.com/s/1eQlKXj0 (at home) but the download progress was very slow. Will try downloading later in the day. Do you have access to pastebin ? Cheers On Sun, May 17, 2015 at 2:07 AM, Louis Hust

Re: HMaster restart with error

2015-05-17 Thread Louis Hust
to settle; checked in 1, slept for 71582 ms, expecting minimum of 1, maximum of 2147483647, master is running. bq. wait the backend HMaster to take over Was there exception in backup master log after it took over ? On Sat, May 16, 2015 at 6:44 PM, Louis Hust louis.h...@gmail.com wrote: hi,ted

Re: HMaster restart with error

2015-05-16 Thread Louis Hust
hi, TED, Any idea? When the HMaster restart, how can i know when it is really can handle request from application? is there any mark in logs? 2015-05-16 14:05 GMT+08:00 Louis Hust louis.h...@gmail.com: @Ted, plz see the log from 12:11:29 to 12:15:28, this timerange the HMaster

Re: HMaster restart with error

2015-05-16 Thread Louis Hust
, Louis Hust louis.h...@gmail.com wrote: Hi all, I use hbase0.96.0 with hadoop 2.2.0, and the custom said they can not write into hbase cluster, So i stop the HMaster and start it soon, But it seems that the HMaster not response to request, following is the HMaster log: {logs

Re: HMaster restart with error

2015-05-16 Thread Louis Hust
@Ted, plz see the log from 12:11:29 to 12:15:28, this timerange the HMaster is in restarting stage, but can not handle request from client? Is the HMaster recovering or do something else? 2015-05-16 13:59 GMT+08:00 Louis Hust louis.h...@gmail.com: OK, you can get the log from http

Re: HMaster restart with error

2015-05-16 Thread Louis Hust
:58 CST 2015 Terminating master Namenode process was running on l-namenode2.dba.cn8, right ? Cheers On Sat, May 16, 2015 at 7:50 AM, Louis Hust louis.h...@gmail.com wrote: hi, TED, Any idea? When the HMaster restart, how can i know when it is really can handle request from application

HMaster restart with error

2015-05-15 Thread Louis Hust
Hi all, I use hbase0.96.0 with hadoop 2.2.0, and the custom said they can not write into hbase cluster, So i stop the HMaster and start it soon, But it seems that the HMaster not response to request, following is the HMaster log: {logs} 2015-05-15 12:13:33,136 INFO [AM.ZK.Worker-pool2-t16]

Re: HBase Block locality always 0

2015-05-15 Thread Louis Hust
that. Regarding the logs its odd that the JN is taking about 1.5 seconds just to send less than 200 bytes. Perhaps some IO contention issue is going on in your cluster? thanks, esteban. -- Cloudera, Inc. On Thu, May 14, 2015 at 5:48 AM, Louis Hust louis.h...@gmail.com wrote: Hi, Esteban

Re: HBase Block locality always 0

2015-05-14 Thread Louis Hust
Hi, Esteban Each region server has about 122 regions, data is large. HDFS replica is defined as default 3, and namenode have some WARN like below. {log} 2015-05-14 20:45:37,463 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Took 1503ms to send a batch of 3 edits (179 bytes)