答复: too many regions

On Thu, Jun 26, 2014 at 7:41 PM, sunweiwei su...@asiainfo-linkage.com wrote: Hi I'm using a hbase0.94.2 cluster, which has 3 zookeepers， 17 regionservers, about 4 regions. each regionserver has about 2300 regions. Maybe the cluster has too many regions. 1、If a regionserver die,how

too many regions

Hi I'm using a hbase0.94.2 cluster, which has 3 zookeepers， 17 regionservers, about 4 regions. each regionserver has about 2300 regions. Maybe the cluster has too many regions. 1、If a regionserver die,how long hmaster reassign the regionserver's 2300 regions . If I restart

答复: 答复: 答复: regionserver 60020 timeout

Thanks, Qiang Tian -邮件原件- 发件人: Qiang Tian [mailto:tian...@gmail.com] 发送时间: 2014年6月23日 10:55 收件人: user@hbase.apache.org 主题: Re: 答复: 答复: regionserver 60020 timeout Hi WeiWei, FYI - the problem should be hbase-11277. On Thu, Jun 12, 2014 at 9:04 PM, sunweiwei su...@asiainfo-linkage.com

答复: regions not balance?

= + serversByLoad.firstKey().getLoad()); } return false; } return true; } 发件人： sunweiwei 发送时间： 2014-06-17 12:55 收件人： user@hbase.apache.org 主题： 答复: regions not balance? I'm using hbase0.96. There is no balancer related logs in master log at that moment. I find

regions not balance?

Hi I have several regionservers. One regionserver has 434 regions and other regionservers only have about 300+ regions. I think it's not balance. Then I run balancer command in hbase shell and return true, but It not works. The regionserver always has 400+ regions. Is this normal?

答复: regions not balance?

. and return true, but It not works Looking at master log, do you see balancer related logs around this time ? If there was region in transition at this moment, balancer wouldn't perform balancing. What version of HBase are you using ? Cheers On Mon, Jun 16, 2014 at 8:19 PM, sunweiwei su

master caught a ClosedChannelException

Hi I'm using hbase0.96. When I start hamster , it's ok. After a little while, I saw these in logs. 2014-06-13 18:10:15,073 WARN [RpcServer.handler=48,port=6] ipc.RpcServer: RpcServer.respondercallId: 206739 service: RegionServerStatusService methodName: RegionServerReport size: 79.5 K

答复: 答复: regionserver 60020 timeout

I check again. The timeout regionserver is just sometimes the namespace server, not always. -邮件原件- 发件人: sunweiwei [mailto:su...@asiainfo-linkage.com] 发送时间: 2014年6月12日 20:35 收件人: user@hbase.apache.org 主题: 答复: 答复: regionserver 60020 timeout Thanks, Qiang Tian. I'm using hbase0.96.0

regionserver 60020 timeout

Hi I find some 6 millis timeout in master log, when master connect regionserver's 60020 port. Log like this: because java.net.SocketTimeoutException: 6 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected

答复: regionserver 60020 timeout

lower the chance of hitting busy reader... this.readThreads = conf.getInt(ipc.server.read.threadpool.size, 10); On Fri, Jun 6, 2014 at 2:35 PM, sunweiwei su...@asiainfo-linkage.com wrote: Hi I find some 6 millis timeout in master log, when master connect regionserver's 60020 port. Log like

答复: regionserver 60020 timeout

@gmail.com [mailto:saint@gmail.com] 代表 Stack 发送时间: 2014年6月7日 0:16 收件人: Hbase-User 主题: Re: regionserver 60020 timeout On Thu, Jun 5, 2014 at 11:35 PM, sunweiwei su...@asiainfo-linkage.com wrote: Hi I find some 6 millis timeout in master log, when master connect regionserver's 60020 port. Log

答复: 答复: 答复: forcing flush not works

Hi The symptom reproduced again. I paste the log in http://paste2.org/D2N6ZDvk,http://paste2.org/a64LXD0X One is the regionserver jstack log. The other is regionserver log, which was grep and only include the unflush region. Thanks -邮件原件- 发件人: sunweiwei [mailto:su...@asiainfo

答复: 答复: 答复: forcing flush not works

, Jun 5, 2014 at 11:06 AM, Stack st...@duboce.net wrote: Always the same two regions that get stuck or does it vary? Another set of example logs may help uncover the sequence of trouble-causing events. Thanks, St.Ack On Wed, Jun 4, 2014 at 7:31 PM, sunweiwei su...@asiainfo-linkage.com wrote

答复: 答复: forcing flush not works

at 2:34 AM, sunweiwei su...@asiainfo-linkage.com wrote: Thanks. Maybe the same as HBase-10499. I stop the regionserver then start it. Then hbase back to normal. This is jstack log when 2 regions can not flush. Thread-17 prio=10 tid=0x7f6210383800 nid=0x6540 waiting on condition

forcing flush not works

Hi I'm using a heavy-write hbase0.96 . I find this in regionserver log: 2014-06-03 15:13:19,445 INFO [regionserver60020.logRoller] wal.FSHLog: Too many hlogs: logs=33, maxlogs=32; forcing flush of 3 regions(s): 1a7dda3c3815c19970ace39fd99abfe8, aff81bc46aa7d3ed51a01f11f23c8320,

答复: forcing flush not works

that there is something wrong with the flush queue(some entry pushed into it can't be poll out). 发件人: sunweiwei [su...@asiainfo-linkage.com] 发送时间: 2014年6月3日 15:43 收件人: user@hbase.apache.org 主题: forcing flush not works Hi I'm using a heavy-write hbase0.96 . I find

答复: 答复: 答复: what in regionserver's Used Heap

Thanks you -邮件原件- 发件人: Ted Yu [mailto:yuzhih...@gmail.com] 发送时间: 2014年5月29日 6:06 收件人: user@hbase.apache.org 主题: Re: 答复: 答复: what in regionserver's Used Heap Weiwei: I logged HBASE-11267 for the heap consumption issue you reported. Cheers On Sun, May 25, 2014 at 6:42 PM, sunweiwei su

答复: 答复: 答复: what in regionserver's Used Heap

月26日 13:10 收件人: user@hbase.apache.org 主题: Re: 答复: 答复: what in regionserver's Used Heap Some metrics2 classes show up in the histogram. I am digging into these classes and polling developers who are more familiar with metrics2 system. FYI On Sun, May 25, 2014 at 6:42 PM, sunweiwei su...@asiainfo

答复: 答复: 答复: 答复: what in regionserver's Used Heap

-metrics2-hbase.properties ? Thanks On Sun, May 25, 2014 at 11:37 PM, sunweiwei su...@asiainfo-linkage.comwrote: Yes, I'm using metrics2 with ganglia. And I have closed ganglia, because there is too many region metrics. But I don't delete metrics2 configuration in $HBASE_HOME/conf/hadoop

答复: 答复: what in regionserver's Used Heap

jmap with -histo option to find out heap consumption: http://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html Please pastebin jmap output. Cheers On Thu, May 22, 2014 at 11:30 PM, sunweiwei su...@asiainfo-linkage.comwrote: Hi Ted I have checked regionsever:60030/rs-status

答复: what in regionserver's Used Heap

and 15.4.4.7) Cheers On Thu, May 22, 2014 at 8:39 PM, sunweiwei su...@asiainfo-linkage.comwrote: Hi I'm using hbase0.96. In hbase web, I see : Max Heap is 16.0 G Memstore Size is 208.6 M Cache Size is 138M Cache Free is 6.2G And Used Heap is 10.2 G. Used Heap is large

what in regionserver's Used Heap

Hi I'm using hbase0.96. In hbase web, I see : Max Heap is 16.0 G Memstore Size is 208.6 M Cache Size is 138M Cache Free is 6.2G And Used Heap is 10.2 G. Used Heap is large. I want to know what could be in then Used Heap. Thanks.

答复: 答复: meta server hungs ?

件人: sunweiwei [mailto:su...@asiainfo-linkage.com] 发送时间: 2014年5月6日 9:27 收件人: user@hbase.apache.org 主题: 答复: 答复: meta server hungs ? HI Samir I think master declared hadoop77/192.168.1.87:60020 as dead server, because of Failed verification of hbase:meta,,1 at address

答复: 答复: 答复: meta server hungs ?

11日 19:11 收件人: user@hbase.apache.org 抄送: user@hbase.apache.org 主题: Re: 答复: 答复: meta server hungs ? What GC parameters did you specify for JVM ? Thanks On May 7, 2014, at 6:27 PM, sunweiwei su...@asiainfo-linkage.com wrote: I find lots of these in gc.log. It seems like CMS gc run many times

答复: 答复: meta server hungs ?

: 1.228/1.228 secs] [Times: user=0.00 sys=0.00, real=1.23 secs] 2014-04-29T13:56:48.231+0800: 2144558.938: [CMS-concurrent-sweep-start] 2014-04-29T13:56:49.490+0800: 2144560.196: [CMS-concurrent-sweep: 1.258/1.258 secs] [Times: user=0.00 sys=0.00, real=1.26 secs] -邮件原件- 发件人: sunweiwei

meta server hungs ?

Hi I'm using hbase0.96.0. I found client can't put data suddenly and hmaster hungs. Then I shutdown the hmaster and start a new hmaster, then the client back to normal. I found this logs in the new hmaster . It seem like meta server hungs and hmaster stop the meta server. 2014-04-29

答复: meta server hungs ?

perm gen total 81072K, used 48660K [0x0007fae0, 0x0007ffd2c000, 0x0008) -邮件原件- 发件人: Samir Ahmic [mailto:ahmic.sa...@gmail.com] 发送时间: 2014年5月5日 16:50 收件人: user@hbase.apache.org 抄送: sunweiwei 主题: Re: meta server hungs ? Hi, This exception: exception

答复: meta server hungs ?

:917) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:901) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:863) -邮件原件- 发件人: sunweiwei [mailto:su...@asiainfo-linkage.com] 发送时间: 2014年5月5日 17:23 收件人: user@hbase.apache.org 主题: 答复: meta server hungs ? Thank you

答复: 答复: meta server hungs ?

hadoop77/192.168.1.87:60020 as dead server. Regards Samir On Mon, May 5, 2014 at 11:39 AM, sunweiwei su...@asiainfo-linkage.comwrote: And this is client log. 2014-04-29 13:53:57,271 WARN [main] org.apache.hadoop.hbase.client.ScannerCallable: Ignore, probably already closed

oldWALs too large

Hi I'm using hbase0.96.0, with 1 hmaster,3 regionservers. Write request is About 1~10w/s. Today I found HBase Master Hangs ,Regionservers dead and oldWALs dir is Very Large. /apps/hbase/data/data is about 800G. /apps/hbase/data/oldWALs is about 4.2T. This cause HDFS Full. any