[
https://issues.apache.org/jira/browse/HBASE-827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12623175#action_12623175
]
Rong-En Fan commented on HBASE-827:
-----------------------------------
I grab the MyClient.java and latest trunk (r686572) to try it out. So, the
example program do the following things:
1. insert 40000 columns under one col family
2. insert a number into the same col family to record the total # of cols in
this family
3. repeat 100,000 times, do read one column value, modify it, insert back
4. delete everything
5. repeat 1 but with much smaller columns to insert
Then I try to get a scanner from java client:
table.getScanner(new String[]{"insPool:"});
The table in question has only few columns but lots of deleted records (about
140,000). The scanner needs some time to finish it's job. The line above
requires 240 seconds to finish (that's 4 mins).
However, in hirb.rb, we have
@configuration.setInt("hbase.client.retries.number", 3)
@configuration.setInt("ipc.client.connect.max.retries", 3)
These are smaller than hbase-default.xml, and this is why scan shell fails.
> Failed to open scanner in shell after series of action executed
> ---------------------------------------------------------------
>
> Key: HBASE-827
> URL: https://issues.apache.org/jira/browse/HBASE-827
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.3.0
> Environment: Java runtime version : 1.6.0_06
> Hbase version : occured in both 0.2.0 RC2 & 0.3.0 trunk (svn 685494)
> Hbase configuration: run in localhost without hadoop
> Reporter: PeterL in
> Attachments: hbase.log, jstack.log, MyClient.java
>
>
> Background introduction :
> I was intent to implement a large queue-liked structure based on HBase.
> Different from normal queue structure, in my structure, a pop action means
> 'randomly' pick one elements(which is key/value pair from a row) in queue,
> and decrease the queueSize of that queue.
> For performance consideration, I implemented pop action by following steps :
> 1.get the queue size
> 2.random a number between 0~queueSize
> 3.take step2 as a qualifier, get the value of that element, and put the last
> element's value to the original qualifier.
> 4.decrease the queue size. (I didn't delete the last elements since storage
> space is not my primary consideration)
> When the queueSize and number of pop action become large, hbase failed to
> open a scanner. (Even the data of database is correct)
> My program can be split into few steps:
> 1.create a large queue.
> 2.pop until the queue empty
> 3.keep popping the queue
> 4.delete the queue ( delete a row with deleteAll )
> 5.create a small queue with same name (i.e: same row)
> After create a small queue, I checked every key/value in the hbase and found
> it is correct. But when I tried to open a scanner, it got Unknown Exception
> as follows:
> In shell :
> NativeException: org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Trying to contact region server 127.0.0.1:54141 for region
> pleaseCrash,,1218621377152, row '', but failed after 3 attempts.
> Exceptions:
> java.net.SocketTimeoutException: timed out waiting for rpc response
> java.net.SocketTimeoutException: timed out waiting for rpc response
> java.net.SocketTimeoutException: timed out waiting for rpc response
> from org/apache/hadoop/hbase/client/HConnectionManager.java:873:in
> `getRegionServerWithRetries'
> from org/apache/hadoop/hbase/client/HTable.java:1415:in `nextScanner'
> from org/apache/hadoop/hbase/client/HTable.java:1360:in `<init>'
> from org/apache/hadoop/hbase/client/HTable.java:985:in `getScanner'
> from org/apache/hadoop/hbase/client/HTable.java:779:in `getScanner'
> from org/apache/hadoop/hbase/client/HTable.java:706:in `getScanner'
> from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
> from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
> from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> from java/lang/reflect/Method.java:597:in `invoke'
> from org/jruby/javasupport/JavaMethod.java:250:in
> `invokeWithExceptionHandling'
> from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
> from org/jruby/javasupport/JavaClass.java:416:in `execute'
> from
> org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in `call'
> from org/jruby/internal/runtime/methods/DynamicMethod.java:78:in
> `call'
> from org/jruby/runtime/CallSite.java:155:in `cacheAndCall'
> ... 128 levels...
> from
> ruby.home.itspeter.hbase_minus_0_dot_3_dot_0_minus_dev.bin.hirbInvokermethod__23$RUBY$startOpt:-1:in
> `call'
> from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in
> `call'
> from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in
> `call'
> from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
> from org/jruby/runtime/CallSite.java:298:in `call'
> from
> ruby/home/itspeter/hbase_minus_0_dot_3_dot_0_minus_dev/bin//home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:350:in
> `__file__'
> from
> ruby/home/itspeter/hbase_minus_0_dot_3_dot_0_minus_dev/bin//home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:-1:in
> `__file__'
> from
> ruby/home/itspeter/hbase_minus_0_dot_3_dot_0_minus_dev/bin//home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:-1:in
> `load'
> from org/jruby/Ruby.java:512:in `runScript'
> from org/jruby/Ruby.java:432:in `runNormally'
> from org/jruby/Ruby.java:312:in `runFromMain'
> from org/jruby/Main.java:144:in `run'
> from org/jruby/Main.java:89:in `run'
> from org/jruby/Main.java:80:in `main'
> from /home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:271:in `scan'
> from (hbase):3:in `binding'
> Side effects :
> After scan got time out, I discovered that I also failed to disable the
> table (But not every time), somehow more strange is that table will be
> dropped successfully even disable failed (but not every time).
> System Resources:
> Hbase will rob the cpu...
> In my platform(4-core), cpu usage will raise from 10% -> 50% ..100% ->
> 200% -> 300% ..
> Procedure to reproduce :
> Run the attached java file.
> Try to type "scan 'pleaseCrash'" in hbase shell after program finished.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
