Failed to open scanner in shell after series of action executed
---------------------------------------------------------------
Key: HBASE-827
URL: https://issues.apache.org/jira/browse/HBASE-827
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.3.0
Environment: Java runtime version : 1.6.0_06
Hbase version : occured in both 0.2.0 RC2 & 0.3.0 trunk (svn 685494)
Hbase configuration: run in localhost without hadoop
Reporter: PeterL in
Background introduction :
I was intent to implement a large queue-liked structure based on HBase.
Different from normal queue structure, in my structure, a pop action means
'randomly' pick one elements(which is key/value pair from a row) in queue, and
decrease the queueSize of that queue.
For performance consideration, I implemented pop action by following steps :
1.get the queue size
2.random a number between 0~queueSize
3.take step2 as a qualifier, get the value of that element, and put the last
element's value to the original qualifier.
4.decrease the queue size. (I didn't delete the last elements since storage
space is not my primary consideration)
When the queueSize and number of pop action become large, hbase failed to open
a scanner. (Even the data of database is correct)
My program can be split into few steps:
1.create a large queue.
2.pop until the queue empty
3.keep popping the queue
4.delete the queue ( delete a row with deleteAll )
5.create a small queue with same name (i.e: same row)
After create a small queue, I checked every key/value in the hbase and found it
is correct. But when I tried to open a scanner, it got Unknown Exception as
follows:
In shell :
NativeException: org.apache.hadoop.hbase.client.RetriesExhaustedException:
Trying to contact region server 127.0.0.1:54141 for region
pleaseCrash,,1218621377152, row '', but failed after 3 attempts.
Exceptions:
java.net.SocketTimeoutException: timed out waiting for rpc response
java.net.SocketTimeoutException: timed out waiting for rpc response
java.net.SocketTimeoutException: timed out waiting for rpc response
from org/apache/hadoop/hbase/client/HConnectionManager.java:873:in
`getRegionServerWithRetries'
from org/apache/hadoop/hbase/client/HTable.java:1415:in `nextScanner'
from org/apache/hadoop/hbase/client/HTable.java:1360:in `<init>'
from org/apache/hadoop/hbase/client/HTable.java:985:in `getScanner'
from org/apache/hadoop/hbase/client/HTable.java:779:in `getScanner'
from org/apache/hadoop/hbase/client/HTable.java:706:in `getScanner'
from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
from java/lang/reflect/Method.java:597:in `invoke'
from org/jruby/javasupport/JavaMethod.java:250:in
`invokeWithExceptionHandling'
from org/jruby/javasupport/JavaMethod.java:219:in `invoke'
from org/jruby/javasupport/JavaClass.java:416:in `execute'
from org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in
`call'
from org/jruby/internal/runtime/methods/DynamicMethod.java:78:in `call'
from org/jruby/runtime/CallSite.java:155:in `cacheAndCall'
... 128 levels...
from
ruby.home.itspeter.hbase_minus_0_dot_3_dot_0_minus_dev.bin.hirbInvokermethod__23$RUBY$startOpt:-1:in
`call'
from org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
from org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
from org/jruby/runtime/CallSite.java:298:in `call'
from
ruby/home/itspeter/hbase_minus_0_dot_3_dot_0_minus_dev/bin//home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:350:in
`__file__'
from
ruby/home/itspeter/hbase_minus_0_dot_3_dot_0_minus_dev/bin//home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:-1:in
`__file__'
from
ruby/home/itspeter/hbase_minus_0_dot_3_dot_0_minus_dev/bin//home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:-1:in
`load'
from org/jruby/Ruby.java:512:in `runScript'
from org/jruby/Ruby.java:432:in `runNormally'
from org/jruby/Ruby.java:312:in `runFromMain'
from org/jruby/Main.java:144:in `run'
from org/jruby/Main.java:89:in `run'
from org/jruby/Main.java:80:in `main'
from /home/itspeter/hbase-0.3.0-dev/bin/../bin/hirb.rb:271:in `scan'
from (hbase):3:in `binding'
Side effects :
After scan got time out, I discovered that I also failed to disable the
table (But not every time), somehow more strange is that table will be dropped
successfully even disable failed (but not every time).
System Resources:
Hbase will rob the cpu...
In my platform(4-core), cpu usage will raise from 10% -> 50% ..100% ->
200% -> 300% ..
Procedure to reproduce :
Run the attached java file.
Try to type "scan 'pleaseCrash'" in hbase shell after program finished.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
