Ryan, It worked. thank you!
I have used truncate command on TestTable before, but I then decided to drop the table completely and start the test from scratch. This problem seem to happen after the hbase server crashed. I experienced both hbase-1634/HBASE-1638 and this count issue. Running the script in 1638 resolve the NPE problem (Thanks stack), running flush/major_compact meta table resolved the count issue. For my education, do you mind explaining the cause/fix for this count issue I am seeing? Thanks Haijun ________________________________ From: Ryan Rawson <[email protected]> To: [email protected] Sent: Tuesday, July 14, 2009 3:14:33 PM Subject: Re: java.io.IOException: HRegionInfo was null or empty in .META. Try doing this on the shell: flush '.META.' major_compact '.META.' did you use 'truncate' command at any point? On Tue, Jul 14, 2009 at 3:10 PM, Haijun Cao<[email protected]> wrote: > > Hi > > I am running hbase PE test, loaded TestTable with 10M records without any > problem, later hbase crashed (during another sequentialWrite test), after > restart, I can't count TestTable, it always get stuck at 1.1 mil records. > > > I am wondering if anybody has encountered the same problem (data got > corrupted after server crash)? How do you recover? At this point, it is just > test data, so it is ok for me to lose some data. Is there a way to > drop/repair the bad region (0001192603)? Does hbase have a auto-repair tool > like fsck? > > Your help is greatly appreciated. > > Haijun > > > Error stack: > > Current count: 1100000, row: 0001099999 > NativeException: java.lang.RuntimeException: > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact > region server null for region , row '0001192603', but failed after 5 attempts. > Exceptions: > java.io.IOException: HRegionInfo was null or empty in .META. > java.io.IOException: HRegionInfo was null or empty in .META. > java.io.IOException: HRegionInfo was null or empty in .META. > java.io.IOException: HRegionInfo was null or empty in .META. > java.io.IOException: HRegionInfo was null or empty in .META. > > from org/apache/hadoop/hbase/client/HTable.java:2083:in `hasNext' > from sun.reflect.GeneratedMethodAccessor9:-1:in `invoke' > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke' > from java/lang/reflect/Method.java:597:in `invoke' > from org/jruby/javasupport/JavaMethod.java:298:in > `invokeWithExceptionHandling' > from org/jruby/javasupport/JavaMethod.java:259:in `invoke' > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call' > from org/jruby/runtime/callsite/CachingCallSite.java:70:in `call' > from org/jruby/ast/CallNoArgNode.java:61:in `interpret' > from org/jruby/ast/WhileNode.java:127:in `interpret' > from org/jruby/ast/NewlineNode.java:104:in `interpret' > from org/jruby/ast/BlockNode.java:71:in `interpret' > from org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in > `call' > from org/jruby/internal/runtime/methods/DefaultMethod.java:144:in > `call' > from org/jruby/runtime/callsite/CachingCallSite.java:273:in > `cacheAndCall' > from org/jruby/runtime/callsite/CachingCallSite.java:112:in `call' > > > I scan the .META. table and noticed that region 0001192603 has no info > columns, only historian columns. I also check the hadoop file system, region > 0001192603 has a oldlogfile.log file in its directory (other regions don't): > > Found 3 items > -rw-r--r-- 3 bamboo supergroup 619 2009-07-13 16:28 > /user/bamboo/hbase/TestTable/954997373/.regioninfo > drwxr-xr-x - bamboo supergroup 0 2009-07-13 16:28 > /user/bamboo/hbase/TestTable/954997373/info > -rw-r--r-- 3 bamboo supergroup 39417817 2009-07-13 16:52 > /user/bamboo/hbase/TestTable/954997373/oldlogfile.log > > >
