[
https://issues.apache.org/jira/browse/HBASE-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863578#action_12863578
]
Charles Glommen commented on HBASE-2509:
----------------------------------------
We had a similar issue during a map/reduce job.
St^Ack provided a quick patch located at http://people.apache.org/~stack/ttr/
to see if the cause and solution is well understood. Results will be posted in
a following comment. I believe the patch to be the following code:
diff --git src/java/org/apache/hadoop/hbase/regionserver/HRegion.java
src/java/org/apache/hadoop/hbase/regionserver/HRegion.java
index 4ed4cbd..8d92fb7 100644
--- src/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++ src/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1974,7 +1974,10 @@ public class HRegion implements HConstants, HeapSize {
// , Writable{
}
public void close() {
- storeHeap.close();
+ if (storeHeap != null) {
+ storeHeap.close();
+ storeHeap = null;
+ }
}
Below is the stack trace that occurred during the job, and also during 'get'
and 'scan' commands. The following (located in full at
http://pastebin.com/74bJeqTY) stack traces occurred upon upgrading from 0.20.3
to 0.20.4.
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server 172.28.10.204:60020 for region
user_events,320\x7Cqf24mcristobal\x7Ca,1272349551722, row
'321\x7C754435419575754318\x7Cc', but failed after 10 attempts.
Exceptions:
java.io.IOException: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:887)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:877)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1744)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.close(HRegion.java:1977)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2513)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2496)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1742)
... 6 more
java.io.IOException: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:887)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:877)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1744)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.close(HRegion.java:1977)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2513)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2496)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1742)
... 6 more
java.io.IOException: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:887)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:877)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1744)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.close(HRegion.java:1977)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2513)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2496)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1742)
... 6 more
> NPEs in various places, HRegion.get, HRS.close
> ----------------------------------------------
>
> Key: HBASE-2509
> URL: https://issues.apache.org/jira/browse/HBASE-2509
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.4
> Reporter: ryan rawson
> Attachments: HBASE-2509-v2.txt, HBASE-2509.txt
>
>
> ttr on irc reported that he was unable to get/scan sometimes, was getting
> NPEs.
> The root cause is a delayed init of the RegionScanner.storeHeap means it can
> be null, not all accessors of it (specifically in close()) checked for that.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
