[jira] [Updated] (HBASE-2077) NullPointerException with an open scanner that expired causing an immediate region server shutdown
[ https://issues.apache.org/jira/browse/HBASE-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-2077: - Release Note: Removes lease from lease monitor while operation is running inside the server. NullPointerException with an open scanner that expired causing an immediate region server shutdown -- Key: HBASE-2077 URL: https://issues.apache.org/jira/browse/HBASE-2077 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.20.2, 0.20.3 Environment: Hadoop 0.20.0, Mac OS X, Java 6 Reporter: Sam Pullara Assignee: Sam Pullara Priority: Critical Fix For: 0.90.4 Attachments: 2077-suggestion.txt, 2077-v4.txt, HBASE-2077-3.patch, HBASE-2077-redux.patch, [Bug_HBASE-2077]_Fixes_a_very_rare_race_condition_between_lease_expiration_and_renewal.patch Original Estimate: 1h Remaining Estimate: 1h 2009-12-29 18:05:55,432 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -4250070597157694417 lease expired 2009-12-29 18:05:55,443 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.NullPointerException at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117) at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641) at java.util.PriorityQueue.siftDown(PriorityQueue.java:612) at java.util.PriorityQueue.poll(PriorityQueue.java:523) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) 2009-12-29 18:05:55,446 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 55260, call next(-4250070597157694417, 1) from 192.168.1.90:54011: error: java.io.IOException: java.lang.NullPointerException java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:869) at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:859) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1965) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117) at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641) at java.util.PriorityQueue.siftDown(PriorityQueue.java:612) at java.util.PriorityQueue.poll(PriorityQueue.java:523) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944) ... 5 more 2009-12-29 18:05:55,447 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call next(-4250070597157694417, 1) from
[jira] [Updated] (HBASE-2077) NullPointerException with an open scanner that expired causing an immediate region server shutdown
[ https://issues.apache.org/jira/browse/HBASE-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-2077: - Attachment: 2077-suggestion.txt Here is a suggestion where we remove lease from leases while we are processing a request then on the way out in a finally we renew lease. NullPointerException with an open scanner that expired causing an immediate region server shutdown -- Key: HBASE-2077 URL: https://issues.apache.org/jira/browse/HBASE-2077 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.20.2, 0.20.3 Environment: Hadoop 0.20.0, Mac OS X, Java 6 Reporter: Sam Pullara Assignee: Sam Pullara Priority: Critical Fix For: 0.92.0 Attachments: 2077-suggestion.txt, HBASE-2077-3.patch, HBASE-2077-redux.patch, [Bug_HBASE-2077]_Fixes_a_very_rare_race_condition_between_lease_expiration_and_renewal.patch Original Estimate: 1h Remaining Estimate: 1h 2009-12-29 18:05:55,432 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -4250070597157694417 lease expired 2009-12-29 18:05:55,443 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.NullPointerException at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117) at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641) at java.util.PriorityQueue.siftDown(PriorityQueue.java:612) at java.util.PriorityQueue.poll(PriorityQueue.java:523) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) 2009-12-29 18:05:55,446 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 55260, call next(-4250070597157694417, 1) from 192.168.1.90:54011: error: java.io.IOException: java.lang.NullPointerException java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:869) at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:859) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1965) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117) at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641) at java.util.PriorityQueue.siftDown(PriorityQueue.java:612) at java.util.PriorityQueue.poll(PriorityQueue.java:523) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944) ... 5 more 2009-12-29 18:05:55,447 WARN org.apache.hadoop.ipc.HBaseServer: IPC
[jira] [Updated] (HBASE-2077) NullPointerException with an open scanner that expired causing an immediate region server shutdown
[ https://issues.apache.org/jira/browse/HBASE-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-2077: - Attachment: 2077-v4.txt Ahemm.. this is a version that actually works (TestFromClientSide is a good test for this change). NullPointerException with an open scanner that expired causing an immediate region server shutdown -- Key: HBASE-2077 URL: https://issues.apache.org/jira/browse/HBASE-2077 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.20.2, 0.20.3 Environment: Hadoop 0.20.0, Mac OS X, Java 6 Reporter: Sam Pullara Assignee: Sam Pullara Priority: Critical Fix For: 0.92.0 Attachments: 2077-suggestion.txt, 2077-v4.txt, HBASE-2077-3.patch, HBASE-2077-redux.patch, [Bug_HBASE-2077]_Fixes_a_very_rare_race_condition_between_lease_expiration_and_renewal.patch Original Estimate: 1h Remaining Estimate: 1h 2009-12-29 18:05:55,432 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -4250070597157694417 lease expired 2009-12-29 18:05:55,443 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.NullPointerException at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117) at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641) at java.util.PriorityQueue.siftDown(PriorityQueue.java:612) at java.util.PriorityQueue.poll(PriorityQueue.java:523) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) 2009-12-29 18:05:55,446 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 55260, call next(-4250070597157694417, 1) from 192.168.1.90:54011: error: java.io.IOException: java.lang.NullPointerException java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:869) at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:859) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1965) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117) at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641) at java.util.PriorityQueue.siftDown(PriorityQueue.java:612) at java.util.PriorityQueue.poll(PriorityQueue.java:523) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944) ... 5 more 2009-12-29 18:05:55,447 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call
[jira] Updated: (HBASE-2077) NullPointerException with an open scanner that expired causing an immediate region server shutdown
[ https://issues.apache.org/jira/browse/HBASE-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ryan rawson updated HBASE-2077: --- Fix Version/s: (was: 0.90.0) 0.92.0 core issue: no concurrency control between next() calls and lease timeouts. While nice to have, this is a corner case and can't hold up 0.90 NullPointerException with an open scanner that expired causing an immediate region server shutdown -- Key: HBASE-2077 URL: https://issues.apache.org/jira/browse/HBASE-2077 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.20.2, 0.20.3 Environment: Hadoop 0.20.0, Mac OS X, Java 6 Reporter: Sam Pullara Assignee: Sam Pullara Priority: Critical Fix For: 0.92.0 Attachments: [Bug_HBASE-2077]_Fixes_a_very_rare_race_condition_between_lease_expiration_and_renewal.patch, HBASE-2077-3.patch, HBASE-2077-redux.patch Original Estimate: 1h Remaining Estimate: 1h 2009-12-29 18:05:55,432 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -4250070597157694417 lease expired 2009-12-29 18:05:55,443 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.NullPointerException at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117) at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641) at java.util.PriorityQueue.siftDown(PriorityQueue.java:612) at java.util.PriorityQueue.poll(PriorityQueue.java:523) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) 2009-12-29 18:05:55,446 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 55260, call next(-4250070597157694417, 1) from 192.168.1.90:54011: error: java.io.IOException: java.lang.NullPointerException java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:869) at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:859) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1965) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1310) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:136) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:127) at org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:117) at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:641) at java.util.PriorityQueue.siftDown(PriorityQueue.java:612) at java.util.PriorityQueue.poll(PriorityQueue.java:523) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:113) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944) ... 5 more 2009-12-29 18:05:55,447 WARN
