[jira] [Commented] (HBASE-16737) NPE during close of RegionScanner
[
https://issues.apache.org/jira/browse/HBASE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552127#comment-15552127
]
Hadoop QA commented on HBASE-16737:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 56s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 59s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 134m 4s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestHTableMultiplexer |
| | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831948/0001-HBASE-16737-Fix-NPE.patch
|
| JIRA Issue | HBASE-16737 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux c01029d2754c 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 912ed17 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3857/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3857/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-16737) NPE during close of RegionScanner
[
https://issues.apache.org/jira/browse/HBASE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551832#comment-15551832
]
Mark Christiaens commented on HBASE-16737:
--
I've provide a patch. I'm unable to reproduce this issue so I have no idea
whether this is an adequate fix. It applies against rev.
f1e1312f9790a7c40f6a4b5a1bab2ea1dd559890
> NPE during close of RegionScanner
> -
>
> Key: HBASE-16737
> URL: https://issues.apache.org/jira/browse/HBASE-16737
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.0.3
>Reporter: Mark Christiaens
> Attachments: 0001-HBASE-16737-Fix-NPE.patch
>
>
> We encountered the following stack trace during high load:
> {noformat}
> Unexpected throwable object
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.CellComparator.compareRows(CellComparator.java:186)
> at
> org.apache.hadoop.hbase.CellComparator.compare(CellComparator.java:63)
> at
> org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:2021)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:202)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:178)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:168)
> at
> java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:719)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
> at java.util.PriorityQueue.poll(PriorityQueue.java:595)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:218)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:5608)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$2.close(BaseScannerRegionObserver.java:279)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$1.close(BaseScannerRegionObserver.java:186)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2378)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2034)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> What I suspect is happening is that the {{RegionScannerImpl}} is being closed
> while the scanner's lease is expired. During this close, the underlying
> {{KeyValueHeap}} is being polled. the {{heap}} tries to read data from
> {{KeyValueScanners}} that then return {{null}} which causes the crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16737) NPE during close of RegionScanner
[
https://issues.apache.org/jira/browse/HBASE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536331#comment-15536331
]
Anoop Sam John commented on HBASE-16737:
Not sure why this was added this way. Dont see any issue in iterating over all
scanner and close each. Want to attach a patch?
> NPE during close of RegionScanner
> -
>
> Key: HBASE-16737
> URL: https://issues.apache.org/jira/browse/HBASE-16737
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Mark Christiaens
>
> We encountered the following stack trace during high load:
> {noformat}
> Unexpected throwable object
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.CellComparator.compareRows(CellComparator.java:186)
> at
> org.apache.hadoop.hbase.CellComparator.compare(CellComparator.java:63)
> at
> org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:2021)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:202)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:178)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:168)
> at
> java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:719)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
> at java.util.PriorityQueue.poll(PriorityQueue.java:595)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:218)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:5608)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$2.close(BaseScannerRegionObserver.java:279)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$1.close(BaseScannerRegionObserver.java:186)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2378)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2034)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> What I suspect is happening is that the {{RegionScannerImpl}} is being closed
> while the scanner's lease is expired. During this close, the underlying
> {{KeyValueHeap}} is being polled. the {{heap}} tries to read data from
> {{KeyValueScanners}} that then return {{null}} which causes the crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16737) NPE during close of RegionScanner
[
https://issues.apache.org/jira/browse/HBASE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536161#comment-15536161
]
Mark Christiaens commented on HBASE-16737:
--
[~anoop.hbase] The call through {{preScannerClose}} seems OK to me.
To me, this code looks surprising
(org.apache.hadoop.hbase.regionserver.KeyValueHeap#close):
{noformat}
public void close() {
if (this.current != null) {
this.current.close();
}
if (this.heap != null) {
KeyValueScanner scanner;
while ((scanner = this.heap.poll()) != null) {
scanner.close();
}
}
}
{noformat}
Looks like it wants to close all scanners that are in the {{heap}}. Instead of
iterating over them and closing them, the code performs a _poll_ to get them?
> NPE during close of RegionScanner
> -
>
> Key: HBASE-16737
> URL: https://issues.apache.org/jira/browse/HBASE-16737
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Mark Christiaens
>
> We encountered the following stack trace during high load:
> {noformat}
> Unexpected throwable object
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.CellComparator.compareRows(CellComparator.java:186)
> at
> org.apache.hadoop.hbase.CellComparator.compare(CellComparator.java:63)
> at
> org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:2021)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:202)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:178)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:168)
> at
> java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:719)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
> at java.util.PriorityQueue.poll(PriorityQueue.java:595)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:218)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:5608)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$2.close(BaseScannerRegionObserver.java:279)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$1.close(BaseScannerRegionObserver.java:186)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2378)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2034)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> What I suspect is happening is that the {{RegionScannerImpl}} is being closed
> while the scanner's lease is expired. During this close, the underlying
> {{KeyValueHeap}} is being polled. the {{heap}} tries to read data from
> {{KeyValueScanners}} that then return {{null}} which causes the crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16737) NPE during close of RegionScanner
[
https://issues.apache.org/jira/browse/HBASE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535960#comment-15535960
]
Anoop Sam John commented on HBASE-16737:
{code}
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:5608)
at
org.apache.phoenix.coprocessor.BaseScannerRegionObserver$2.close(BaseScannerRegionObserver.java:279)
at
org.apache.phoenix.coprocessor.BaseScannerRegionObserver$1.close(BaseScannerRegionObserver.java:186)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2378)
{code}
Seems this is preScannerClose() CP hook flow. But I can not see traces of that.
The Observer is trying to close a RegionScanner on its own? Is this scanner
created by the CP?
> NPE during close of RegionScanner
> -
>
> Key: HBASE-16737
> URL: https://issues.apache.org/jira/browse/HBASE-16737
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Mark Christiaens
>
> We encountered the following stack trace during high load:
> {noformat}
> Unexpected throwable object
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.CellComparator.compareRows(CellComparator.java:186)
> at
> org.apache.hadoop.hbase.CellComparator.compare(CellComparator.java:63)
> at
> org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:2021)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:202)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:178)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:168)
> at
> java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:719)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
> at java.util.PriorityQueue.poll(PriorityQueue.java:595)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:218)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:5608)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$2.close(BaseScannerRegionObserver.java:279)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$1.close(BaseScannerRegionObserver.java:186)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2378)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2034)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> What I suspect is happening is that the {{RegionScannerImpl}} is being closed
> while the scanner's lease is expired. During this close, the underlying
> {{KeyValueHeap}} is being polled. the {{heap}} tries to read data from
> {{KeyValueScanners}} that then return {{null}} which causes the crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16737) NPE during close of RegionScanner
[
https://issues.apache.org/jira/browse/HBASE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535497#comment-15535497
]
ramkrishna.s.vasudevan commented on HBASE-16737:
I think you have some phoenix coprocessor also. That lease removal is to ensure
that the scanner lease expiry does occur while scan is in process.
The scanner does not return any cell on peeking and that causes this NPE. But
why scanners are not returning a cell is something to see. It could be because
of the region observer logic also.
> NPE during close of RegionScanner
> -
>
> Key: HBASE-16737
> URL: https://issues.apache.org/jira/browse/HBASE-16737
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Mark Christiaens
>
> We encountered the following stack trace during high load:
> {noformat}
> Unexpected throwable object
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.CellComparator.compareRows(CellComparator.java:186)
> at
> org.apache.hadoop.hbase.CellComparator.compare(CellComparator.java:63)
> at
> org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:2021)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:202)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:178)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:168)
> at
> java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:719)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
> at java.util.PriorityQueue.poll(PriorityQueue.java:595)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:218)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:5608)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$2.close(BaseScannerRegionObserver.java:279)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$1.close(BaseScannerRegionObserver.java:186)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2378)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2034)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> What I suspect is happening is that the {{RegionScannerImpl}} is being closed
> while the scanner's lease is expired. During this close, the underlying
> {{KeyValueHeap}} is being polled. the {{heap}} tries to read data from
> {{KeyValueScanners}} that then return {{null}} which causes the crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16737) NPE during close of RegionScanner
[
https://issues.apache.org/jira/browse/HBASE-16737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535479#comment-15535479
]
Mark Christiaens commented on HBASE-16737:
--
I suspect that this ticket is related to HBASE-2077. Looking at the code of
{{org.apache.hadoop.hbase.regionserver.RSRpcServices#scan}}, you see that
during normal operation of the scan, the {{lease}} is removed and added again
once the scan is completed.
{noformat}
...
lease = regionServer.leases.removeLease(scannerName);
...
if (lease != null) regionServer.leases.addLease(lease);
...
{noformat}
However, the {{scanner.close()}} call occurs after re-instating the lease.
> NPE during close of RegionScanner
> -
>
> Key: HBASE-16737
> URL: https://issues.apache.org/jira/browse/HBASE-16737
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Mark Christiaens
>
> We encountered the following stack trace during high load:
> {noformat}
> Unexpected throwable object
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.CellComparator.compareRows(CellComparator.java:186)
> at
> org.apache.hadoop.hbase.CellComparator.compare(CellComparator.java:63)
> at
> org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:2021)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:202)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:178)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:168)
> at
> java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:719)
> at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
> at java.util.PriorityQueue.poll(PriorityQueue.java:595)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:218)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:5608)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$2.close(BaseScannerRegionObserver.java:279)
> at
> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$1.close(BaseScannerRegionObserver.java:186)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2378)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2034)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> What I suspect is happening is that the {{RegionScannerImpl}} is being closed
> while the scanner's lease is expired. During this close, the underlying
> {{KeyValueHeap}} is being polled. the {{heap}} tries to read data from
> {{KeyValueScanners}} that then return {{null}} which causes the crash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
