Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

[Sean Busbey]

have we experimentally confirmed that wire compatibility is
maintained? I saw one mention of expecting wire compatibility to be
fine, but nothing with someone using e.g. the clusterdock work or
something to mix servers / clients or do replication.

On Fri, Sep 30, 2016 at 6:30 PM, Stack  wrote:
> I intend to do a mass commit late this weekend that moves us on to a shaded
> protobuf-3.1.0, either Sunday night or Monday morning.
>
> If objection, please speak up or if need more time for
> consideration/review, just shout.
>
> I want to merge the branch HBASE-16264 into master (it is running here up
> on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-16264/).
> The branch at HBASE-16264 has three significant bodies-of-work that
> unfortunately are tangled and can only go in of a piece.
>
>  * HBASE-16264  The
> shading of our protobuf usage so we can upgrade and/or run with a patched
> protobuf WITHOUT breaking REST, Spark, and in particular, Coprocessor
> Endpoints.
>  * HBASE-16567  A move
> up on to (shaded) protobuf-3.1.0
>  * HBASE-16741  An
> amendment of our generate protobufs step to include shading and a bundling
> of protobuf src (with a means of calling a patch srcs hook)
>
> Together we're talking about 40MB of change mostly made of the movement of
> generated files or the application of a pattern that alters where we get
> imports from. When done, you should notice no difference and should be able
> to go about your business as per usual. Upside is that we will be able to
> avoid coming onheap doing protobuf marshalling/unmarshalling as protobuf
> 2.5.0 requires. Downside is that we repeat a good portion of our internal
> protos, once non-shaded so Coprocessor Endpoints can keep working and then
> again as shaded for internal use.
>
> I provide some more overview below on the changes. See the shading doc
> here:
> https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGbyDcXtdF5iAfDIEk/edit#
> for more detail (Patches are up on review board -- except the latest
> HBASE-16264 which is too big for JIRA and RB). I am currently working on a
> devs chapter for the book on protobuf going forward that will go in as part
> of this patch.
>
> Thanks,
> St.Ack
>
> Items of note:
>
>  * Two new modules; one named hbase-protocol-shaded that is used by hbase
> core. It has in it a shaded (and later patched) protobuf. The other new
> module is hbase-endpoint which goes after hbase-server and has those
> bundled endpoints that I was able to break out of core (there are a few
> that are hopelessly entangled that need to be undone as CPEPs but
> fortunately belong in core: Auth, Access, MultiRow).
>  * I've tested running a branch-1 CPEP against a master with these patches
> in place and stuff like ACL (A CPEP) run from the branch-1 shell work
> against the branch-2 server.
>
>
>
>
>
> On Mon, Aug 22, 2016 at 5:20 PM, Stack  wrote:
>
>> This project goes on. I updated HBASE-1563 "Shade protobuf" with some doc
>> on a final approach. We need to be able to refer to both shaded and
>> non-shaded protobuf so we can support sending HDFS old-school pb Messages
>> but also so Coprocessor Endpoints keep working though internally protobufs
>> have been relocated. Funny you should ask, but yes, there are some
>> downsides (as predicted by contributors on the JIRA). I'd be interested to
>> hear if they are too burdensome. In particular, your IDE experience gets a
>> little convoluted as you will need to add to your build path, a jar with
>> the relocated pbs. A pain.
>>
>> Thanks,
>> St.Ack
>>
>>
>> On Wed, Apr 13, 2016 at 6:09 AM, Stack  wrote:
>>
>>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey  wrote:
>>>
 On Tue, Apr 12, 2016 at 6:17 PM, Stack  wrote:
 >
 >
 > On an initial pass, the only difficult part seems to be interaction
 with
 > HDFS in asyncwal (might just pull in the HDFS messages).
 >
 >

 I have some idea how we can make this work either by pushing asyncwal
 upstream to HDFS or through some maven tricks, depending on how much
 time we have.

>>>
>>> Maven tricks? Tell us more. Here or drop a note up in the issue.
>>> Thanks Sean,
>>> St.Ack
>>>
>>
>>



-- 
busbey

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

[Andrew Purtell]

+1

Difficult work. Glad to see it landing. 

> On Sep 30, 2016, at 4:30 PM, Stack  wrote:
> 
> I intend to do a mass commit late this weekend that moves us on to a shaded
> protobuf-3.1.0, either Sunday night or Monday morning.
> 
> If objection, please speak up or if need more time for
> consideration/review, just shout.
> 
> I want to merge the branch HBASE-16264 into master (it is running here up
> on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-16264/).
> The branch at HBASE-16264 has three significant bodies-of-work that
> unfortunately are tangled and can only go in of a piece.
> 
> * HBASE-16264  The
> shading of our protobuf usage so we can upgrade and/or run with a patched
> protobuf WITHOUT breaking REST, Spark, and in particular, Coprocessor
> Endpoints.
> * HBASE-16567  A move
> up on to (shaded) protobuf-3.1.0
> * HBASE-16741  An
> amendment of our generate protobufs step to include shading and a bundling
> of protobuf src (with a means of calling a patch srcs hook)
> 
> Together we're talking about 40MB of change mostly made of the movement of
> generated files or the application of a pattern that alters where we get
> imports from. When done, you should notice no difference and should be able
> to go about your business as per usual. Upside is that we will be able to
> avoid coming onheap doing protobuf marshalling/unmarshalling as protobuf
> 2.5.0 requires. Downside is that we repeat a good portion of our internal
> protos, once non-shaded so Coprocessor Endpoints can keep working and then
> again as shaded for internal use.
> 
> I provide some more overview below on the changes. See the shading doc
> here:
> https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGbyDcXtdF5iAfDIEk/edit#
> for more detail (Patches are up on review board -- except the latest
> HBASE-16264 which is too big for JIRA and RB). I am currently working on a
> devs chapter for the book on protobuf going forward that will go in as part
> of this patch.
> 
> Thanks,
> St.Ack
> 
> Items of note:
> 
> * Two new modules; one named hbase-protocol-shaded that is used by hbase
> core. It has in it a shaded (and later patched) protobuf. The other new
> module is hbase-endpoint which goes after hbase-server and has those
> bundled endpoints that I was able to break out of core (there are a few
> that are hopelessly entangled that need to be undone as CPEPs but
> fortunately belong in core: Auth, Access, MultiRow).
> * I've tested running a branch-1 CPEP against a master with these patches
> in place and stuff like ACL (A CPEP) run from the branch-1 shell work
> against the branch-2 server.
> 
> 
> 
> 
> 
>> On Mon, Aug 22, 2016 at 5:20 PM, Stack  wrote:
>> 
>> This project goes on. I updated HBASE-1563 "Shade protobuf" with some doc
>> on a final approach. We need to be able to refer to both shaded and
>> non-shaded protobuf so we can support sending HDFS old-school pb Messages
>> but also so Coprocessor Endpoints keep working though internally protobufs
>> have been relocated. Funny you should ask, but yes, there are some
>> downsides (as predicted by contributors on the JIRA). I'd be interested to
>> hear if they are too burdensome. In particular, your IDE experience gets a
>> little convoluted as you will need to add to your build path, a jar with
>> the relocated pbs. A pain.
>> 
>> Thanks,
>> St.Ack
>> 
>> 
>>> On Wed, Apr 13, 2016 at 6:09 AM, Stack  wrote:
>>> 
 On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey  wrote:
 
> On Tue, Apr 12, 2016 at 6:17 PM, Stack  wrote:
> 
> 
> On an initial pass, the only difficult part seems to be interaction
 with
> HDFS in asyncwal (might just pull in the HDFS messages).
> 
> 
 
 I have some idea how we can make this work either by pushing asyncwal
 upstream to HDFS or through some maven tricks, depending on how much
 time we have.
 
>>> 
>>> Maven tricks? Tell us more. Here or drop a note up in the issue.
>>> Thanks Sean,
>>> St.Ack
>>> 
>> 
>> 

[jira] [Created] (HBASE-16744) Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions

[Appy (JIRA)]

Appy created HBASE-16744:


 Summary: Procedure V2 - Lock procedures to allow clients to 
acquire locks on tables/namespaces/regions
 Key: HBASE-16744
 URL: https://issues.apache.org/jira/browse/HBASE-16744
 Project: HBase
  Issue Type: New Feature
Reporter: Appy
Assignee: Appy


Will help us get rid of ZK locks.
Will be useful for external tools like hbck, future backup manager, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16743) TestSimpleRpcScheduler#testCoDelScheduling is broke

[stack (JIRA)]

stack created HBASE-16743:
-

 Summary: TestSimpleRpcScheduler#testCoDelScheduling is broke
 Key: HBASE-16743
 URL: https://issues.apache.org/jira/browse/HBASE-16743
 Project: HBase
  Issue Type: Bug
  Components: rpc
Reporter: stack


The testCoDelScheduling test is broke. Here are some notes on it. I have 
disabled it in the HBASE-15638 shading patch.

{code}
I don't get this test. When I time this test, the minDelay is > 2 * codel delay 
from the get go. So we are always overloaded. The test below would seem to 
complete the queuing of all the CallRunners inside the codel check interval. I 
don't think we are skipping codel checking. Second, I think this test has been  
broken since HBASE-16089 Add on FastPath for CoDel went in. The thread name we 
were looking for was the name BEFORE we updated: i.e. 
"RpcServer.CodelBQ.default.handler". But same patch changed the name of the 
codel  fastpath thread to: new 
FastPathBalancedQueueRpcExecutor("CodelFPBQ.default", handlerCount, 
numCallQueues...

Codel is hard to test. This test is going to be flakey given it all 
timer-based. Disabling for now till chat
{code}

FYI [~mantonov]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

[Stack]

I intend to do a mass commit late this weekend that moves us on to a shaded
protobuf-3.1.0, either Sunday night or Monday morning.

If objection, please speak up or if need more time for
consideration/review, just shout.

I want to merge the branch HBASE-16264 into master (it is running here up
on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-16264/).
The branch at HBASE-16264 has three significant bodies-of-work that
unfortunately are tangled and can only go in of a piece.

 * HBASE-16264  The
shading of our protobuf usage so we can upgrade and/or run with a patched
protobuf WITHOUT breaking REST, Spark, and in particular, Coprocessor
Endpoints.
 * HBASE-16567  A move
up on to (shaded) protobuf-3.1.0
 * HBASE-16741  An
amendment of our generate protobufs step to include shading and a bundling
of protobuf src (with a means of calling a patch srcs hook)

Together we're talking about 40MB of change mostly made of the movement of
generated files or the application of a pattern that alters where we get
imports from. When done, you should notice no difference and should be able
to go about your business as per usual. Upside is that we will be able to
avoid coming onheap doing protobuf marshalling/unmarshalling as protobuf
2.5.0 requires. Downside is that we repeat a good portion of our internal
protos, once non-shaded so Coprocessor Endpoints can keep working and then
again as shaded for internal use.

I provide some more overview below on the changes. See the shading doc
here:
https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGbyDcXtdF5iAfDIEk/edit#
for more detail (Patches are up on review board -- except the latest
HBASE-16264 which is too big for JIRA and RB). I am currently working on a
devs chapter for the book on protobuf going forward that will go in as part
of this patch.

Thanks,
St.Ack

Items of note:

 * Two new modules; one named hbase-protocol-shaded that is used by hbase
core. It has in it a shaded (and later patched) protobuf. The other new
module is hbase-endpoint which goes after hbase-server and has those
bundled endpoints that I was able to break out of core (there are a few
that are hopelessly entangled that need to be undone as CPEPs but
fortunately belong in core: Auth, Access, MultiRow).
 * I've tested running a branch-1 CPEP against a master with these patches
in place and stuff like ACL (A CPEP) run from the branch-1 shell work
against the branch-2 server.





On Mon, Aug 22, 2016 at 5:20 PM, Stack  wrote:

> This project goes on. I updated HBASE-1563 "Shade protobuf" with some doc
> on a final approach. We need to be able to refer to both shaded and
> non-shaded protobuf so we can support sending HDFS old-school pb Messages
> but also so Coprocessor Endpoints keep working though internally protobufs
> have been relocated. Funny you should ask, but yes, there are some
> downsides (as predicted by contributors on the JIRA). I'd be interested to
> hear if they are too burdensome. In particular, your IDE experience gets a
> little convoluted as you will need to add to your build path, a jar with
> the relocated pbs. A pain.
>
> Thanks,
> St.Ack
>
>
> On Wed, Apr 13, 2016 at 6:09 AM, Stack  wrote:
>
>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey  wrote:
>>
>>> On Tue, Apr 12, 2016 at 6:17 PM, Stack  wrote:
>>> >
>>> >
>>> > On an initial pass, the only difficult part seems to be interaction
>>> with
>>> > HDFS in asyncwal (might just pull in the HDFS messages).
>>> >
>>> >
>>>
>>> I have some idea how we can make this work either by pushing asyncwal
>>> upstream to HDFS or through some maven tricks, depending on how much
>>> time we have.
>>>
>>
>> Maven tricks? Tell us more. Here or drop a note up in the issue.
>> Thanks Sean,
>> St.Ack
>>
>
>

[jira] [Created] (HBASE-16742) Add chapter for devs on how we do protobufs going forward

[stack (JIRA)]

stack created HBASE-16742:
-

 Summary: Add chapter for devs on how we do protobufs going forward
 Key: HBASE-16742
 URL: https://issues.apache.org/jira/browse/HBASE-16742
 Project: HBase
  Issue Type: Sub-task
  Components: documentation
Reporter: stack
Assignee: stack


Add chapter on shaded vs non-shaded, CPEPs vs internal usage, checked-in and 
patched protobuf (3.1.0?) vs the protobuf that other components include 
(pb2.5.0).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16741) Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf

[stack (JIRA)]

stack created HBASE-16741:
-

 Summary: Amend the generate protobufs out-of-band build step to 
include shade, pulling in protobuf source and a hook for patching protobuf
 Key: HBASE-16741
 URL: https://issues.apache.org/jira/browse/HBASE-16741
 Project: HBase
  Issue Type: Sub-task
Reporter: stack
Assignee: stack


As part of the protobuf shading work, I need to amend the build step that 
builds protobuf sources. For the module used by hbase internally -- the one 
that has our protos and that does the protobuf shading -- I need to enhance the 
generate protobuf sources step to also do:

 * Shading/relocating so we avoid clashing with protos used by CPEPs out in the 
hbase-protocol module.
 * Pulling down the protobuf lib and including its sources to make IDEs happy 
else they'll moan about missing (shaded) protobuf.
 * A hook that allows us to patch protobuf lib, at least temporarily until our 
needed changes make it upstream.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16740) start-docker.sh fails to run by complaining bzip2 error

[Xiaobing Zhou (JIRA)]

Xiaobing Zhou created HBASE-16740:
-

 Summary: start-docker.sh fails to run by complaining bzip2 error
 Key: HBASE-16740
 URL: https://issues.apache.org/jira/browse/HBASE-16740
 Project: HBase
  Issue Type: Bug
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-16517) Make a 1.2.3 release

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-16517.
---
   Resolution: Fixed
 Assignee: stack
Fix Version/s: 1.2.3

Resovling. Done.

> Make a 1.2.3 release
> 
>
> Key: HBASE-16517
> URL: https://issues.apache.org/jira/browse/HBASE-16517
> Project: HBase
>  Issue Type: Umbrella
>Reporter: stack
>Assignee: stack
> Fix For: 1.2.3
>
>
> Umbrella issue under which will do all tasks related to making a 1.2.3 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16739) Timed out exception message should include encoded region name

[Ted Yu (JIRA)]

Ted Yu created HBASE-16739:
--

 Summary: Timed out exception message should include encoded region 
name
 Key: HBASE-16739
 URL: https://issues.apache.org/jira/browse/HBASE-16739
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


Saw the following in region server log repeatedly:
{code}
2016-09-26 10:13:33,219 WARN org.apache.hadoop.hbase.regionserver.HRegion: 
Failed getting lock in batch put, row=1
java.io.IOException: Timed out waiting for lock for row: 1
  at 
org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5151)
  at 
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3046)
  at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2902)
  at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2844)
  at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
  at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
  at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2040)
{code}
Region name was not logged - making troubleshooting a bit difficult.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Use hemcrest in hbase

[Sean Busbey]

BSD licensed works are fine for inclusion in ASF projects as
dependencies or 3rd party incorporation.

On Fri, Sep 30, 2016 at 1:02 PM, Apekshit Sharma  wrote:
> Hey guys,
> Hemcrest seems like a wonderful library of matchers for unit testing. Am
> sure our tests can use that improvement. Any good reason it's not widely
> used in hbase?
> I'll start using it more often unless there's a good reason not to.
> It's BSD licence, that's compatible with Apache?
>
> -- Appy



-- 
busbey

Use hemcrest in hbase

[Apekshit Sharma]

Hey guys,
Hemcrest seems like a wonderful library of matchers for unit testing. Am
sure our tests can use that improvement. Any good reason it's not widely
used in hbase?
I'll start using it more often unless there's a good reason not to.
It's BSD licence, that's compatible with Apache?

-- Appy

Successful: HBase Generate Website

[Apache Jenkins Server]

Build status: Successful

If successful, the website and docs have been generated. To update the live 
site, follow the instructions below. If failed, skip to the bottom of this 
email.

Use the following commands to download the patch and apply it to a clean branch 
based on origin/asf-site. If you prefer to keep the hbase-site repo around 
permanently, you can skip the clone step.

  git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git

  cd hbase-site
  wget -O- 
https://builds.apache.org/job/hbase_generate_website/360/artifact/website.patch.zip
 | funzip > 3757da643d43bf0eaf8a0bd4c30b56f24c95fb6c.patch
  git fetch
  git checkout -b asf-site-3757da643d43bf0eaf8a0bd4c30b56f24c95fb6c 
origin/asf-site
  git am --whitespace=fix 3757da643d43bf0eaf8a0bd4c30b56f24c95fb6c.patch

At this point, you can preview the changes by opening index.html or any of the 
other HTML pages in your local 
asf-site-3757da643d43bf0eaf8a0bd4c30b56f24c95fb6c branch.

There are lots of spurious changes, such as timestamps and CSS styles in 
tables, so a generic git diff is not very useful. To see a list of files that 
have been added, deleted, renamed, changed type, or are otherwise interesting, 
use the following command:

  git diff --name-status --diff-filter=ADCRTXUB origin/asf-site

To see only files that had 100 or more lines changed:

  git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'

When you are satisfied, publish your changes to origin/asf-site using these 
commands:

  git commit --allow-empty -m "Empty commit" # to work around a current ASF 
INFRA bug
  git push origin asf-site-3757da643d43bf0eaf8a0bd4c30b56f24c95fb6c:asf-site
  git checkout asf-site
  git branch -D asf-site-3757da643d43bf0eaf8a0bd4c30b56f24c95fb6c

Changes take a couple of minutes to be propagated. You can verify whether they 
have been propagated by looking at the Last Published date at the bottom of 
http://hbase.apache.org/. It should match the date in the index.html on the 
asf-site branch in Git.

As a courtesy- reply-all to this email to let other committers know you pushed 
the site.



If failed, see https://builds.apache.org/job/hbase_generate_website/360/console

[jira] [Created] (HBASE-16738) L1 cache caching shared memory HFile block when blocks promoted from L2 to L1

[Anoop Sam John (JIRA)]

Anoop Sam John created HBASE-16738:
--

 Summary: L1 cache caching shared memory HFile block when blocks 
promoted from L2 to L1
 Key: HBASE-16738
 URL: https://issues.apache.org/jira/browse/HBASE-16738
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 2.0.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Fix For: 2.0.0


This is an issue when L1 and L2 cache used with combinedMode = false.
See in getBlock
{code}
if (victimHandler != null && !repeat) {
Cacheable result = victimHandler.getBlock(cacheKey, caching, repeat, 
updateCacheMetrics);

// Promote this to L1.
if (result != null && caching) {
  cacheBlock(cacheKey, result, /* inMemory = */ false, /* cacheData = 
*/ true);
}
return result;
  }
{code}
When block is not there in L1 and have it in L2, we will return the block read 
from L2 and promote that block to L1 by adding it in LRUCache.  But if the 
Block buffer is having shared memory (Off heap bucket cache for eg:) , we can 
not directly cache this block. The buffer memory area under this block can get 
cleaned up at any time. So we may get block data corruption.
In such a case, we need to do a deep copy of the block (Including its buffer) 
and then add that to L1 cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16737) NPE during close of RegionScanner

[Mark Christiaens (JIRA)]

Mark Christiaens created HBASE-16737:


 Summary: NPE during close of RegionScanner
 Key: HBASE-16737
 URL: https://issues.apache.org/jira/browse/HBASE-16737
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Mark Christiaens


We encountered the following stack trace during high load:
{noformat}
Unexpected throwable object 
java.lang.NullPointerException
at 
org.apache.hadoop.hbase.CellComparator.compareRows(CellComparator.java:186)
at 
org.apache.hadoop.hbase.CellComparator.compare(CellComparator.java:63)
at 
org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:2021)
at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:202)
at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:178)
at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:168)
at 
java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:719)
at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
at java.util.PriorityQueue.poll(PriorityQueue.java:595)
at 
org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:218)
at 
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.close(HRegion.java:5608)
at 
org.apache.phoenix.coprocessor.BaseScannerRegionObserver$2.close(BaseScannerRegionObserver.java:279)
at 
org.apache.phoenix.coprocessor.BaseScannerRegionObserver$1.close(BaseScannerRegionObserver.java:186)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2378)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2034)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
{noformat}

What I suspect is happening is that the {{RegionScannerImpl}} is being closed 
while the scanner's lease is expired.  During this close, the underlying 
{{KeyValueHeap}} is being polled.  the {{heap}} tries to read data from 
{{KeyValueScanners}} that then return {{null}} which causes the crash.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)