[jira] [Commented] (HBASE-15581) Reduce garbage created by PB in write path

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222656#comment-15222656
 ] 

Anoop Sam John commented on HBASE-15581:


+1.. Which is that Jira Elliot?  I can take that up first or request some one 
help :-)


> Reduce garbage created by PB in write path
> --
>
> Key: HBASE-15581
> URL: https://issues.apache.org/jira/browse/HBASE-15581
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>
> Changes such that PB code will not make much garbage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15437) Response size calculated in RPCServer for warning tooLarge responses does NOT count CellScanner payload

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15223735#comment-15223735
 ] 

Anoop Sam John commented on HBASE-15437:


{code}
 currentCall.processingTime = processingTime;
2303currentCall.queueTime = processingTime;
{code}
Both set with same value. Copy paste issue?

Its pity that we dont pass Call to this method and so we need to do get it from 
ThreadLoal which will have slight perf overhead.  And this method is marked non 
private. 

> Response size calculated in RPCServer for warning tooLarge responses does NOT 
> count CellScanner payload
> ---
>
> Key: HBASE-15437
> URL: https://issues.apache.org/jira/browse/HBASE-15437
> Project: HBase
>  Issue Type: Bug
>  Components: IPC/RPC
>Reporter: deepankar
>Assignee: deepankar
> Attachments: HBASE-15437.patch
>
>
> After HBASE-13158 where we respond back to RPCs with cells in the payload , 
> the protobuf response will just have the count the cells to read from 
> payload, but there are set of features where we log warn in RPCServer 
> whenever the response is tooLarge, but this size now is not considering the 
> sizes of the cells in the PayloadCellScanner. Code form RPCServer
> {code}
>   long responseSize = result.getSerializedSize();
>   // log any RPC responses that are slower than the configured warn
>   // response time or larger than configured warning size
>   boolean tooSlow = (processingTime > warnResponseTime && 
> warnResponseTime > -1);
>   boolean tooLarge = (responseSize > warnResponseSize && warnResponseSize 
> > -1);
>   if (tooSlow || tooLarge) {
> // when tagging, we let TooLarge trump TooSmall to keep output simple
> // note that large responses will often also be slow.
> logResponse(new Object[]{param},
> md.getName(), md.getName() + "(" + param.getClass().getName() + 
> ")",
> (tooLarge ? "TooLarge" : "TooSlow"),
> status.getClient(), startTime, processingTime, qTime,
> responseSize);
>   }
> {code}
> Should this feature be not supported any more or should we add a method to 
> CellScanner or a new interface which returns the serialized size (but this 
> might not include the compression codecs which might be used during response 
> ?) Any other Idea this could be fixed ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15437) Response size calculated in RPCServer for warning tooLarge responses does NOT count CellScanner payload

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224754#comment-15224754
 ] 

Anoop Sam John commented on HBASE-15437:


Exposed to Phoenix so we can not simply change the signature?  Can overload 
method with taking Call

> Response size calculated in RPCServer for warning tooLarge responses does NOT 
> count CellScanner payload
> ---
>
> Key: HBASE-15437
> URL: https://issues.apache.org/jira/browse/HBASE-15437
> Project: HBase
>  Issue Type: Bug
>  Components: IPC/RPC
>Reporter: deepankar
>Assignee: deepankar
> Attachments: HBASE-15437-v1.patch, HBASE-15437.patch
>
>
> After HBASE-13158 where we respond back to RPCs with cells in the payload , 
> the protobuf response will just have the count the cells to read from 
> payload, but there are set of features where we log warn in RPCServer 
> whenever the response is tooLarge, but this size now is not considering the 
> sizes of the cells in the PayloadCellScanner. Code form RPCServer
> {code}
>   long responseSize = result.getSerializedSize();
>   // log any RPC responses that are slower than the configured warn
>   // response time or larger than configured warning size
>   boolean tooSlow = (processingTime > warnResponseTime && 
> warnResponseTime > -1);
>   boolean tooLarge = (responseSize > warnResponseSize && warnResponseSize 
> > -1);
>   if (tooSlow || tooLarge) {
> // when tagging, we let TooLarge trump TooSmall to keep output simple
> // note that large responses will often also be slow.
> logResponse(new Object[]{param},
> md.getName(), md.getName() + "(" + param.getClass().getName() + 
> ")",
> (tooLarge ? "TooLarge" : "TooSlow"),
> status.getClient(), startTime, processingTime, qTime,
> responseSize);
>   }
> {code}
> Should this feature be not supported any more or should we add a method to 
> CellScanner or a new interface which returns the serialized size (but this 
> might not include the compression codecs which might be used during response 
> ?) Any other Idea this could be fixed ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15485) Filter.reset() should not be called between batches

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15225587#comment-15225587
 ] 

Anoop Sam John commented on HBASE-15485:


Thanks for committing Ted.   Ya was away over the weekend and I totally forgot 
it yesterday.
Thanks for the patch Phil

> Filter.reset() should not be called between batches
> ---
>
> Key: HBASE-15485
> URL: https://issues.apache.org/jira/browse/HBASE-15485
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.5, 1.2.2
>
> Attachments: HBASE-15485-branch-1-v2.patch, 
> HBASE-15485-branch-1-v3.patch, HBASE-15485-branch-1.1-v1.patch, 
> HBASE-15485-branch-1.1-v2.patch, HBASE-15485-branch-1.patch, 
> HBASE-15485-v1.patch, HBASE-15485-v2.patch, HBASE-15485-v3.patch, 
> HBASE-15485-v4.patch, HBASE-15485-v5.patch
>
>
> As discussed in HBASE-15325, now we will resetFilters if partial result not 
> formed, but we should not reset filters when batch limit reached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15437) Response size calculated in RPCServer for warning tooLarge responses does NOT count CellScanner payload

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15225592#comment-15225592
 ] 

Anoop Sam John commented on HBASE-15437:


bq.@InterfaceAudience.LimitedPrivate({HBaseInterfaceAudience.COPROC, 
HBaseInterfaceAudience.PHOENIX})
As it is exposed to coproc and Phoenix,  better handle the format way of 
deprecate and overload?  We can be more formal there than checking whether 
Phoenix uses it or not?  IMHO that is better always 

> Response size calculated in RPCServer for warning tooLarge responses does NOT 
> count CellScanner payload
> ---
>
> Key: HBASE-15437
> URL: https://issues.apache.org/jira/browse/HBASE-15437
> Project: HBase
>  Issue Type: Bug
>  Components: IPC/RPC
>Reporter: deepankar
>Assignee: deepankar
> Attachments: HBASE-15437-v1.patch, HBASE-15437.patch
>
>
> After HBASE-13158 where we respond back to RPCs with cells in the payload , 
> the protobuf response will just have the count the cells to read from 
> payload, but there are set of features where we log warn in RPCServer 
> whenever the response is tooLarge, but this size now is not considering the 
> sizes of the cells in the PayloadCellScanner. Code form RPCServer
> {code}
>   long responseSize = result.getSerializedSize();
>   // log any RPC responses that are slower than the configured warn
>   // response time or larger than configured warning size
>   boolean tooSlow = (processingTime > warnResponseTime && 
> warnResponseTime > -1);
>   boolean tooLarge = (responseSize > warnResponseSize && warnResponseSize 
> > -1);
>   if (tooSlow || tooLarge) {
> // when tagging, we let TooLarge trump TooSmall to keep output simple
> // note that large responses will often also be slow.
> logResponse(new Object[]{param},
> md.getName(), md.getName() + "(" + param.getClass().getName() + 
> ")",
> (tooLarge ? "TooLarge" : "TooSlow"),
> status.getClient(), startTime, processingTime, qTime,
> responseSize);
>   }
> {code}
> Should this feature be not supported any more or should we add a method to 
> CellScanner or a new interface which returns the serialized size (but this 
> might not include the compression codecs which might be used during response 
> ?) Any other Idea this could be fixed ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15525:
---
Attachment: HBASE-15525_V1.patch

[~dvdreddy]  Can u pls help to test this in ur cluster

> OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
> --
>
> Key: HBASE-15525
> URL: https://issues.apache.org/jira/browse/HBASE-15525
> Project: HBase
>  Issue Type: Sub-task
>  Components: IPC/RPC
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Attachments: HBASE-15525_V1.patch, HBASE-15525_WIP.patch, WIP.patch
>
>
> After HBASE-13819 the system some times run out of direct memory whenever 
> there is some network congestion or some client side issues.
> This was because of pending RPCs in the RPCServer$Connection.responseQueue 
> and since all the responses in this queue hold a buffer for cellblock from 
> BoundedByteBufferPool this could takeup a lot of memory if the 
> BoundedByteBufferPool's moving average settles down towards a higher value 
> See the discussion here 
> [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15525:
---
Status: Patch Available  (was: Open)

> OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
> --
>
> Key: HBASE-15525
> URL: https://issues.apache.org/jira/browse/HBASE-15525
> Project: HBase
>  Issue Type: Sub-task
>  Components: IPC/RPC
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Attachments: HBASE-15525_V1.patch, HBASE-15525_WIP.patch, WIP.patch
>
>
> After HBASE-13819 the system some times run out of direct memory whenever 
> there is some network congestion or some client side issues.
> This was because of pending RPCs in the RPCServer$Connection.responseQueue 
> and since all the responses in this queue hold a buffer for cellblock from 
> BoundedByteBufferPool this could takeup a lot of memory if the 
> BoundedByteBufferPool's moving average settles down towards a higher value 
> See the discussion here 
> [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15525:
---
Fix Version/s: 2.0.0

> OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
> --
>
> Key: HBASE-15525
> URL: https://issues.apache.org/jira/browse/HBASE-15525
> Project: HBase
>  Issue Type: Sub-task
>  Components: IPC/RPC
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15525_V1.patch, HBASE-15525_WIP.patch, WIP.patch
>
>
> After HBASE-13819 the system some times run out of direct memory whenever 
> there is some network congestion or some client side issues.
> This was because of pending RPCs in the RPCServer$Connection.responseQueue 
> and since all the responses in this queue hold a buffer for cellblock from 
> BoundedByteBufferPool this could takeup a lot of memory if the 
> BoundedByteBufferPool's moving average settles down towards a higher value 
> See the discussion here 
> [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15594) [YCSB] Improvements

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15226634#comment-15226634
 ] 

Anoop Sam John commented on HBASE-15594:


So u are doing this test on trunk or branch-1?

> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
>  Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my 
> testing, a few handlers (100) with as many readers as I had CPUs (48), and 
> upping connections on clients to same as #cpus made for 2-3x the throughput. 
> The above config changes came of lore; which configurations need tweaking is 
> not obvious going by their names, there were no indications from the app on 
> where/why we were blocked or on which metrics are important to consider. Nor 
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable 
> to overrun a server though 8 client nodes trying to beat up a single node 
> (workloadc, all random-read, with no data returned -p  readallfields=false). 
> There is also a strange phenomenon where if I add a few machines, rather than 
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing 
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how 
> to get good numbers running YCSB. In particular, I want to be able to 
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to 
> figure client-side if they are optimal (weird is how big a difference 
> client-side changes can make -- need to fix this). On my 48 cpu machine, I 
> can do about 370k random reads a second from data totally cached in 
> bucketcache. If I short-circuit the user gets so they don't do any work but 
> return immediately, I can do 600k ops a second but the CPUs are at 60-70% 
> only. I cannot get them to go above this. Working on it.
> {code}
> 
> 
> hbase.ipc.server.read.threadpool.size
> 
> 48
> 
> 
> 
> hbase.regionserver.handler.count
> 
> 100
> 
> 
> 
> hbase.client.ipc.pool.size
> 
> 100
> 
> 
> 
> hbase.htable.threads.max
> 
> 48
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15594) [YCSB] Improvements

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15226658#comment-15226658
 ] 

Anoop Sam John commented on HBASE-15594:


Just wanted to know whether 11425 work helped wrt more CPU usage..We can 
check this later also.

> [YCSB] Improvements
> ---
>
> Key: HBASE-15594
> URL: https://issues.apache.org/jira/browse/HBASE-15594
> Project: HBase
>  Issue Type: Umbrella
>Reporter: stack
>Priority: Critical
>
> Running YCSB and getting good results is an arcane art. For example, in my 
> testing, a few handlers (100) with as many readers as I had CPUs (48), and 
> upping connections on clients to same as #cpus made for 2-3x the throughput. 
> The above config changes came of lore; which configurations need tweaking is 
> not obvious going by their names, there were no indications from the app on 
> where/why we were blocked or on which metrics are important to consider. Nor 
> was any of this stuff written down in docs.
> Even still, I am stuck trying to make use of all of the machine. I am unable 
> to overrun a server though 8 client nodes trying to beat up a single node 
> (workloadc, all random-read, with no data returned -p  readallfields=false). 
> There is also a strange phenomenon where if I add a few machines, rather than 
> 3x the YCSB throughput when 3 nodes in cluster, each machine instead is doing 
> about 1/3rd.
> This umbrella issue is to host items that improve our defaults and noting how 
> to get good numbers running YCSB. In particular, I want to be able to 
> saturate a machine.
> Here are the configs I'm currently working with. I've not done the work to 
> figure client-side if they are optimal (weird is how big a difference 
> client-side changes can make -- need to fix this). On my 48 cpu machine, I 
> can do about 370k random reads a second from data totally cached in 
> bucketcache. If I short-circuit the user gets so they don't do any work but 
> return immediately, I can do 600k ops a second but the CPUs are at 60-70% 
> only. I cannot get them to go above this. Working on it.
> {code}
> 
> 
> hbase.ipc.server.read.threadpool.size
> 
> 48
> 
> 
> 
> hbase.regionserver.handler.count
> 
> 100
> 
> 
> 
> hbase.client.ipc.pool.size
> 
> 100
> 
> 
> 
> hbase.htable.threads.max
> 
> 48
> 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15525:
---
Attachment: HBASE-15525_V2.patch

> OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
> --
>
> Key: HBASE-15525
> URL: https://issues.apache.org/jira/browse/HBASE-15525
> Project: HBase
>  Issue Type: Sub-task
>  Components: IPC/RPC
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15525_V1.patch, HBASE-15525_V2.patch, 
> HBASE-15525_WIP.patch, WIP.patch
>
>
> After HBASE-13819 the system some times run out of direct memory whenever 
> there is some network congestion or some client side issues.
> This was because of pending RPCs in the RPCServer$Connection.responseQueue 
> and since all the responses in this queue hold a buffer for cellblock from 
> BoundedByteBufferPool this could takeup a lot of memory if the 
> BoundedByteBufferPool's moving average settles down towards a higher value 
> See the discussion here 
> [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15228825#comment-15228825
 ] 

Anoop Sam John commented on HBASE-15525:


I see in test result
bq.Tests run: 102, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 128.399 
sec - in org.apache.hadoop.hbase.regionserver.TestHRegion
Seem all test cases passed and no time out
TestMasterFailoverWithProcedures -> Seems not related failures

> OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
> --
>
> Key: HBASE-15525
> URL: https://issues.apache.org/jira/browse/HBASE-15525
> Project: HBase
>  Issue Type: Sub-task
>  Components: IPC/RPC
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15525_V1.patch, HBASE-15525_V2.patch, 
> HBASE-15525_WIP.patch, WIP.patch
>
>
> After HBASE-13819 the system some times run out of direct memory whenever 
> there is some network congestion or some client side issues.
> This was because of pending RPCs in the RPCServer$Connection.responseQueue 
> and since all the responses in this queue hold a buffer for cellblock from 
> BoundedByteBufferPool this could takeup a lot of memory if the 
> BoundedByteBufferPool's moving average settles down towards a higher value 
> See the discussion here 
> [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14490) [RpcServer] reuse request read buffer

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229745#comment-15229745
 ] 

Anoop Sam John commented on HBASE-14490:


bq.For first one, we can use a fixed DirectBB in each Connection.They can reuse 
to reading first part of protocol.Just ensure
MaxSize(head + params) less than our fixed size.
So how we know the max size of head+ params?  In case the write reqs comes from 
a Java client using a codec, the Cells are part of cell block. But when the 
client do not support cell blocking (No codec and/or non java client) it will 
be part of this PB message so in the 1st part.  So sizing is not so easy.  Ya 
we can know whether the incoming req is from a cell block supported client or 
not.

In 15525, we are changing the BBBPool so as to have fixed sized BB pool.  So my 
idea is that for incoming reqs also, we can read it into N BBs from the pool.  

> [RpcServer] reuse request read buffer
> -
>
> Key: HBASE-14490
> URL: https://issues.apache.org/jira/browse/HBASE-14490
> Project: HBase
>  Issue Type: Improvement
>  Components: IPC/RPC
>Affects Versions: 2.0.0, 1.0.2
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>  Labels: performance
> Fix For: 2.0.0, 1.0.2
>
> Attachments: 14490.hack.to.1.2.patch, ByteBufferPool.java, 
> HBASE-14490-v1.patch, HBASE-14490-v10.patch, HBASE-14490-v11.patch, 
> HBASE-14490-v12.patch, HBASE-14490-v2.patch, HBASE-14490-v3.patch, 
> HBASE-14490-v4.patch, HBASE-14490-v5.patch, HBASE-14490-v6.patch, 
> HBASE-14490-v7.patch, HBASE-14490-v8.patch, HBASE-14490-v9.patch, gc.png, 
> hits.png, test-v12-patch
>
>
> Reusing buffer to read request.It's not necessary to every request free 
> buffer.The idea of optimization is to reduce the times that allocate 
> ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}}, 
> {{processConnectionHeader}} accept a ByteBuffer instead of byte[].They can 
> move {{ByteBuffer.position}} correctly when we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a buffer pool in each {{Connection}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229823#comment-15229823
 ] 

Anoop Sam John commented on HBASE-15600:


bq.Similarly we can allow to write mutations from coprocessors to memstore as 
well
So u mean write to memstore directly not via the Region -> Store -> Memstore 
path.  No that is not correct as those data written to memstore will not get 
accounted in HRegion. See
bq.private final AtomicLong memstoreSize = new AtomicLong(0);

bq.Or else we should provide the batch mutation API allow write in batch mutate 
coprocessors.
Sorry did not get u.  Within the CP hook (like preBatchMutate) we can get 
reference to Region and can call the batch method on that?  So u want that 
write should not do row locking? Already the main row is locked by now and 
there is no chance another thread locked this corresponding index row. So that 
is ok. So abt size checks? 
bq.checkResources()

> Add provision for adding mutations to memstore or able to write to same 
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
>  Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0, 0.98.19, 1.1.5, 1.2.2, 1.0.5
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region 
> from coprocessors but writing from batchMutate API is not allowed because of 
> mvcc. 
> Raised PHOENIX-2742 to discuss any alternative way to write to the same 
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set 
> wal edits in MiniBatchOperationInProgress.
> {noformat}
>   /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
>   public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
>   }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as 
> well. Or else we should provide the batch mutation API allow write in batch 
> mutate coprocessors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230055#comment-15230055
 ] 

Anoop Sam John commented on HBASE-15600:


{code}
for (int i = firstIndex; i < ( lastIndexExclusive + 
(familyMapsFromCoprocessors==null ? 0 :
3214  familyMapsFromCoprocessors.length)); i++) {
3215if (i < lastIndexExclusive && 
batchOp.retCodeDetails[i].getOperationStatusCode() != 
OperationStatusCode.NOT_RUN) {
3216  continue;
3217}
3218addedSize += applyFamilyMapToMemstore(i < lastIndexExclusive ? 
familyMaps[i]:
3219familyMapsFromCoprocessors[i-lastIndexExclusive], 
replay,
3220replay? batchOp.getReplaySequenceId(): 
writeEntry.getWriteNumber());
3221  }
{code}
Better split it to 2 loops. 1st (existing) for the familyMap and new one for 
familyMapsFromCoprocessors

One Q though
When the CP adds extra Mutations via this 
MiniBatchOperationInProgress#operationsFromCoprocessors, those also needs to be 
added to WALEdit?   Because on Mutation one can set Durability.  When we add 
this feature support, we need to do it generically.

> Add provision for adding mutations to memstore or able to write to same 
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
>  Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0, 0.98.19, 1.1.5, 1.2.2, 1.0.5
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region 
> from coprocessors but writing from batchMutate API is not allowed because of 
> mvcc. 
> Raised PHOENIX-2742 to discuss any alternative way to write to the same 
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set 
> wal edits in MiniBatchOperationInProgress.
> {noformat}
>   /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
>   public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
>   }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as 
> well. Or else we should provide the batch mutation API allow write in batch 
> mutate coprocessors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230057#comment-15230057
 ] 

Anoop Sam John commented on HBASE-15600:


Also on these new Mutations, we will not try to acquire row locks. In case of 
Phoenix local index it may be ok. But when we add this as a generic CP feature 
into HBase, we need to worry abt that also?

> Add provision for adding mutations to memstore or able to write to same 
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
>  Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0, 0.98.19, 1.1.5, 1.2.2, 1.0.5
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region 
> from coprocessors but writing from batchMutate API is not allowed because of 
> mvcc. 
> Raised PHOENIX-2742 to discuss any alternative way to write to the same 
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set 
> wal edits in MiniBatchOperationInProgress.
> {noformat}
>   /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
>   public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
>   }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as 
> well. Or else we should provide the batch mutation API allow write in batch 
> mutate coprocessors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15605) Remove PB references from HCD and HTD for 2.0

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230096#comment-15230096
 ] 

Anoop Sam John commented on HBASE-15605:


ColumnFamilySchema convertToColumnFamilySchema(byte[] name, Map 
values,
  Map conf) 
Better pass HCD to this method and return ColumnFamilySchema. The other do the 
reverse way.
convertToColumnFamilySchema, convert  -> Can the naming be consistent?
Same for HTD related 2 methods

Regarding not using the empty constructor now.  That will be no issue I 
believe.  It will do some unwanted ops but no logical issue as such I guess. 
Now the private constructor in HCD is of no use and can be removed?

> Remove PB references from HCD and HTD for 2.0
> -
>
> Key: HBASE-15605
> URL: https://issues.apache.org/jira/browse/HBASE-15605
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-15605.patch
>
>
> This task is sub-task for HBASE-15174. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15605) Remove PB references from HCD and HTD for 2.0

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230551#comment-15230551
 ] 

Anoop Sam John commented on HBASE-15605:


I was also abt to ask this.  We seems added this in 1.x 
In 2.0 we would like to see PB classes removed from public facing APIs.  So can 
we deprecate in say 1.3/1.4  and remove in 2.0?

> Remove PB references from HCD and HTD for 2.0
> -
>
> Key: HBASE-15605
> URL: https://issues.apache.org/jira/browse/HBASE-15605
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-15605.patch, HBASE-15605_1.patch
>
>
> This task is sub-task for HBASE-15174. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230576#comment-15230576
 ] 

Anoop Sam John commented on HBASE-15600:


bq. acquiredRowLocks.add(getRowLock(mutation.getRow(), true));
What if it can not acquire the lock within the time out also?  It will return 
null then.  How to handle?

When diff Mutations in a batch contain different Durability, we will go with 
highest priority for that batch.  In this evaluation these Mutations from CP 
also to be considered

{code}
if( walEditForCpOperations.getCells() != null && 
!walEditForCpOperations.getCells().isEmpty()) {
3145miniBatchOp.setWalEdit(0, walEditForCpOperations);
3146}
{code}
Why set to 0th item?   What will happen to some thing already added to this 
index.  By the time code is here, it might be already in place.

> Add provision for adding mutations to memstore or able to write to same 
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
>  Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0, 0.98.19, 1.1.5, 1.2.2, 1.0.5
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch, 
> HBASE-15600_v2.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region 
> from coprocessors but writing from batchMutate API is not allowed because of 
> mvcc. 
> Raised PHOENIX-2742 to discuss any alternative way to write to the same 
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set 
> wal edits in MiniBatchOperationInProgress.
> {noformat}
>   /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
>   public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
>   }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as 
> well. Or else we should provide the batch mutation API allow write in batch 
> mutate coprocessors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15380) Purge rollback support in Store etc.

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230592#comment-15230592
 ] 

Anoop Sam John commented on HBASE-15380:


[~saint@gmail.com]  Planning to give a new patch by removing rollback() 
from Memstore impl classes also? Or if u r busy I can do this simple task. :-)
Trying to get all the subtasks of the parent into completion. This seems very 
imp (compacting memstore) for the off heap write path work.

> Purge rollback support in Store etc.
> 
>
> Key: HBASE-15380
> URL: https://issues.apache.org/jira/browse/HBASE-15380
> Project: HBase
>  Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 15380.patch
>
>
> Rollback is no longer needed after "HBASE-15158 Change order in which we do 
> write pipeline operations; do all under row locks". Purge support. Will 
> simplify this segment work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230595#comment-15230595
 ] 

Anoop Sam John commented on HBASE-14921:


Any chance for a new patch with comments fix? Need any help ?

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15625) Make minimum values configurable and smaller

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234491#comment-15234491
 ] 

Anoop Sam John commented on HBASE-15625:


In another attempt, we want to make use of off heap memory for the bigger parts 
like BC and Memstore.  So we want to work with lower heap size only. JFYI.
For BC there can be even 0 size.  For memstore at least 5% memory is what we 
mandate as per code as of now.

> Make minimum values configurable and smaller
> 
>
> Key: HBASE-15625
> URL: https://issues.apache.org/jira/browse/HBASE-15625
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jean-Marc Spaggiari
>Assignee: Asher Bartch
>Priority: Minor
>  Labels: beginner
>
> When we start a RS, we check 
> HConstants.HBASE_CLUSTER_MINIMUM_MEMORY_THRESHOLD to make sure we always keep 
> 20% of the heap for HBase (See below). In the past maximum heap size was 
> about 20GB, which means 4GB for HBase.
> Today, with huge heaps and GC1, 20% gives a lot to HBase. Like with 80GB 
> heap, it gives 16GB, which I think it not required.
> We need to make HBASE_CLUSTER_MINIMUM_MEMORY_THRESHOLD configurable and lower 
> it's default value to 10%. It will not make any difference on any HBase 
> configuration but will allow admins to be more flexible.
> Same thing for the minimum memstore and blockcache sizes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234529#comment-15234529
 ] 

Anoop Sam John commented on HBASE-15525:


Clean way is to return the BB to pool in same place where it was obtained.  
That is why kept in the OS. To keep in BufferChain, the main thing is to know 
whether a BB came from pool or not. When the pool is not having BB now, we will 
make a new temp one.   All these details are kept in one place now. Also we 
might want the pool BBs to read reqs into also.  
We would want the users to tune the max #BBs so that there is enough off heap 
BBs in pool to handle the reqs. Ya it can so happen that on a high load case, 
it may go out of size.  But IMHO its ok to have temp byte[]s these are very 
short lived and in most chance will get cleaned in Young GC.  No need to have 
extra complex things like weak ref based another pooling.   If it comes out as 
a need later, we can handle then.  As of now keep it small and simple.

> OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
> --
>
> Key: HBASE-15525
> URL: https://issues.apache.org/jira/browse/HBASE-15525
> Project: HBase
>  Issue Type: Sub-task
>  Components: IPC/RPC
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15525_V1.patch, HBASE-15525_V2.patch, 
> HBASE-15525_WIP.patch, WIP.patch
>
>
> After HBASE-13819 the system some times run out of direct memory whenever 
> there is some network congestion or some client side issues.
> This was because of pending RPCs in the RPCServer$Connection.responseQueue 
> and since all the responses in this queue hold a buffer for cellblock from 
> BoundedByteBufferPool this could takeup a lot of memory if the 
> BoundedByteBufferPool's moving average settles down towards a higher value 
> See the discussion here 
> [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15234536#comment-15234536
 ] 

Anoop Sam John commented on HBASE-15525:


Ya actually speaking, we might not need create a full Cell block in advance. 
When it is of size 2 MB, we need 32 BBs (64 KB sized).  As and when needed we 
can encode cells to a BB.  We will need multiple Responder threads then or else 
we will block it for longer time.   (Now we have single Responder thread).. 
This is some thing to explore and may be as part of another issue?

> OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
> --
>
> Key: HBASE-15525
> URL: https://issues.apache.org/jira/browse/HBASE-15525
> Project: HBase
>  Issue Type: Sub-task
>  Components: IPC/RPC
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15525_V1.patch, HBASE-15525_V2.patch, 
> HBASE-15525_WIP.patch, WIP.patch
>
>
> After HBASE-13819 the system some times run out of direct memory whenever 
> there is some network congestion or some client side issues.
> This was because of pending RPCs in the RPCServer$Connection.responseQueue 
> and since all the responses in this queue hold a buffer for cellblock from 
> BoundedByteBufferPool this could takeup a lot of memory if the 
> BoundedByteBufferPool's moving average settles down towards a higher value 
> See the discussion here 
> [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15315) Remove always set super user call as high priority

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236554#comment-15236554
 ] 

Anoop Sam John commented on HBASE-15315:


So this jira undo the patch in HBASE-13375 completely right?

> Remove always set super user call as high priority
> --
>
> Key: HBASE-15315
> URL: https://issues.apache.org/jira/browse/HBASE-15315
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yong Zhang
>Assignee: Yong Zhang
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15315.001.patch
>
>
> Current implementation set superuser call as ADMIN_QOS, but we have many 
> customers use superuser to do normal table operation such as put/get data and 
> so on. If client put much data during region assignment, RPC from HMaster may 
> timeout because of no handle. so it is better to remove always set super user 
> call as high priority. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15607) Remove PB references from Admin for 2.0

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15237470#comment-15237470
 ] 

Anoop Sam John commented on HBASE-15607:


Ya those are user exposed APIs.

> Remove PB references from Admin for 2.0
> ---
>
> Key: HBASE-15607
> URL: https://issues.apache.org/jira/browse/HBASE-15607
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0
>
>
> This is a sub-task for HBASE-15174.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15650) Remove TimeRangeTracker as point of contention when many threads reading a StoreFile

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240648#comment-15240648
 ] 

Anoop Sam John commented on HBASE-15650:


This is excellent boss..  :-)

> Remove TimeRangeTracker as point of contention when many threads reading a 
> StoreFile
> 
>
> Key: HBASE-15650
> URL: https://issues.apache.org/jira/browse/HBASE-15650
> Project: HBase
>  Issue Type: Bug
>  Components: Performance
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 15650.patch, Point-of-contention-on-random-read.png
>
>
> HBASE-12148 is about "Remove TimeRangeTracker as point of contention when 
> many threads writing a Store". It is also a point of contention when reading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15607) Remove PB references from Admin for 2.0

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240661#comment-15240661
 ] 

Anoop Sam John commented on HBASE-15607:


bq.MasterProtos.SnapshotResponse takeSnapshotAsync
This is not changed

{code}
if (response.getCompactionState() != null) {
2265return 
Admin.CompactionState.valueOf(response.getCompactionState().toString());
2266  } 
{code}
We can keep this conversion of PB object to our POJO (and reverse) in PBUtil?

We need keep the POJOs within Admin or can it be outside?

> Remove PB references from Admin for 2.0
> ---
>
> Key: HBASE-15607
> URL: https://issues.apache.org/jira/browse/HBASE-15607
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-15607.patch, HBASE-15607_1.patch
>
>
> This is a sub-task for HBASE-15174.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15296) Break out writer and reader from StoreFile

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240745#comment-15240745
 ] 

Anoop Sam John commented on HBASE-15296:


bq.NOTE! Changes RegionObserver Coprocessor Interface so incompatible change.
With out deprecation, this breaking change is ok?

> Break out writer and reader from StoreFile
> --
>
> Key: HBASE-15296
> URL: https://issues.apache.org/jira/browse/HBASE-15296
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: HBASE-15296-branch-1.1.patch, 
> HBASE-15296-branch-1.2.patch, HBASE-15296-branch-1.patch, 
> HBASE-15296-master-v2.patch, HBASE-15296-master-v3.patch, 
> HBASE-15296-master-v4.patch, HBASE-15296-master-v5.patch, 
> HBASE-15296-master.patch
>
>
> StoreFile.java is trending to become a monolithic class, it's ~1800 lines. 
> Would it make sense to break out reader and writer (~500 lines each) into 
> separate files.
> We are doing so many different things in a single class: comparators, reader, 
> writer, other stuff; and it hurts readability a lot, to the point that just 
> reading through a piece of code require scrolling up and down to see which 
> level (reader/writer/base class level) it belongs to. These small-small 
> things really don't help while trying to understanding the code. There are 
> good reasons we don't do these often (affects existing patches, needs to be 
> done for all branches, etc). But this and a few other classes can really use 
> a single iteration of refactoring to make things a lot better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15650) Remove TimeRangeTracker as point of contention when many threads reading a StoreFile

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15242418#comment-15242418
 ] 

Anoop Sam John commented on HBASE-15650:


bq.synchronized boolean includesTimeRange(final TimeRange tr) 
We need this to be synchronized ?  

> Remove TimeRangeTracker as point of contention when many threads reading a 
> StoreFile
> 
>
> Key: HBASE-15650
> URL: https://issues.apache.org/jira/browse/HBASE-15650
> Project: HBase
>  Issue Type: Bug
>  Components: Performance
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 15650.branch-1.patch, 15650.patch, 15650.patch, 
> 15650v2.patch, 15650v3.patch, 15650v4.patch, 
> Point-of-contention-on-random-read.png
>
>
> HBASE-12148 is about "Remove TimeRangeTracker as point of contention when 
> many threads writing a Store". It is also a point of contention when reading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15650) Remove TimeRangeTracker as point of contention when many threads reading a StoreFile

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15242432#comment-15242432
 ] 

Anoop Sam John commented on HBASE-15650:


+1 Great...

{code}
 public void includeTimestamp(final Cell cell) {
92  // TODO: Why is this necessary? We already did this when we added 
the Cells to the memstore.
93  // Won't this run-through just do nothing except slow us down?
94  includeTimestamp(cell.getTimestamp());
{code}
I see this TODO added.. When Cell is added to memstore, we call 
Segment#updateMetaInfo(Cell toAdd, long s) which call this method on TRT 
passing the Cell..   So ya when we add cell to memstore, we are doing this  via 
this method.. I think no issue here and better remove this TODO also.

> Remove TimeRangeTracker as point of contention when many threads reading a 
> StoreFile
> 
>
> Key: HBASE-15650
> URL: https://issues.apache.org/jira/browse/HBASE-15650
> Project: HBase
>  Issue Type: Bug
>  Components: Performance
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 15650.branch-1.patch, 15650.patch, 15650.patch, 
> 15650v2.patch, 15650v3.patch, 15650v4.patch, 15650v5.patch, 
> Point-of-contention-on-random-read.png
>
>
> HBASE-12148 is about "Remove TimeRangeTracker as point of contention when 
> many threads writing a Store". It is also a point of contention when reading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15616) CheckAndMutate will encouter NPE if qualifier to check is null

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15243169#comment-15243169
 ] 

Anoop Sam John commented on HBASE-15616:


There are some other places in RequestConverter, we are doing this 
setQualifier(ByteStringer.wrap(qualifier)) 
See buildIncrementRequest() eg:
Some places we do have null check
if (qualifier != null) {
  permissionBuilder.setQualifier(ByteStringer.wrap(qualifier));
}
So we can do this way of not setting qualifier on PB when qualifier is null? Do 
we need pass empty string to be set?

> CheckAndMutate will encouter NPE if qualifier to check is null
> --
>
> Key: HBASE-15616
> URL: https://issues.apache.org/jira/browse/HBASE-15616
> Project: HBase
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 2.0.0
>Reporter: Jianwei Cui
>Assignee: Jianwei Cui
> Attachments: HBASE-15616-v1.patch, HBASE-15616-v2.patch
>
>
> If qualifier to check is null, the checkAndMutate/checkAndPut/checkAndDelete 
> will encounter NPE.
> The test code:
> {code}
> table.checkAndPut(row, family, null, Bytes.toBytes(0), new 
> Put(row).addColumn(family, null, Bytes.toBytes(1)));
> {code}
> The exception:
> {code}
> Exception in thread "main" 
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
> attempts=3, exceptions:
> Fri Apr 08 15:51:31 CST 2016, 
> RpcRetryingCaller{globalStartTime=1460101891615, pause=100, maxAttempts=3}, 
> java.io.IOException: com.google.protobuf.ServiceException: 
> java.lang.NullPointerException
> Fri Apr 08 15:51:31 CST 2016, 
> RpcRetryingCaller{globalStartTime=1460101891615, pause=100, maxAttempts=3}, 
> java.io.IOException: com.google.protobuf.ServiceException: 
> java.lang.NullPointerException
> Fri Apr 08 15:51:32 CST 2016, 
> RpcRetryingCaller{globalStartTime=1460101891615, pause=100, maxAttempts=3}, 
> java.io.IOException: com.google.protobuf.ServiceException: 
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:120)
>   at org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:772)
>   at ...
> Caused by: java.io.IOException: com.google.protobuf.ServiceException: 
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:341)
>   at org.apache.hadoop.hbase.client.HTable$7.call(HTable.java:768)
>   at org.apache.hadoop.hbase.client.HTable$7.call(HTable.java:755)
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:99)
>   ... 2 more
> Caused by: com.google.protobuf.ServiceException: 
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:239)
>   at 
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.mutate(ClientProtos.java:35252)
>   at org.apache.hadoop.hbase.client.HTable$7.call(HTable.java:765)
>   ... 4 more
> Caused by: java.lang.NullPointerException
>   at com.google.protobuf.LiteralByteString.size(LiteralByteString.java:76)
>   at 
> com.google.protobuf.CodedOutputStream.computeBytesSizeNoTag(CodedOutputStream.java:767)
>   at 
> com.google.protobuf.CodedOutputStream.computeBytesSize(CodedOutputStream.java:539)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$Condition.getSerializedSize(ClientProtos.java:7483)
>   at 
> com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:749)
>   at 
> com.google.protobuf.CodedOutputStream.computeMessageSize(CodedOutputStream.java:530)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutateRequest.getSerializedSize(ClientProtos.java:12431)
>   at 
> org.apache.hadoop.hbase.ipc.IPCUtil.getTotalSizeWhenWrittenDelimited(IPCUtil.java:311)
>   at 
> org.apache.hadoop.hbase.ipc.AsyncRpcChannel.writeRequest(AsyncRpcChannel.java:409)
>   at 
> org.apache.hadoop.hbase.ipc.AsyncRpcChannel.callMethod(AsyncRpcChannel.java:333)
>   at 
> org.apache.hadoop.hbase.ipc.AsyncRpcClient.call(AsyncRpcClient.java:245)
>   at 
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:226)
>   ... 7 more
> {code}
> The reason is {{LiteralByteString.size()}} will throw NPE if wrapped byte 
> array is null. It is possible to invoke {{put}} and {{checkAndMutate}} on the 
> same column, because null qualifier is allowed for {{Put}},  users may be 
> confused if null qualifier is not allowed 

[jira] [Commented] (HBASE-14920) Compacting Memstore

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245400#comment-15245400
 ] 

Anoop Sam John commented on HBASE-14920:


Excellent.  Ya Specially when the key and/or value size of cells are not much, 
the heap size overhead because of Cell and CSLM node is considerably high.  
This will 100 bytes per cell. So when data size is like 100-200  bytes we will 
have flushes (128 MB memstore size reached) with actual data getting flushed is 
only half of that size.  And in memory flush/compaction which result in kind of 
flat bytes representation (with overhead of 8 bytes per Cell for meta data) 
looks much better.  This will help us in our off heap write path and memstore 
impl also.  Waiting for this patch to land in.

 bq.How do you like this solution? Better than the current?
This mean per region one pool right?  

> Compacting Memstore
> ---
>
> Key: HBASE-14920
> URL: https://issues.apache.org/jira/browse/HBASE-14920
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-14920-V01.patch, HBASE-14920-V02.patch, 
> move.to.junit4.patch
>
>
> Implementation of a new compacting memstore with non-optimized immutable 
> segment representation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245522#comment-15245522
 ] 

Anoop Sam John commented on HBASE-14921:


It will be bad, if we have to continue with Cell ref in case of flat map (No 
extra copy to new chunks as not much of the cells getting expired/removed).  
Cells having ref to chunk data (byte[] now).  Can we make the meta data here as 
ref + offset  ( 8 = 4 = 12 bytes per Cell)..Ya it is 4 bytes more but its ok 
and better than 40 bytes per cell overhead.   We need to mark the Cells created 
out of copy to MSLAB in a special way so as to retrieve the byte[] ref.
Also one more thing is in case of Cells in CSLM, the seqId is a state (long) on 
Cell object.  That will be an issue in above approach.  So when the Cell is 
recreated after copying to MSLAB (mayCloneWithAllocator),  we can keep append 
the seqId 8 bytes after Cell key, value , [tags].

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-14921) Memory optimizations

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245522#comment-15245522
 ] 

Anoop Sam John edited comment on HBASE-14921 at 4/18/16 11:55 AM:
--

It will be bad, if we have to continue with Cell ref in case of flat map (No 
extra copy to new chunks as not much of the cells getting expired/removed).  
Cells having ref to chunk data (byte[] now).  Can we make the meta data here as 
ref + offset  ( 8 = 4 = 12 bytes per Cell)..Ya it is 4 bytes more but its ok 
and better than 40 bytes per cell overhead.   We need to mark the Cells created 
out of copy to MSLAB in a special way so as to retrieve the byte[] ref.
Also one more thing is in case of Cells in CSLM, the seqId is a state (long) on 
Cell object.  That will be an issue in above approach.  So when the Cell is 
recreated after copying to MSLAB (mayCloneWithAllocator),  we can keep append 
the seqId 8 bytes after Cell key, value , [tags].

Am I making the explanation here?  Any help needed, I can do. :-)


was (Author: anoop.hbase):
It will be bad, if we have to continue with Cell ref in case of flat map (No 
extra copy to new chunks as not much of the cells getting expired/removed).  
Cells having ref to chunk data (byte[] now).  Can we make the meta data here as 
ref + offset  ( 8 = 4 = 12 bytes per Cell)..Ya it is 4 bytes more but its ok 
and better than 40 bytes per cell overhead.   We need to mark the Cells created 
out of copy to MSLAB in a special way so as to retrieve the byte[] ref.
Also one more thing is in case of Cells in CSLM, the seqId is a state (long) on 
Cell object.  That will be an issue in above approach.  So when the Cell is 
recreated after copying to MSLAB (mayCloneWithAllocator),  we can keep append 
the seqId 8 bytes after Cell key, value , [tags].

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15513) hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247158#comment-15247158
 ] 

Anoop Sam John commented on HBASE-15513:


It is like fixed at the init time. Say we have 10GB max heap size and 40% as 
memstore size (ie. 4GB) and if hbase.hregion.memstore.chunkpool.maxsize = 1, 
then 4 GB is reserved for MSLAB pool.  So it is not changing with memstore size.
Ya when the auto tuner changes the memstore size (up or down), this chunk size 
wont get changed too.. May be that is some thing wrong and to be corrected?

> hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default
> --
>
> Key: HBASE-15513
> URL: https://issues.apache.org/jira/browse/HBASE-15513
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-15513-v1.patch
>
>
> That results in excessive MemStoreLAB chunk allocations because we can not 
> reuse them. Not sure, why it has been disabled, by default. May be the code 
> has not been tested well?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15544) HFileReaderImpl.readBlock allocates memory for HFileBlock on every call

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247168#comment-15247168
 ] 

Anoop Sam John commented on HBASE-15544:


So this will keep buffers in pool permanently. What will be max size of all 
such buffers?  Running it for a long time do you see improvement wrt collective 
GC pause time and perf?

> HFileReaderImpl.readBlock allocates memory for HFileBlock on every call
> ---
>
> Key: HBASE-15544
> URL: https://issues.apache.org/jira/browse/HBASE-15544
> Project: HBase
>  Issue Type: Sub-task
>  Components: Compaction
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-15544-v1.patch, HBASE-15544-v2.patch
>
>
> This accounts for 25% of memory allocation during compaction when compression 
> is enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15672) hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247222#comment-15247222
 ] 

Anoop Sam John commented on HBASE-15672:


Checking it now.

> hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails
> --
>
> Key: HBASE-15672
> URL: https://issues.apache.org/jira/browse/HBASE-15672
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Vladimir Rodionov
>
> 016-04-18 15:02:50,632 WARN  [member: '10.22.11.177,55156,1461016964801' 
> subprocedure-pool5-thread-1] 
> flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(275): 
> Got Exception in FlushSubprocedurePool
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
> Timestamp cannot be negative. minStamp:-9223372036854775808, maxStamp:128
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>   at 
> org.apache.hadoop.hbase.procedure.flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool.waitForOutstandingTasks(RegionServerFlushTableProcedureManager.java:251)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.flushRegions(FlushTableSubprocedure.java:102)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.acquireBarrier(FlushTableSubprocedure.java:113)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:-9223372036854775808, maxStamp:128
>   at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:81)
>   at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:204)
>   at 
> org.apache.hadoop.hbase.regionserver.ImmutableSegment.(ImmutableSegment.java:44)
>   at 
> org.apache.hadoop.hbase.regionserver.SegmentFactory.createImmutableSegment(SegmentFactory.java:57)
>   at 
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.snapshot(DefaultMemStore.java:93)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.prepare(HStore.java:2151)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2324)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2192)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2163)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2054)
>   at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1980)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:68)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15672) hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247274#comment-15247274
 ] 

Anoop Sam John commented on HBASE-15672:


This is because of an issue in VC#prePrepareTimeStampForDeleteVersion.  It was 
this way from long time.  May be because of the latest changes in handling TTR 
and TR for immutable and mutable sets it got visible now.



> hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails
> --
>
> Key: HBASE-15672
> URL: https://issues.apache.org/jira/browse/HBASE-15672
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Vladimir Rodionov
>
> 016-04-18 15:02:50,632 WARN  [member: '10.22.11.177,55156,1461016964801' 
> subprocedure-pool5-thread-1] 
> flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(275): 
> Got Exception in FlushSubprocedurePool
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
> Timestamp cannot be negative. minStamp:-9223372036854775808, maxStamp:128
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>   at 
> org.apache.hadoop.hbase.procedure.flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool.waitForOutstandingTasks(RegionServerFlushTableProcedureManager.java:251)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.flushRegions(FlushTableSubprocedure.java:102)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.acquireBarrier(FlushTableSubprocedure.java:113)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:-9223372036854775808, maxStamp:128
>   at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:81)
>   at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:204)
>   at 
> org.apache.hadoop.hbase.regionserver.ImmutableSegment.(ImmutableSegment.java:44)
>   at 
> org.apache.hadoop.hbase.regionserver.SegmentFactory.createImmutableSegment(SegmentFactory.java:57)
>   at 
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.snapshot(DefaultMemStore.java:93)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.prepare(HStore.java:2151)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2324)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2192)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2163)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2054)
>   at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1980)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:68)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15672) hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15672:
---
Status: Patch Available  (was: Open)

> hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails
> --
>
> Key: HBASE-15672
> URL: https://issues.apache.org/jira/browse/HBASE-15672
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Vladimir Rodionov
> Attachments: HBASE-15672.patch
>
>
> 016-04-18 15:02:50,632 WARN  [member: '10.22.11.177,55156,1461016964801' 
> subprocedure-pool5-thread-1] 
> flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(275): 
> Got Exception in FlushSubprocedurePool
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
> Timestamp cannot be negative. minStamp:-9223372036854775808, maxStamp:128
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>   at 
> org.apache.hadoop.hbase.procedure.flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool.waitForOutstandingTasks(RegionServerFlushTableProcedureManager.java:251)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.flushRegions(FlushTableSubprocedure.java:102)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.acquireBarrier(FlushTableSubprocedure.java:113)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:-9223372036854775808, maxStamp:128
>   at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:81)
>   at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:204)
>   at 
> org.apache.hadoop.hbase.regionserver.ImmutableSegment.(ImmutableSegment.java:44)
>   at 
> org.apache.hadoop.hbase.regionserver.SegmentFactory.createImmutableSegment(SegmentFactory.java:57)
>   at 
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.snapshot(DefaultMemStore.java:93)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.prepare(HStore.java:2151)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2324)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2192)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2163)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2054)
>   at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1980)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:68)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15672) hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15672:
---
Attachment: HBASE-15672.patch

> hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails
> --
>
> Key: HBASE-15672
> URL: https://issues.apache.org/jira/browse/HBASE-15672
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Vladimir Rodionov
> Attachments: HBASE-15672.patch
>
>
> 016-04-18 15:02:50,632 WARN  [member: '10.22.11.177,55156,1461016964801' 
> subprocedure-pool5-thread-1] 
> flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(275): 
> Got Exception in FlushSubprocedurePool
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
> Timestamp cannot be negative. minStamp:-9223372036854775808, maxStamp:128
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>   at 
> org.apache.hadoop.hbase.procedure.flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool.waitForOutstandingTasks(RegionServerFlushTableProcedureManager.java:251)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.flushRegions(FlushTableSubprocedure.java:102)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.acquireBarrier(FlushTableSubprocedure.java:113)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:-9223372036854775808, maxStamp:128
>   at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:81)
>   at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:204)
>   at 
> org.apache.hadoop.hbase.regionserver.ImmutableSegment.(ImmutableSegment.java:44)
>   at 
> org.apache.hadoop.hbase.regionserver.SegmentFactory.createImmutableSegment(SegmentFactory.java:57)
>   at 
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.snapshot(DefaultMemStore.java:93)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.prepare(HStore.java:2151)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2324)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2192)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2163)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2054)
>   at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1980)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:68)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-15672) hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John reassigned HBASE-15672:
--

Assignee: Anoop Sam John

> hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails
> --
>
> Key: HBASE-15672
> URL: https://issues.apache.org/jira/browse/HBASE-15672
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Vladimir Rodionov
>Assignee: Anoop Sam John
> Attachments: HBASE-15672.patch
>
>
> 016-04-18 15:02:50,632 WARN  [member: '10.22.11.177,55156,1461016964801' 
> subprocedure-pool5-thread-1] 
> flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(275): 
> Got Exception in FlushSubprocedurePool
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
> Timestamp cannot be negative. minStamp:-9223372036854775808, maxStamp:128
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>   at 
> org.apache.hadoop.hbase.procedure.flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool.waitForOutstandingTasks(RegionServerFlushTableProcedureManager.java:251)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.flushRegions(FlushTableSubprocedure.java:102)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.acquireBarrier(FlushTableSubprocedure.java:113)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:-9223372036854775808, maxStamp:128
>   at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:81)
>   at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:204)
>   at 
> org.apache.hadoop.hbase.regionserver.ImmutableSegment.(ImmutableSegment.java:44)
>   at 
> org.apache.hadoop.hbase.regionserver.SegmentFactory.createImmutableSegment(SegmentFactory.java:57)
>   at 
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.snapshot(DefaultMemStore.java:93)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.prepare(HStore.java:2151)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2324)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2192)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2163)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2054)
>   at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1980)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:68)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15672) hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15672:
---
Affects Version/s: 0.99.0
   0.98.4

> hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails
> --
>
> Key: HBASE-15672
> URL: https://issues.apache.org/jira/browse/HBASE-15672
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.99.0, 0.98.4
>Reporter: Vladimir Rodionov
>Assignee: Anoop Sam John
> Attachments: HBASE-15672.patch
>
>
> 016-04-18 15:02:50,632 WARN  [member: '10.22.11.177,55156,1461016964801' 
> subprocedure-pool5-thread-1] 
> flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(275): 
> Got Exception in FlushSubprocedurePool
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
> Timestamp cannot be negative. minStamp:-9223372036854775808, maxStamp:128
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>   at 
> org.apache.hadoop.hbase.procedure.flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool.waitForOutstandingTasks(RegionServerFlushTableProcedureManager.java:251)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.flushRegions(FlushTableSubprocedure.java:102)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.acquireBarrier(FlushTableSubprocedure.java:113)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:-9223372036854775808, maxStamp:128
>   at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:81)
>   at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:204)
>   at 
> org.apache.hadoop.hbase.regionserver.ImmutableSegment.(ImmutableSegment.java:44)
>   at 
> org.apache.hadoop.hbase.regionserver.SegmentFactory.createImmutableSegment(SegmentFactory.java:57)
>   at 
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.snapshot(DefaultMemStore.java:93)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.prepare(HStore.java:2151)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2324)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2192)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2163)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2054)
>   at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1980)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:68)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15672) hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249234#comment-15249234
 ] 

Anoop Sam John commented on HBASE-15672:


Thanks for committing this Stack.
Ya at least in this case of VC, there is no need to add Cell with -ve 
timestamp.  

> hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails
> --
>
> Key: HBASE-15672
> URL: https://issues.apache.org/jira/browse/HBASE-15672
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.99.0, 0.98.4
>Reporter: Vladimir Rodionov
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15672.patch, HBASE-15672.patch
>
>
> 016-04-18 15:02:50,632 WARN  [member: '10.22.11.177,55156,1461016964801' 
> subprocedure-pool5-thread-1] 
> flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(275): 
> Got Exception in FlushSubprocedurePool
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
> Timestamp cannot be negative. minStamp:-9223372036854775808, maxStamp:128
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>   at 
> org.apache.hadoop.hbase.procedure.flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool.waitForOutstandingTasks(RegionServerFlushTableProcedureManager.java:251)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.flushRegions(FlushTableSubprocedure.java:102)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.acquireBarrier(FlushTableSubprocedure.java:113)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:-9223372036854775808, maxStamp:128
>   at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:81)
>   at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:204)
>   at 
> org.apache.hadoop.hbase.regionserver.ImmutableSegment.(ImmutableSegment.java:44)
>   at 
> org.apache.hadoop.hbase.regionserver.SegmentFactory.createImmutableSegment(SegmentFactory.java:57)
>   at 
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.snapshot(DefaultMemStore.java:93)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.prepare(HStore.java:2151)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2324)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2192)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2163)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2054)
>   at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1980)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:68)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15672) hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249236#comment-15249236
 ] 

Anoop Sam John commented on HBASE-15672:


We have to commit this to all branches where HBASE-15650 went in right?

> hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails
> --
>
> Key: HBASE-15672
> URL: https://issues.apache.org/jira/browse/HBASE-15672
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.99.0, 0.98.4
>Reporter: Vladimir Rodionov
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15672.patch, HBASE-15672.patch
>
>
> 016-04-18 15:02:50,632 WARN  [member: '10.22.11.177,55156,1461016964801' 
> subprocedure-pool5-thread-1] 
> flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(275): 
> Got Exception in FlushSubprocedurePool
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
> Timestamp cannot be negative. minStamp:-9223372036854775808, maxStamp:128
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>   at 
> org.apache.hadoop.hbase.procedure.flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool.waitForOutstandingTasks(RegionServerFlushTableProcedureManager.java:251)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.flushRegions(FlushTableSubprocedure.java:102)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.acquireBarrier(FlushTableSubprocedure.java:113)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:-9223372036854775808, maxStamp:128
>   at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:81)
>   at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:204)
>   at 
> org.apache.hadoop.hbase.regionserver.ImmutableSegment.(ImmutableSegment.java:44)
>   at 
> org.apache.hadoop.hbase.regionserver.SegmentFactory.createImmutableSegment(SegmentFactory.java:57)
>   at 
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.snapshot(DefaultMemStore.java:93)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.prepare(HStore.java:2151)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2324)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2192)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2163)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2054)
>   at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1980)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:68)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249241#comment-15249241
 ] 

Anoop Sam John commented on HBASE-14921:


For the seqId I mean this
{code}
public Cell maybeCloneWithAllocator(Cell cell) {
if (getMemStoreLAB() == null) {
  return cell;
}

int len = KeyValueUtil.length(cell);
ByteRange alloc = getMemStoreLAB().allocateBytes(len);
if (alloc == null) {
  // The allocation was too large, allocator decided
  // not to do anything with it.
  return cell;
}
assert alloc.getBytes() != null;
KeyValueUtil.appendToByteArray(cell, alloc.getBytes(), alloc.getOffset());
KeyValue newKv = new KeyValue(alloc.getBytes(), alloc.getOffset(), len);
newKv.setSequenceId(cell.getSequenceId());
return newKv;
  }
{code}
So the seqId is a long state on the Cell object. (We make KeyValue only here)
What I was saying is we will have a new impl for Cell which dont have seqId 
state. We will ask Allocator to give cell length + 8 bytes and keep the seqId 
as the last 8 bytes after the cell key and value part.  So when off heap MSLAB 
in place, we can keep these 8 bytes also off heap.   And then this can be a 
special cell which can return the ref to MSLAB chunk so that when we flatten 
this CSLM as part of in memory flush/compaction, we dont have to keep the Cell 
object as such and we can get ref to MSLAB chunk and offset there.   Or may be 
we can even have this Cell to have the chunkId too..  That will make us same 
way as 8 bytes per cell meta data. (chunkId + offset)

Any way we can do those as a follow up issue.

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249291#comment-15249291
 ] 

Anoop Sam John commented on HBASE-15669:


bq.stores.get(j).getStoreFileSize();
Do we need a check like hasStoreFileSize()? getStoreFileSize(): 0?

int totalEdits = edit.size();
752   for (int i = 0; i < totalEdits; i++) {
totalEdits? This is cells in this edit right?  In loop condition part u can 
have i < cells.size()?  Other places also similar way.  Will it add more burden 
on other normal edits size calc?  Like we have qualifier check on each and 
every cell.
There can be one WALEdit with a mix of bulk load cells + normal cells?  I dont 
think so.  So we can early out when 1st cell in WALEdit is not a bulk load 
cell?  May be this optimization can come in some other places also?

Can we mark the WALEdit itself as bulk load related? Am not sure.. May be see 
that later.


> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch
>
>
> In a single replication request from source cluster a RS can send either at 
> most {{replication.source.size.capacity}} size of data or 
> {{replication.source.nb.capacity}} entries. 
> The size is calculated by considering the cells size in each entry which will 
> get calculated wrongly in case of bulk loaded data replication, in this case 
> we need to consider the size of hfiles not cell.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249567#comment-15249567
 ] 

Anoop Sam John commented on HBASE-15669:


bq.The same thing came up when we were working on the main jira (HBASE-13153), 
but we are not sure if in future an edit can contain a mix of mutation and bulk 
load marker cells. If that happens then it will break the replication. So to 
avoid that we are handling it in that way
How that will be possible really?  I dont think so..  It will be of great help 
while handling the normal WALEdits (for normal writes).  That may contain many 
cells and the Q check can be limited to 1 per edit. Now it will be so many 
compares to handle the bulk load replication.  We even had a boolean to enable 
the bulk load replication right?  Even that check also not done here?We 
should not be adding so many unwanted compare ops.

> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch
>
>
> In a single replication request from source cluster a RS can send either at 
> most {{replication.source.size.capacity}} size of data or 
> {{replication.source.nb.capacity}} entries. 
> The size is calculated by considering the cells size in each entry which will 
> get calculated wrongly in case of bulk loaded data replication, in this case 
> we need to consider the size of hfiles not cell.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249568#comment-15249568
 ] 

Anoop Sam John commented on HBASE-15669:


bq.By default its 0.
It is a PB object right? If not set, the getter may throw exception. Yes we are 
setting it.. I was just asking to add like a safe guard.

> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch
>
>
> In a single replication request from source cluster a RS can send either at 
> most {{replication.source.size.capacity}} size of data or 
> {{replication.source.nb.capacity}} entries. 
> The size is calculated by considering the cells size in each entry which will 
> get calculated wrongly in case of bulk loaded data replication, in this case 
> we need to consider the size of hfiles not cell.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15669) HFile size is not considered correctly in a replication request

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249581#comment-15249581
 ] 

Anoop Sam John commented on HBASE-15669:


bq.Shall I do that as part of another jira as I know there are many places 
where we can do such kind off optimization for bulk loaded data replication
Ya that will be better and handle it in all places.

> HFile size is not considered correctly in a replication request
> ---
>
> Key: HBASE-15669
> URL: https://issues.apache.org/jira/browse/HBASE-15669
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15669.patch
>
>
> In a single replication request from source cluster a RS can send either at 
> most {{replication.source.size.capacity}} size of data or 
> {{replication.source.nb.capacity}} entries. 
> The size is calculated by considering the cells size in each entry which will 
> get calculated wrongly in case of bulk loaded data replication, in this case 
> we need to consider the size of hfiles not cell.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15672) hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249612#comment-15249612
 ] 

Anoop Sam John commented on HBASE-15672:


You are right.  Just pushed to addendum to 0.98.  Should be fine now..  Thanks 
for noticing it [~Apache9]

> hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails
> --
>
> Key: HBASE-15672
> URL: https://issues.apache.org/jira/browse/HBASE-15672
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Affects Versions: 0.99.0, 0.98.4
>Reporter: Vladimir Rodionov
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0
>
> Attachments: HBASE-15672.patch, HBASE-15672.patch
>
>
> 016-04-18 15:02:50,632 WARN  [member: '10.22.11.177,55156,1461016964801' 
> subprocedure-pool5-thread-1] 
> flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(275): 
> Got Exception in FlushSubprocedurePool
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
> Timestamp cannot be negative. minStamp:-9223372036854775808, maxStamp:128
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>   at 
> org.apache.hadoop.hbase.procedure.flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool.waitForOutstandingTasks(RegionServerFlushTableProcedureManager.java:251)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.flushRegions(FlushTableSubprocedure.java:102)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.acquireBarrier(FlushTableSubprocedure.java:113)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:166)
>   at 
> org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Timestamp cannot be negative. 
> minStamp:-9223372036854775808, maxStamp:128
>   at org.apache.hadoop.hbase.io.TimeRange.(TimeRange.java:81)
>   at 
> org.apache.hadoop.hbase.regionserver.TimeRangeTracker.toTimeRange(TimeRangeTracker.java:204)
>   at 
> org.apache.hadoop.hbase.regionserver.ImmutableSegment.(ImmutableSegment.java:44)
>   at 
> org.apache.hadoop.hbase.regionserver.SegmentFactory.createImmutableSegment(SegmentFactory.java:57)
>   at 
> org.apache.hadoop.hbase.regionserver.DefaultMemStore.snapshot(DefaultMemStore.java:93)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.prepare(HStore.java:2151)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2324)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2192)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2163)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2054)
>   at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1980)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:68)
>   at 
> org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure$RegionFlushTask.call(FlushTableSubprocedure.java:1)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15381) Implement a distributed MOB compaction by procedure

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15251517#comment-15251517
 ] 

Anoop Sam John commented on HBASE-15381:


There can be N partition group for one region. Pls mention whether these N 
compact ops will be serial or parallel and how those gets executed using a 
thread pool.
Also pls mention E2E flow like how we will update the ref cells with new path.

> Implement a distributed MOB compaction by procedure
> ---
>
> Key: HBASE-15381
> URL: https://issues.apache.org/jira/browse/HBASE-15381
> Project: HBase
>  Issue Type: Improvement
>  Components: mob
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HBASE-15381-v2.patch, HBASE-15381.patch, mob distributed 
> compaction design-v2.pdf, mob distributed compaction design.pdf
>
>
> In MOB, there is a periodical compaction which runs in HMaster (It can be 
> disabled by configuration), some small mob files are merged into bigger ones. 
> Now the compaction only runs in HMaster which is not efficient and might 
> impact the running of HMaster. In this JIRA, a distributed MOB compaction is 
> introduced, it is triggered by HMaster, but all the compaction jobs are 
> distributed to HRegionServers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15686) unable to dynamically load a table coprocessor if it belongs in org.apache.hadoop

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253391#comment-15253391
 ] 

Anoop Sam John commented on HBASE-15686:


Ping [~enis]

> unable to dynamically load a table coprocessor if it belongs in 
> org.apache.hadoop
> -
>
> Key: HBASE-15686
> URL: https://issues.apache.org/jira/browse/HBASE-15686
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors
>Affects Versions: 1.0.1
>Reporter: Sangjin Lee
> Attachments: 15686.wip
>
>
> As part of Hadoop's Timeline Service v.2 (YARN-2928), we're adding a table 
> coprocessor (YARN-4062). However, we're finding that the coprocessor cannot 
> be loaded dynamically. A relevant snippet for the exception:
> {noformat}
> java.io.IOException: Class 
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor 
> cannot be loaded
> at 
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1329)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1269)
> at 
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398)
> at 
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Class 
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor 
> cannot be loaded
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:324)
> at 
> org.apache.hadoop.hbase.master.HMaster.checkClassLoading(HMaster.java:1483)
> at 
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1327)
> ... 8 more
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at 
> org.apache.hadoop.hbase.util.CoprocessorClassLoader.loadClass(CoprocessorClassLoader.java:275)
> at 
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:322)
> ... 10 more
> {noformat}
> We tracked it down to the fact that {{CoprocessorClassLoader}} regarding all 
> hadoop classes as exempt from loading from the coprocessor jar. Since our 
> coprocessor sits in the coprocessor jar, and yet the loading of this class is 
> delegated to the parent which does not have this jar, the classloading fails.
> What would be nice is the ability to exclude certain classes from the exempt 
> classes so that they can be loaded via table coprocessor classloader. See 
> hadoop's {{ApplicationClassLoader}} for a similar feature.
> Is there any other way to load this coprocessor at the table scope?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14920) Compacting Memstore

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253829#comment-15253829
 ] 

Anoop Sam John commented on HBASE-14920:


bq.public final static double IN_MEMORY_FLUSH_THRESHOLD_FACTOR = 0.9;

So we check after every cell addition to the active segment, whether it is 
worth for an in memory flush now.  The size calc for that , why we consider 
FlushLargeStoresPolicy.DEFAULT_HREGION_COLUMNFAMILY_FLUSH_SIZE_LOWER_BOUND_MIN 
and then multiply that with this factor of 90%?
FlushLargeStoresPolicy#configureForRegion  sets a bound for each memstore by
{code}
protected void configureForRegion(HRegion region) {
super.configureForRegion(region);
int familyNumber = region.getTableDesc().getFamilies().size();
if (familyNumber <= 1) {
  // No need to parse and set flush size lower bound if only one family
  // Family number might also be zero in some of our unit test case
  return;
}
// For multiple families, lower bound is the "average flush size" by default
// unless setting in configuration is larger.
long flushSizeLowerBound = region.getMemstoreFlushSize() / familyNumber;
long minimumLowerBound =
getConf().getLong(HREGION_COLUMNFAMILY_FLUSH_SIZE_LOWER_BOUND_MIN,
  DEFAULT_HREGION_COLUMNFAMILY_FLUSH_SIZE_LOWER_BOUND_MIN);
if (minimumLowerBound > flushSizeLowerBound) {
  flushSizeLowerBound = minimumLowerBound;
}
{code}

Can we simplify our calc like we get avg size for each memstore size when a 
normal flush (ie.  Memstore size , def 128 MB  / #stores)  and multiply that 
with a factor for deciding the in memory flush.   Table have 2 stores. So avg 
max size for each memstore is 64 MB.  And we keep a factor of say 25% . So when 
memstore size reaches 16 MB, we do an in memory flush.

Another concern is when a flush request comes (It can be because of global 
memstore size above high or lower watermark or because of region memstore size 
reaches limit, def 128MB  or because of an explicit flush call from user via 
API ), why we flush to disk only some part?  Only the tail of pipeline.   IMHO, 
when a to disk flush request comes, we must flush whole memstore.
In case of flush because of lower/higher water mark crossed, we pick up regions 
for flush n increasing order of region memstore size.  This size includes all 
segment's size.  And we may end up in flushing much lesser size!

Another thing on general is we account the memstore size in many places now..  
RS level, Region level as state vars.  And within the memstore it has a size.  
Now with all the in memory flush, the size changes after an in memory flush. I 
see we have a call via RegionServicesForStores.  But all these make us more 
error prone?  Do we need some sort of cleanup in this size accounting area?  cc 
[~saint@gmail.com]


> Compacting Memstore
> ---
>
> Key: HBASE-14920
> URL: https://issues.apache.org/jira/browse/HBASE-14920
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-14920-V01.patch, HBASE-14920-V02.patch, 
> HBASE-14920-V03.patch, HBASE-14920-V04.patch, move.to.junit4.patch
>
>
> Implementation of a new compacting memstore with non-optimized immutable 
> segment representation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14920) Compacting Memstore

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15253879#comment-15253879
 ] 

Anoop Sam John commented on HBASE-14920:


After an in memory flush, we will reduce some heap overhead and will reduce 
that delta from memstore size.  I can see a call to reduce the variable what we 
keep at RS level.  Another we have at every HRegion level, do we update that 
also?  Because of these I was thinking whether we should simplify this and 
avoid keeping the size in multiple places.

> Compacting Memstore
> ---
>
> Key: HBASE-14920
> URL: https://issues.apache.org/jira/browse/HBASE-14920
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-14920-V01.patch, HBASE-14920-V02.patch, 
> HBASE-14920-V03.patch, HBASE-14920-V04.patch, move.to.junit4.patch
>
>
> Implementation of a new compacting memstore with non-optimized immutable 
> segment representation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257632#comment-15257632
 ] 

Anoop Sam John commented on HBASE-15600:


The new way of implementation looks good and handle things in more generic way.
{code}
for (int i = firstIndex; i < lastIndexExclusive; i++) {
3123if (miniBatchOp.getOperationsFromCoprocessors(i) == null) {
3124  continue;
3125}
{code}
Here the index 'i' is from 'firstIndex' to 'lastIndexExclusive' and the call 
getOperationsFromCoprocessors(i)  will try to again add the firstIndex in 
getAbsoluteIndex().  getOperationsFromCoprocessors(int) is supposed to called 
with index 0 - miniBatchOp.size()

> Add provision for adding mutations to memstore or able to write to same 
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
>  Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch, 
> HBASE-15600_v2.patch, hbase-15600_v3.patch, hbase-15600_v4.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region 
> from coprocessors but writing from batchMutate API is not allowed because of 
> mvcc. 
> Raised PHOENIX-2742 to discuss any alternative way to write to the same 
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set 
> wal edits in MiniBatchOperationInProgress.
> {noformat}
>   /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
>   public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
>   }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as 
> well. Or else we should provide the batch mutation API allow write in batch 
> mutate coprocessors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257636#comment-15257636
 ] 

Anoop Sam John commented on HBASE-15600:


{code}
checkAndPrepareMutation(mutation, replay, cpFamilyMap, now);
3131
3132  // Acquire row locks. If not, the whole batch will fail.
3133  acquiredRowLocks.add(getRowLock(mutation.getRow(), true));
{code}
I can see there is a checkRow() call within checkAndPrepareMutation() and 
getRowLock() again have checkRow() call..  Can we avoid?

Before the call to CP, I can see ops on Mutation like
{code}
Mutation mutation = batchOp.getMutation(i);
if (mutation instanceof Put) {
  updateCellTimestamps(familyMaps[i].values(), byteNow);
  noOfPuts++;
} else {
  prepareDeleteTimestamps(mutation, familyMaps[i], byteNow);
  noOfDeletes++;
}
rewriteCellTags(familyMaps[i], mutation);
{code}
This is within doMiniBatchMutate(). checkAndPrepareMutation  seems doing some 
thing else?

> Add provision for adding mutations to memstore or able to write to same 
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
>  Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch, 
> HBASE-15600_v2.patch, hbase-15600_v3.patch, hbase-15600_v4.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region 
> from coprocessors but writing from batchMutate API is not allowed because of 
> mvcc. 
> Raised PHOENIX-2742 to discuss any alternative way to write to the same 
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set 
> wal edits in MiniBatchOperationInProgress.
> {noformat}
>   /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
>   public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
>   }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as 
> well. Or else we should provide the batch mutation API allow write in batch 
> mutate coprocessors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15706) HFilePrettyPrinter should print out nicely formatted tags

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257641#comment-15257641
 ] 

Anoop Sam John commented on HBASE-15706:


+1
You mind changing TagUtil#getValueAsString also to use toStringBinary() rather 
than toString?

> HFilePrettyPrinter should print out nicely formatted tags
> -
>
> Key: HBASE-15706
> URL: https://issues.apache.org/jira/browse/HBASE-15706
> Project: HBase
>  Issue Type: Improvement
>  Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Priority: Minor
> Attachments: HBASE-15706-v001.patch
>
>
> When I was using HFile to print out a rows with tags, the output is like:
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase 
> org.apache.hadoop.hbase.io.hfile.HFile -f 
> /tmp/71afa45b1cb94ea1858a99f31197274f -p
> 2016-04-25 11:40:40,409 WARN  [main] util.NativeCodeLoader: Unable to load 
> native-hadoop library for your platform... using builtin-java classes where 
> applicable
> 2016-04-25 11:40:40,580 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
> K: b/b:b/1461608231279/Maximum/vlen=0/seqid=0 V: 
> K: b/b:b/1461608231278/Put/vlen=1/seqid=0 V: b T[0]: �
> Scanned kv count -> 2
> {code}
> With attached patch, the print is now like:
> {code}
> 2016-04-25 11:57:05,849 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
> K: b/b:b/1461609876838/Maximum/vlen=0/seqid=0 V: 
> K: b/b:b/1461609876837/Put/vlen=1/seqid=0 V: b T[0]: [Tag type : 8, value : 
> \x00\x0E\xEE\xEE]
> Scanned kv count -> 2
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259597#comment-15259597
 ] 

Anoop Sam John commented on HBASE-15536:


So if the config 'hbase.wal.provider' is given as multiwal, it will be old way 
WAL with multiple WALs.  We have added a new config to specify this multi wal 
thing any way.  Just for confirmation asked.

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259599#comment-15259599
 ] 

Anoop Sam John commented on HBASE-15600:


Looks like the set of ops doing in doMiniBatchMutate and processRowsWithLocks 
(as part of mutateRow) is different ?  Ya we can handle all such existing 
things in another issue.  And make these 2 share code as much as it can.. 

> Add provision for adding mutations to memstore or able to write to same 
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
>  Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch, 
> HBASE-15600_v2.patch, hbase-15600_v3.patch, hbase-15600_v4.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region 
> from coprocessors but writing from batchMutate API is not allowed because of 
> mvcc. 
> Raised PHOENIX-2742 to discuss any alternative way to write to the same 
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set 
> wal edits in MiniBatchOperationInProgress.
> {noformat}
>   /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
>   public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
>   }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as 
> well. Or else we should provide the batch mutation API allow write in batch 
> mutate coprocessors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15706) HFilePrettyPrinter should print out nicely formatted tags

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259664#comment-15259664
 ] 

Anoop Sam John commented on HBASE-15706:


Better leave TagUtil as is then. In case of visibility label, if user need to 
use special chars or so, they have to quote() the expression.  I can see the 
util is used in MOB code path as well where the tag stores table name.
In KeyValue#toStringMap  also used the TagUtil.  Can u change there are also to 
use Tag.toString() instead.





> HFilePrettyPrinter should print out nicely formatted tags
> -
>
> Key: HBASE-15706
> URL: https://issues.apache.org/jira/browse/HBASE-15706
> Project: HBase
>  Issue Type: Improvement
>  Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Priority: Minor
> Attachments: HBASE-15706-v001.patch
>
>
> When I was using HFile to print out a rows with tags, the output is like:
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase 
> org.apache.hadoop.hbase.io.hfile.HFile -f 
> /tmp/71afa45b1cb94ea1858a99f31197274f -p
> 2016-04-25 11:40:40,409 WARN  [main] util.NativeCodeLoader: Unable to load 
> native-hadoop library for your platform... using builtin-java classes where 
> applicable
> 2016-04-25 11:40:40,580 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
> K: b/b:b/1461608231279/Maximum/vlen=0/seqid=0 V: 
> K: b/b:b/1461608231278/Put/vlen=1/seqid=0 V: b T[0]: �
> Scanned kv count -> 2
> {code}
> With attached patch, the print is now like:
> {code}
> 2016-04-25 11:57:05,849 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
> K: b/b:b/1461609876838/Maximum/vlen=0/seqid=0 V: 
> K: b/b:b/1461609876837/Put/vlen=1/seqid=0 V: b T[0]: [Tag type : 8, value : 
> \x00\x0E\xEE\xEE]
> Scanned kv count -> 2
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259741#comment-15259741
 ] 

Anoop Sam John commented on HBASE-15600:


bq.Do we need to check the status of the Operation if it is NOT_RUN and then 
only add the CP mutations to it?  Wil there be such  a case?
Ideally when the CP itself handle a particular mutation and mark it as RUN, 
(success or failure) it should NOT be adding any new mutations into the 
corresponding index.  Now the thing is even if we we merge familyMaps, it is no 
use as we will just ignore mutation (familyMap) of this index.  So the added 
mutations are or no use.  Ya thing is we can avoid the not useful work of this 
merge.  Chances are less user will do such things.. Still can handle it if you 
like [~enis]

> Add provision for adding mutations to memstore or able to write to same 
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
>  Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch, 
> HBASE-15600_v2.patch, hbase-15600_v3.patch, hbase-15600_v4.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region 
> from coprocessors but writing from batchMutate API is not allowed because of 
> mvcc. 
> Raised PHOENIX-2742 to discuss any alternative way to write to the same 
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set 
> wal edits in MiniBatchOperationInProgress.
> {noformat}
>   /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
>   public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
>   }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as 
> well. Or else we should provide the batch mutation API allow write in batch 
> mutate coprocessors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15721) Optimization in cloning cells into MSLAB

[Anoop Sam John (JIRA)]

Anoop Sam John created HBASE-15721:
--

 Summary: Optimization in cloning cells into MSLAB
 Key: HBASE-15721
 URL: https://issues.apache.org/jira/browse/HBASE-15721
 Project: HBase
  Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Fix For: 2.0.0


Before cells added to memstore CSLM, there is a clone of cell after copying it 
to MSLAB chunk area.  This is done not in an efficient way.
{code}
public static int appendToByteArray(final Cell cell, final byte[] output, final 
int offset) {
int pos = offset;
pos = Bytes.putInt(output, pos, keyLength(cell));
pos = Bytes.putInt(output, pos, cell.getValueLength());
pos = appendKeyTo(cell, output, pos);
pos = CellUtil.copyValueTo(cell, output, pos);
if ((cell.getTagsLength() > 0)) {
  pos = Bytes.putAsShort(output, pos, cell.getTagsLength());
  pos = CellUtil.copyTagTo(cell, output, pos);
}
return pos;
  }
{code}
Copied in 9 steps and we end up parsing all lengths.  When the cell 
implementation is backed by a single byte[] (Like KeyValue) this can be done in 
single step copy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15721) Optimization in cloning cells into MSLAB

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15721:
---
Attachment: HBASE-15721.patch

This also avoid lot of garbage. With every cell, we used to create a ByteRange 
instance to pass the allocated  chunk + offset. Now that is also avoided.

Removed the deprecation in KeyValue#getBuffer().  Now we have Cell in read and 
write paths.  Ya in Cell we dont have any assumption that the entire thing is 
backed by single byte[]. But in KeyValue it is backed by single byte[] and so a 
getter for getting this ref is just fine IMO.

> Optimization in cloning cells into MSLAB
> 
>
> Key: HBASE-15721
> URL: https://issues.apache.org/jira/browse/HBASE-15721
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15721.patch
>
>
> Before cells added to memstore CSLM, there is a clone of cell after copying 
> it to MSLAB chunk area.  This is done not in an efficient way.
> {code}
> public static int appendToByteArray(final Cell cell, final byte[] output, 
> final int offset) {
> int pos = offset;
> pos = Bytes.putInt(output, pos, keyLength(cell));
> pos = Bytes.putInt(output, pos, cell.getValueLength());
> pos = appendKeyTo(cell, output, pos);
> pos = CellUtil.copyValueTo(cell, output, pos);
> if ((cell.getTagsLength() > 0)) {
>   pos = Bytes.putAsShort(output, pos, cell.getTagsLength());
>   pos = CellUtil.copyTagTo(cell, output, pos);
> }
> return pos;
>   }
> {code}
> Copied in 9 steps and we end up parsing all lengths.  When the cell 
> implementation is backed by a single byte[] (Like KeyValue) this can be done 
> in single step copy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15721) Optimization in cloning cells into MSLAB

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15721:
---
Status: Patch Available  (was: Open)

> Optimization in cloning cells into MSLAB
> 
>
> Key: HBASE-15721
> URL: https://issues.apache.org/jira/browse/HBASE-15721
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15721.patch
>
>
> Before cells added to memstore CSLM, there is a clone of cell after copying 
> it to MSLAB chunk area.  This is done not in an efficient way.
> {code}
> public static int appendToByteArray(final Cell cell, final byte[] output, 
> final int offset) {
> int pos = offset;
> pos = Bytes.putInt(output, pos, keyLength(cell));
> pos = Bytes.putInt(output, pos, cell.getValueLength());
> pos = appendKeyTo(cell, output, pos);
> pos = CellUtil.copyValueTo(cell, output, pos);
> if ((cell.getTagsLength() > 0)) {
>   pos = Bytes.putAsShort(output, pos, cell.getTagsLength());
>   pos = CellUtil.copyTagTo(cell, output, pos);
> }
> return pos;
>   }
> {code}
> Copied in 9 steps and we end up parsing all lengths.  When the cell 
> implementation is backed by a single byte[] (Like KeyValue) this can be done 
> in single step copy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15721) Optimization in cloning cells into MSLAB

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15260001#comment-15260001
 ] 

Anoop Sam John commented on HBASE-15721:


Oh in the other work space have already an off heap implementation of MSLAB and 
so we have BB backing the chunks.  I will correct that.
Ya for trunk as of now, this can be byte[] based.
Later when we have off heaping in MSLAB also, we can change this byte[] to 
ByteBuffer.  All these are private interfaces and so no problem. Any way al 
changed in trunk alone.

> Optimization in cloning cells into MSLAB
> 
>
> Key: HBASE-15721
> URL: https://issues.apache.org/jira/browse/HBASE-15721
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15721.patch
>
>
> Before cells added to memstore CSLM, there is a clone of cell after copying 
> it to MSLAB chunk area.  This is done not in an efficient way.
> {code}
> public static int appendToByteArray(final Cell cell, final byte[] output, 
> final int offset) {
> int pos = offset;
> pos = Bytes.putInt(output, pos, keyLength(cell));
> pos = Bytes.putInt(output, pos, cell.getValueLength());
> pos = appendKeyTo(cell, output, pos);
> pos = CellUtil.copyValueTo(cell, output, pos);
> if ((cell.getTagsLength() > 0)) {
>   pos = Bytes.putAsShort(output, pos, cell.getTagsLength());
>   pos = CellUtil.copyTagTo(cell, output, pos);
> }
> return pos;
>   }
> {code}
> Copied in 9 steps and we end up parsing all lengths.  When the cell 
> implementation is backed by a single byte[] (Like KeyValue) this can be done 
> in single step copy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15706) HFilePrettyPrinter should print out nicely formatted tags

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15261444#comment-15261444
 ] 

Anoop Sam John commented on HBASE-15706:


Sorry if I was not clearly saying it.. I dont mean that.
bq.leave TagUtil as is then. 
No need to change and the MOB part and VC will use this util API only.

bq.In KeyValue#toStringMap also used the TagUtil. Can u change there are also 
to use Tag.toString() instead.
U can see in KeyValue there is a usage of TagUtil.getValueAsString.  There can 
u change that to Tag.toString() just like in HFilePrettyPrinter case.

> HFilePrettyPrinter should print out nicely formatted tags
> -
>
> Key: HBASE-15706
> URL: https://issues.apache.org/jira/browse/HBASE-15706
> Project: HBase
>  Issue Type: Improvement
>  Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Priority: Minor
> Attachments: HBASE-15706-v001.patch
>
>
> When I was using HFile to print out a rows with tags, the output is like:
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase 
> org.apache.hadoop.hbase.io.hfile.HFile -f 
> /tmp/71afa45b1cb94ea1858a99f31197274f -p
> 2016-04-25 11:40:40,409 WARN  [main] util.NativeCodeLoader: Unable to load 
> native-hadoop library for your platform... using builtin-java classes where 
> applicable
> 2016-04-25 11:40:40,580 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
> K: b/b:b/1461608231279/Maximum/vlen=0/seqid=0 V: 
> K: b/b:b/1461608231278/Put/vlen=1/seqid=0 V: b T[0]: �
> Scanned kv count -> 2
> {code}
> With attached patch, the print is now like:
> {code}
> 2016-04-25 11:57:05,849 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
> K: b/b:b/1461609876838/Maximum/vlen=0/seqid=0 V: 
> K: b/b:b/1461609876837/Put/vlen=1/seqid=0 V: b T[0]: [Tag type : 8, value : 
> \x00\x0E\xEE\xEE]
> Scanned kv count -> 2
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15261757#comment-15261757
 ] 

Anoop Sam John commented on HBASE-15525:


bq.implementation than the one I created (and you denied) several months ago.
Which one?

> OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
> --
>
> Key: HBASE-15525
> URL: https://issues.apache.org/jira/browse/HBASE-15525
> Project: HBase
>  Issue Type: Sub-task
>  Components: IPC/RPC
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15525_V1.patch, HBASE-15525_V2.patch, 
> HBASE-15525_WIP.patch, WIP.patch
>
>
> After HBASE-13819 the system some times run out of direct memory whenever 
> there is some network congestion or some client side issues.
> This was because of pending RPCs in the RPCServer$Connection.responseQueue 
> and since all the responses in this queue hold a buffer for cellblock from 
> BoundedByteBufferPool this could takeup a lot of memory if the 
> BoundedByteBufferPool's moving average settles down towards a higher value 
> See the discussion here 
> [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15714) We are calling checkRow() twice in doMiniBatchMutation()

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15261859#comment-15261859
 ] 

Anoop Sam John commented on HBASE-15714:


LGTM.
We have some inconsistency with the ops doing in doMiniBatch and in 
mutateRows().. (I have added a comment abt that in the other Jira)..  We can 
handle it in another jira. (Ya a different topic it is).


> We are calling checkRow() twice in doMiniBatchMutation()
> 
>
> Key: HBASE-15714
> URL: https://issues.apache.org/jira/browse/HBASE-15714
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15714.patch
>
>
> In {{multi()}} -> {{doMiniBatchMutation()}} code path, we end up calling 
> {{checkRow()}} twice, once from {{checkBatchOp()}} and once from 
> {{getRowLock()}}. 
> See [~anoop.hbase]'s comments at 
> https://issues.apache.org/jira/browse/HBASE-15600?focusedCommentId=15257636&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15257636.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15609) Remove PB references from Result, DoubleColumnInterpreter and any such public facing class for 2.0

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15261929#comment-15261929
 ] 

Anoop Sam John commented on HBASE-15609:


Ya even the base class is Private marked..

> Remove PB references from Result, DoubleColumnInterpreter and any such public 
> facing class for 2.0
> --
>
> Key: HBASE-15609
> URL: https://issues.apache.org/jira/browse/HBASE-15609
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-15609.patch
>
>
> This is a sub-task for HBASE-15174.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15706) HFilePrettyPrinter should print out nicely formatted tags

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263490#comment-15263490
 ] 

Anoop Sam John commented on HBASE-15706:


+1

> HFilePrettyPrinter should print out nicely formatted tags
> -
>
> Key: HBASE-15706
> URL: https://issues.apache.org/jira/browse/HBASE-15706
> Project: HBase
>  Issue Type: Improvement
>  Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Priority: Minor
> Attachments: HBASE-15706-v001.patch, HBASE-15706-v002.patch
>
>
> When I was using HFile to print out a rows with tags, the output is like:
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase 
> org.apache.hadoop.hbase.io.hfile.HFile -f 
> /tmp/71afa45b1cb94ea1858a99f31197274f -p
> 2016-04-25 11:40:40,409 WARN  [main] util.NativeCodeLoader: Unable to load 
> native-hadoop library for your platform... using builtin-java classes where 
> applicable
> 2016-04-25 11:40:40,580 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
> K: b/b:b/1461608231279/Maximum/vlen=0/seqid=0 V: 
> K: b/b:b/1461608231278/Put/vlen=1/seqid=0 V: b T[0]: �
> Scanned kv count -> 2
> {code}
> With attached patch, the print is now like:
> {code}
> 2016-04-25 11:57:05,849 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
> K: b/b:b/1461609876838/Maximum/vlen=0/seqid=0 V: 
> K: b/b:b/1461609876837/Put/vlen=1/seqid=0 V: b T[0]: [Tag type : 8, value : 
> \x00\x0E\xEE\xEE]
> Scanned kv count -> 2
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15706) HFilePrettyPrinter should print out nicely formatted tags

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15706:
---
Assignee: huaxiang sun

> HFilePrettyPrinter should print out nicely formatted tags
> -
>
> Key: HBASE-15706
> URL: https://issues.apache.org/jira/browse/HBASE-15706
> Project: HBase
>  Issue Type: Improvement
>  Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-15706-v001.patch, HBASE-15706-v002.patch
>
>
> When I was using HFile to print out a rows with tags, the output is like:
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase 
> org.apache.hadoop.hbase.io.hfile.HFile -f 
> /tmp/71afa45b1cb94ea1858a99f31197274f -p
> 2016-04-25 11:40:40,409 WARN  [main] util.NativeCodeLoader: Unable to load 
> native-hadoop library for your platform... using builtin-java classes where 
> applicable
> 2016-04-25 11:40:40,580 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
> K: b/b:b/1461608231279/Maximum/vlen=0/seqid=0 V: 
> K: b/b:b/1461608231278/Put/vlen=1/seqid=0 V: b T[0]: �
> Scanned kv count -> 2
> {code}
> With attached patch, the print is now like:
> {code}
> 2016-04-25 11:57:05,849 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
> K: b/b:b/1461609876838/Maximum/vlen=0/seqid=0 V: 
> K: b/b:b/1461609876837/Put/vlen=1/seqid=0 V: b T[0]: [Tag type : 8, value : 
> \x00\x0E\xEE\xEE]
> Scanned kv count -> 2
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15706) HFilePrettyPrinter should print out nicely formatted tags

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15706:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master.  Thanks for the patch.

> HFilePrettyPrinter should print out nicely formatted tags
> -
>
> Key: HBASE-15706
> URL: https://issues.apache.org/jira/browse/HBASE-15706
> Project: HBase
>  Issue Type: Improvement
>  Components: HFile
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15706-v001.patch, HBASE-15706-v002.patch
>
>
> When I was using HFile to print out a rows with tags, the output is like:
> {code}
> hsun-MBP:hbase-2.0.0-SNAPSHOT hsun$ hbase 
> org.apache.hadoop.hbase.io.hfile.HFile -f 
> /tmp/71afa45b1cb94ea1858a99f31197274f -p
> 2016-04-25 11:40:40,409 WARN  [main] util.NativeCodeLoader: Unable to load 
> native-hadoop library for your platform... using builtin-java classes where 
> applicable
> 2016-04-25 11:40:40,580 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
> K: b/b:b/1461608231279/Maximum/vlen=0/seqid=0 V: 
> K: b/b:b/1461608231278/Put/vlen=1/seqid=0 V: b T[0]: �
> Scanned kv count -> 2
> {code}
> With attached patch, the print is now like:
> {code}
> 2016-04-25 11:57:05,849 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
> K: b/b:b/1461609876838/Maximum/vlen=0/seqid=0 V: 
> K: b/b:b/1461609876837/Put/vlen=1/seqid=0 V: b T[0]: [Tag type : 8, value : 
> \x00\x0E\xEE\xEE]
> Scanned kv count -> 2
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15721) Optimization in cloning cells into MSLAB

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15721:
---
Attachment: HBASE-15721_V2.patch

Patch fixing comments from Ram and Ted.

> Optimization in cloning cells into MSLAB
> 
>
> Key: HBASE-15721
> URL: https://issues.apache.org/jira/browse/HBASE-15721
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15721.patch, HBASE-15721_V2.patch
>
>
> Before cells added to memstore CSLM, there is a clone of cell after copying 
> it to MSLAB chunk area.  This is done not in an efficient way.
> {code}
> public static int appendToByteArray(final Cell cell, final byte[] output, 
> final int offset) {
> int pos = offset;
> pos = Bytes.putInt(output, pos, keyLength(cell));
> pos = Bytes.putInt(output, pos, cell.getValueLength());
> pos = appendKeyTo(cell, output, pos);
> pos = CellUtil.copyValueTo(cell, output, pos);
> if ((cell.getTagsLength() > 0)) {
>   pos = Bytes.putAsShort(output, pos, cell.getTagsLength());
>   pos = CellUtil.copyTagTo(cell, output, pos);
> }
> return pos;
>   }
> {code}
> Copied in 9 steps and we end up parsing all lengths.  When the cell 
> implementation is backed by a single byte[] (Like KeyValue) this can be done 
> in single step copy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263527#comment-15263527
 ] 

Anoop Sam John commented on HBASE-15536:


+1 
bq.Recently we observed performance downgrade when using multiwal 
How many WALs per RS in this test?

> Make AsyncFSWAL as our default WAL
> --
>
> Key: HBASE-15536
> URL: https://issues.apache.org/jira/browse/HBASE-15536
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, 
> HBASE-15536.patch
>
>
> As it should be predicated on passing basic cluster ITBLL



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263528#comment-15263528
 ] 

Anoop Sam John commented on HBASE-15600:


{code}
// we pass (i - firstIndex) below since the call expects a relative index
3128if (miniBatchOp.getOperationsFromCoprocessors(i - 
firstIndex) == null) {
3129  continue;
3130}
3131// Else Coprocessor added more Mutations corresponding to 
the Mutation at this index.
3132for (int j = 0; j < 
miniBatchOp.getOperationsFromCoprocessors(i).length; j++) {
{code}
U changed the index in 1st place but missed the second  
"miniBatchOp.getOperationsFromCoprocessors(i)."

Better take this miniBatchOp.getOperationsFromCoprocessors(i - firstIndex)   
into a local variable and use that.. Else there is always a chance to miss one 
place :-)


> Add provision for adding mutations to memstore or able to write to same 
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
>  Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch, 
> HBASE-15600_v2.patch, hbase-15600_v3.patch, hbase-15600_v4.patch, 
> hbase-15600_v5.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region 
> from coprocessors but writing from batchMutate API is not allowed because of 
> mvcc. 
> Raised PHOENIX-2742 to discuss any alternative way to write to the same 
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set 
> wal edits in MiniBatchOperationInProgress.
> {noformat}
>   /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
>   public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
>   }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as 
> well. Or else we should provide the batch mutation API allow write in batch 
> mutate coprocessors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263530#comment-15263530
 ] 

Anoop Sam John commented on HBASE-15600:


bq.acquiredRowLocks.add(getRowLock(mutation.getRow(), true));
If the other jira can committed 1st, we can change here to use 
getRowLockInternal?

Also abt the inconsistent ops in doMiniBatch and mutateRows() u plan to raise a 
Jira?  We can handle that there.




> Add provision for adding mutations to memstore or able to write to same 
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
>  Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.20
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch, 
> HBASE-15600_v2.patch, hbase-15600_v3.patch, hbase-15600_v4.patch, 
> hbase-15600_v5.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region 
> from coprocessors but writing from batchMutate API is not allowed because of 
> mvcc. 
> Raised PHOENIX-2742 to discuss any alternative way to write to the same 
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set 
> wal edits in MiniBatchOperationInProgress.
> {noformat}
>   /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
>   public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
>   }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as 
> well. Or else we should provide the batch mutation API allow write in batch 
> mutate coprocessors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15735) Tightening of the CP contract

[Anoop Sam John (JIRA)]

Anoop Sam John created HBASE-15735:
--

 Summary: Tightening of the CP contract 
 Key: HBASE-15735
 URL: https://issues.apache.org/jira/browse/HBASE-15735
 Project: HBase
  Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: Anoop Sam John


This is after the discussion in dev@ under 'DISCUSSION: A tightening of the CP 
contract in hbase 2.0.0'
"CPs can read Cells passed them on a CP hook invocation but
they must not retain references beyond the life of the invocation; they
must copy Cells if they intend to retain them beyond the invocation, pass
the Cells off to upper tiers, or return them otherwise to clients."

Also this Jira will cleanup some code which was doing the copy of the Cells 
while passing to CP hooks assuming it might keep cell refs. Now as we say it 
explicitly in CP APIs,  we dont need this copy.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15735) Tightening of the CP contract

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15735:
---
Attachment: HBASE-15735.patch

> Tightening of the CP contract 
> --
>
> Key: HBASE-15735
> URL: https://issues.apache.org/jira/browse/HBASE-15735
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15735.patch
>
>
> This is after the discussion in dev@ under 'DISCUSSION: A tightening of the 
> CP contract in hbase 2.0.0'
> "CPs can read Cells passed them on a CP hook invocation but
> they must not retain references beyond the life of the invocation; they
> must copy Cells if they intend to retain them beyond the invocation, pass
> the Cells off to upper tiers, or return them otherwise to clients."
> Also this Jira will cleanup some code which was doing the copy of the Cells 
> while passing to CP hooks assuming it might keep cell refs. Now as we say it 
> explicitly in CP APIs,  we dont need this copy.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15735) Tightening of the CP contract

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15735:
---
Status: Patch Available  (was: Open)

> Tightening of the CP contract 
> --
>
> Key: HBASE-15735
> URL: https://issues.apache.org/jira/browse/HBASE-15735
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15735.patch
>
>
> This is after the discussion in dev@ under 'DISCUSSION: A tightening of the 
> CP contract in hbase 2.0.0'
> "CPs can read Cells passed them on a CP hook invocation but
> they must not retain references beyond the life of the invocation; they
> must copy Cells if they intend to retain them beyond the invocation, pass
> the Cells off to upper tiers, or return them otherwise to clients."
> Also this Jira will cleanup some code which was doing the copy of the Cells 
> while passing to CP hooks assuming it might keep cell refs. Now as we say it 
> explicitly in CP APIs,  we dont need this copy.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15735) Tightening of the CP contract

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15735:
---
Attachment: HBASE-15735.patch

Corrected the compile issue.

> Tightening of the CP contract 
> --
>
> Key: HBASE-15735
> URL: https://issues.apache.org/jira/browse/HBASE-15735
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15735.patch, HBASE-15735.patch
>
>
> This is after the discussion in dev@ under 'DISCUSSION: A tightening of the 
> CP contract in hbase 2.0.0'
> "CPs can read Cells passed them on a CP hook invocation but
> they must not retain references beyond the life of the invocation; they
> must copy Cells if they intend to retain them beyond the invocation, pass
> the Cells off to upper tiers, or return them otherwise to clients."
> Also this Jira will cleanup some code which was doing the copy of the Cells 
> while passing to CP hooks assuming it might keep cell refs. Now as we say it 
> explicitly in CP APIs,  we dont need this copy.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15736) TSV Bulk loader not supporting multi-line strings

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263937#comment-15263937
 ] 

Anoop Sam John commented on HBASE-15736:


Ya it is written to work with one row data in one line, this way.. 

> TSV Bulk loader not supporting multi-line strings
> -
>
> Key: HBASE-15736
> URL: https://issues.apache.org/jira/browse/HBASE-15736
> Project: HBase
>  Issue Type: Bug
>Reporter: Federico Ponzi
>  Labels: bulkloader, tsv
>   Original Estimate: 2m
>  Remaining Estimate: 2m
>
> Example tsv:
> {code}
> 1,"This string is
> on two lines"
> 2,"Hello world"
> {code}
> Bad line at offset: 18:
> No delimiter



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15735) Tightening of the CP contract

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15735:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to master. Thanks for the reviews.

> Tightening of the CP contract 
> --
>
> Key: HBASE-15735
> URL: https://issues.apache.org/jira/browse/HBASE-15735
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15735.patch, HBASE-15735.patch
>
>
> This is after the discussion in dev@ under 'DISCUSSION: A tightening of the 
> CP contract in hbase 2.0.0'
> "CPs can read Cells passed them on a CP hook invocation but
> they must not retain references beyond the life of the invocation; they
> must copy Cells if they intend to retain them beyond the invocation, pass
> the Cells off to upper tiers, or return them otherwise to clients."
> Also this Jira will cleanup some code which was doing the copy of the Cells 
> while passing to CP hooks assuming it might keep cell refs. Now as we say it 
> explicitly in CP APIs,  we dont need this copy.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15721) Optimization in cloning cells into MSLAB

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266153#comment-15266153
 ] 

Anoop Sam John commented on HBASE-15721:


MemStoreLAB says
{quote}
Without the MSLAB, the byte array allocated during insertion end up interleaved 
throughout the
 * heap, and the old generation gets progressively more fragmented until a 
stop-the-world compacting
 * collection occurs.
{quote}
So instead of allocating a chunk from interface and copy and recreate Cell in 
memstore class, we can allocate and  clone within the MSLAB impl. May be we 
need change the comments on class.
Ah I missed reading the javadocs in Streamble..  Ya agree bit odd to add a 
write method into byte[].  So the best way will be to have a Writable 
interface?  What do u say?  It will help if we can write in one shot rather 
than individual components

getBuffer() can be returned by any KeyValue.  Not Cell..  Ya KeyValue we say it 
is backed by single byte[].  So I thought it is just fine. We deprecated it 
when the read and write flow had KeyValue as passed type (in some places) 
rather than Cell..  Now we have Cell every where.  And Cell dont have 
getBuffer()..  I thought it is just fine.

> Optimization in cloning cells into MSLAB
> 
>
> Key: HBASE-15721
> URL: https://issues.apache.org/jira/browse/HBASE-15721
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15721.patch, HBASE-15721_V2.patch
>
>
> Before cells added to memstore CSLM, there is a clone of cell after copying 
> it to MSLAB chunk area.  This is done not in an efficient way.
> {code}
> public static int appendToByteArray(final Cell cell, final byte[] output, 
> final int offset) {
> int pos = offset;
> pos = Bytes.putInt(output, pos, keyLength(cell));
> pos = Bytes.putInt(output, pos, cell.getValueLength());
> pos = appendKeyTo(cell, output, pos);
> pos = CellUtil.copyValueTo(cell, output, pos);
> if ((cell.getTagsLength() > 0)) {
>   pos = Bytes.putAsShort(output, pos, cell.getTagsLength());
>   pos = CellUtil.copyTagTo(cell, output, pos);
> }
> return pos;
>   }
> {code}
> Copied in 9 steps and we end up parsing all lengths.  When the cell 
> implementation is backed by a single byte[] (Like KeyValue) this can be done 
> in single step copy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15608) Remove PB references from SnapShot related Exceptions

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15268442#comment-15268442
 ] 

Anoop Sam John commented on HBASE-15608:


{code}
new org.apache.hadoop.hbase.client.SnapshotDescription(reqSnapshot.getName(),
788   reqSnapshot.getTable(), 
ProtobufUtil.createSnapshotType(reqSnapshot.getType()),
789   reqSnapshot.getOwner(), reqSnapshot.getCreationTime(), 
reqSnapshot.getVersion()));
{code}
This will convert a PB SnapshotDescription into our POJO SnapshotDescription 
right?  Can we add this in a Util method and just use that in all these places?
Like PBUtil.toSnapshotDescription(PB object)

> Remove PB references from SnapShot related Exceptions
> -
>
> Key: HBASE-15608
> URL: https://issues.apache.org/jira/browse/HBASE-15608
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-15608.patch
>
>
> This is a sub-task for HBASE-15174.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14920) Compacting Memstore

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15268734#comment-15268734
 ] 

Anoop Sam John commented on HBASE-14920:


IMHO, when a user calls to flush a region, all the cells must get flushed to 
disk. It is an explicit call for that from user.
Also yes agree to the fact that the flush tried to reduce the heap usage of the 
memstore and flushing only part do this.  But the selection of the region for 
flushing is done with assumption that the flush call will release all memstore 
heap size. (What is there at that time)   It can happen that when flush is 
called the pipeline have entries and active also almost near to in memory flush 
size.   The heap size occupied by active is considerably high (specially with 
small sized cells) because of the overhead part.  

> Compacting Memstore
> ---
>
> Key: HBASE-14920
> URL: https://issues.apache.org/jira/browse/HBASE-14920
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-14920-V01.patch, HBASE-14920-V02.patch, 
> HBASE-14920-V03.patch, HBASE-14920-V04.patch, HBASE-14920-V05.patch, 
> HBASE-14920-V06.patch, move.to.junit4.patch
>
>
> Implementation of a new compacting memstore with non-optimized immutable 
> segment representation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14920) Compacting Memstore

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15268737#comment-15268737
 ] 

Anoop Sam John commented on HBASE-14920:


bq.None of the counters (RS, region, segment) are new, all of them existed 
before this patch, so I fail to see the problem.
I agree. Am not saying this patch added any..  What I say is with the size math 
happens in 3 places and with added more calls now after this patch to change 
the size, it is very complex.  It is more error prone..  So just asking whether 
we should visit this area and some way simplify it.  Not direct on this patch.

> Compacting Memstore
> ---
>
> Key: HBASE-14920
> URL: https://issues.apache.org/jira/browse/HBASE-14920
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-14920-V01.patch, HBASE-14920-V02.patch, 
> HBASE-14920-V03.patch, HBASE-14920-V04.patch, HBASE-14920-V05.patch, 
> HBASE-14920-V06.patch, move.to.junit4.patch
>
>
> Implementation of a new compacting memstore with non-optimized immutable 
> segment representation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15759) RegionObserver.preStoreScannerOpen() doesn't have acces to current readpoint

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15268748#comment-15268748
 ] 

Anoop Sam John commented on HBASE-15759:


So we should overload the pre hook with readPoint as extra param.  Deprecate 
the existing CP hook.  You will give a patch?

> RegionObserver.preStoreScannerOpen() doesn't have acces to current readpoint
> 
>
> Key: HBASE-15759
> URL: https://issues.apache.org/jira/browse/HBASE-15759
> Project: HBase
>  Issue Type: Improvement
>  Components: Coprocessors
>Reporter: Marek Srank
>Priority: Minor
>  Labels: coprocessors
>
> RegionObserver.preStoreScannerOpen() doesn't have acces to current readpoint. 
> This is crucial e.g. when creating a new StoreScanner.
> Although it is possible to obtain the readpoint via following workaround:
> ((HStore)store).getHRegion().getReadpoint(IsolationLevel.READ_COMMITTED)
> ...it presumes some knowledge of HBase internals and is not a clear option 
> for user.
> The problem was introduced in HBASE-9754, where readpoint was introduced as 
> an argument to StoreScanner constructor, but wasn't passed to the 
> preStoreScannerOpen() hook in HStore.getScanner().
> In our case it lead to bug, when we were incorrectly using 
> smallestReadpoint() instead, which made coprocessor missing data in some 
> cases [1].
> __
> [1] 
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201604.mbox/%3CCAOUjMkwptCGacQ0d-4yOinDVBT%2Bbi2NOZee0fSbq%2Bk1R6V4-aw%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15590) Add ACL for requesting table backup

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15269103#comment-15269103
 ] 

Anoop Sam John commented on HBASE-15590:


bq.public boolean preBackupTables
What is the use of this return type. Seems that will be unused. I can not see 
in MasterCoprocessorHost the return of the pre hook is used.  The bypass usage 
is enough?  Or we should return a Pair as the real op doing?  This is inline 
with preScannerOpen() and all..  To pre hook we pass the type and table names. 
Other details also needed?  Like request.getTargetRootDir()?
For post hook, we need to pass the Pair result of call master.backupTables ()?  
So we need Global Admin privilege to do the backup op.   Table admin can do 
back up of that table?



> Add ACL for requesting table backup
> ---
>
> Key: HBASE-15590
> URL: https://issues.apache.org/jira/browse/HBASE-15590
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15590.v1.patch
>
>
> This issue adds necessary coprocessor hooks for table backup request along 
> with enforcing permission check in AccessController through the new hooks.
> To perform backup, admin privilege is required in secure deployment. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14920) Compacting Memstore

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272754#comment-15272754
 ] 

Anoop Sam John commented on HBASE-14920:


So how will it behave when system call region flush before a graceful stop 
(region close)?  This will happen before split, after a replay etc?   Then also 
we wont do the full flush?
I agree to the point that in ur use case where mostly increment mutations, 
delaying the flush of cells as much as possible helps you.  Considering the 
general usage where most of the cells will survive a flush,  dont you think 
this part only flush will make more number of small sized files and so more 
#compactions?The normal decision making of when 2 flush is depending on the 
memstore size.  This feature helps to reduce the overall memstore heap size.   
Say we will in memory flush when 25% of memstore size reaches. This will reduce 
the heap size as it avoid the Cell object and CSLM entry overhead.   Also this 
will help to provide bigger value for memstore (provided we have more -Xmx 
value)..  In normal memstore the more the value for memstore, the more #cells 
it will hold and when it increases CSLM perf become poor.   But in this 
feature, we do clear off the CSLM in btw and so we are better.  So these way it 
already helps much to hold off cells for longer time in memory.  Am still not 
convinced on the part that an explicit need/call to flush is still not flushing 
the whole memstore.


> Compacting Memstore
> ---
>
> Key: HBASE-14920
> URL: https://issues.apache.org/jira/browse/HBASE-14920
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-14920-V01.patch, HBASE-14920-V02.patch, 
> HBASE-14920-V03.patch, HBASE-14920-V04.patch, HBASE-14920-V05.patch, 
> HBASE-14920-V06.patch, HBASE-14920-V07.patch, move.to.junit4.patch
>
>
> Implementation of a new compacting memstore with non-optimized immutable 
> segment representation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15759) RegionObserver.preStoreScannerOpen() doesn't have acces to current readpoint

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272844#comment-15272844
 ] 

Anoop Sam John commented on HBASE-15759:


BaseRegionObserver  new method should call the old method.  This will give 
better chance to get BC. We follow that way from long time.

> RegionObserver.preStoreScannerOpen() doesn't have acces to current readpoint
> 
>
> Key: HBASE-15759
> URL: https://issues.apache.org/jira/browse/HBASE-15759
> Project: HBase
>  Issue Type: Improvement
>  Components: Coprocessors
>Reporter: Marek Srank
>Assignee: Ted Yu
>Priority: Minor
>  Labels: coprocessors
> Fix For: 2.0.0
>
> Attachments: 15759.v1.patch, 15759.v2.patch, 15759.v3.patch
>
>
> RegionObserver.preStoreScannerOpen() doesn't have acces to current readpoint. 
> This is crucial e.g. when creating a new StoreScanner.
> Although it is possible to obtain the readpoint via following workaround:
> ((HStore)store).getHRegion().getReadpoint(IsolationLevel.READ_COMMITTED)
> ...it presumes some knowledge of HBase internals and is not a clear option 
> for user.
> The problem was introduced in HBASE-9754, where readpoint was introduced as 
> an argument to StoreScanner constructor, but wasn't passed to the 
> preStoreScannerOpen() hook in HStore.getScanner().
> In our case it led to bug, when we were incorrectly using smallestReadpoint() 
> instead, which made coprocessor missing data in some cases [1].
> __
> [1] 
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201604.mbox/%3CCAOUjMkwptCGacQ0d-4yOinDVBT%2Bbi2NOZee0fSbq%2Bk1R6V4-aw%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15757) Reverse scan fails with no obvious cause

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273640#comment-15273640
 ] 

Anoop Sam John commented on HBASE-15757:


Can u capture the RS log at this time around and pass it here?

> Reverse scan fails with no obvious cause
> 
>
> Key: HBASE-15757
> URL: https://issues.apache.org/jira/browse/HBASE-15757
> Project: HBase
>  Issue Type: Bug
>  Components: Client, Scanners
>Affects Versions: 0.98.12
> Environment: ubuntu 14.04, amazon cloud; 10 datanodes d2.4xlarge - 
> 16cores, 12x200GB HDD, 122GB RAM
>Reporter: Robert Fiser
>
> related issue on stackoverflow: 
> http://stackoverflow.com/questions/37001169/hbase-reverse-scan-error?noredirect=1#comment61558097_37001169
> this works well:
> scan = new Scan(startRow, stopRow);
> this throws exception sometimes:
> scan = new Scan(stopRow, startRow);
>   scan.setReversed(true);
> throwing exception while traffic is at least 100 req/s. there are actually no 
> timeouts, exception is fired immediately for 1-10% requests
> hbase: 0.98.12-hadoop2;
> hadoop: 2.7.0;
> cluster in AWS, 10 datanodes: d2.4xlarge
> I think it's maybe related with this issue but I'm not using any filters 
> http://apache-hbase.679495.n3.nabble.com/Exception-during-a-reverse-scan-with-filter-td4069721.html
>   java.lang.RuntimeException: 
> org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of 
> OutOfOrderScannerNextException: was there a rpc timeout?
>   at 
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
>   at 
> com.socialbakers.broker.client.hbase.htable.AbstractHtableListScanner.scanToList(AbstractHtableListScanner.java:30)
>   at 
> com.socialbakers.broker.client.hbase.htable.AbstractHtableListSingleScanner.invokeOperation(AbstractHtableListSingleScanner.java:23)
>   at 
> com.socialbakers.broker.client.hbase.htable.AbstractHtableListSingleScanner.invokeOperation(AbstractHtableListSingleScanner.java:11)
>   at 
> com.socialbakers.broker.client.hbase.AbstractHbaseApi.endPointMethod(AbstractHbaseApi.java:40)
>   at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown 
> Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> com.socialbakers.broker.client.Route.invoke(Route.java:241)
>   at 
> com.socialbakers.broker.client.handler.EndpointHandler.invoke(EndpointHandler.java:173)
>   at 
> com.socialbakers.broker.client.handler.EndpointHandler.process(EndpointHandler.java:69)
>   at 
> com.thetransactioncompany.jsonrpc2.server.Dispatcher.process(Dispatcher.java:196)
>   at 
> com.socialbakers.broker.client.RejectableRunnable.run(RejectableRunnable.java:38)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>   Caused by: 
> org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of 
> OutOfOrderScannerNextException: was there a rpc timeout?
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:430)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:333)
>   at 
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:91)
>   ... 15 more
>   Caused by: 
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: 
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: Expected 
> nextCallSeq: 2 But the nextCallSeq got from client: 1; request=scanner_id: 
> 27700695 number_of_rows: 100 close_scanner: false next_call_seq: 1
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3231)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30946)
>   at 
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093)
>   at 
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
>   at 
> org.apache.hadoop.hbas

[jira] [Commented] (HBASE-15608) Remove PB references from SnapShot related Exceptions

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273658#comment-15273658
 ] 

Anoop Sam John commented on HBASE-15608:


+1 once QA is fine

> Remove PB references from SnapShot related Exceptions
> -
>
> Key: HBASE-15608
> URL: https://issues.apache.org/jira/browse/HBASE-15608
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-15608.patch, HBASE-15608_1.patch
>
>
> This is a sub-task for HBASE-15174.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11432) [AccessController] Remove cell first strategy

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-11432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273694#comment-15273694
 ] 

Anoop Sam John commented on HBASE-11432:


AccessControlConstants which is Public class still have reference
 /** Cell level ACL evaluation strategy */
  public static final String OP_ATTRIBUTE_ACL_STRATEGY = "acl.strategy";
  /** Default cell ACL evaluation strategy: Table and CF first, then ACL */
  public static final byte[] OP_ATTRIBUTE_ACL_STRATEGY_DEFAULT = new byte[] { 0 
};
  /** Alternate cell ACL evaluation strategy: Cell ACL first, then table and CF 
*/
  public static final byte[] OP_ATTRIBUTE_ACL_STRATEGY_CELL_FIRST = new byte[] 
{ 1 };
We need to remove them.  And also in hbase book if any ref ?

> [AccessController] Remove cell first strategy
> -
>
> Key: HBASE-11432
> URL: https://issues.apache.org/jira/browse/HBASE-11432
> Project: HBase
>  Issue Type: Bug
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 0.99.0, 0.98.4
>
> Attachments: HBASE-11432-0.98.patch, HBASE-11432.patch, 
> HBASE-11432.patch
>
>
> The cell first evaluation strategy for cell ACLs has been a problem since 
> introduction. It was an afterthought and a mistake. It's not possible to use 
> at all with the default config on trunk after HBASE-11077. Fairly certain 
> there are no users. Remove. Deprecate related client API methods on Query. 
> (Remove on trunk?)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15783) AccessControlConstants#OP_ATTRIBUTE_ACL_STRATEGY_CELL_FIRST not used any more.

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273716#comment-15273716
 ] 

Anoop Sam John commented on HBASE-15783:


Removed the feature in HBASE-11432.
As I added in a comment in that issue now

AccessControlConstants which is Public class still have reference
/** Cell level ACL evaluation strategy */
public static final String OP_ATTRIBUTE_ACL_STRATEGY = "acl.strategy";
/** Default cell ACL evaluation strategy: Table and CF first, then ACL */
public static final byte[] OP_ATTRIBUTE_ACL_STRATEGY_DEFAULT = new byte[]
{ 0 }
;
/** Alternate cell ACL evaluation strategy: Cell ACL first, then table and CF */
public static final byte[] OP_ATTRIBUTE_ACL_STRATEGY_CELL_FIRST = new byte[]
{ 1 }
;
We need to remove them. And also in hbase book if any ref ?


> AccessControlConstants#OP_ATTRIBUTE_ACL_STRATEGY_CELL_FIRST not used any more.
> --
>
> Key: HBASE-15783
> URL: https://issues.apache.org/jira/browse/HBASE-15783
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 1.3.0, 1.0.4, 1.4.0
>
>
> This is based on a mail in the user list. 
> OP_ATTRIBUTE_ACL_STRATEGY_CELL_FIRST in AccessControlConstants is not used 
> any more in the code and AccessControlconstants is Public. We need to either 
> bring in this feature or remove the constant from the Public APi which may be 
> misleading. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15784) MIsuse core/maxPoolSize of LinkedBlockingQueue in ThreadPoolExecutor

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273786#comment-15273786
 ] 

Anoop Sam John commented on HBASE-15784:


bq.getThreadPool(int maxThreads, int coreThreads,
Better  change this method signature itself to avoid one param.. just pass 
maxThreads.   Similar in other classes also.. This will make sure that later, 
in same class, no one will do the same mistake again.
Pls see whether any of the removed configs mentioned in habse book as well. If 
so we have to clear those as well.
So now we will have much more number of threads initially after the process is 
started.  (As core size is same as max size).. After the time out  period many 
of the threads will get died.

> MIsuse core/maxPoolSize of LinkedBlockingQueue in ThreadPoolExecutor
> 
>
> Key: HBASE-15784
> URL: https://issues.apache.org/jira/browse/HBASE-15784
> Project: HBase
>  Issue Type: Bug
>  Components: Client, Replication, Thrift
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HBASE-15784-v2.patch, HBASE-15784.patch
>
>
> LinkedBlockingQueue is usually used in ThreadPoolExecutor. It allows the 
> thread pool not to be blocked if the number of running threads in the pool is 
> less than the max pool size and the queue is not full.
> But when the core pool size of ThreadPoolExecutor is different from the max 
> pool size, the things don't go as expected. When the number of running 
> threads is the same with the core size, more requests of executions are added 
> into the LinkedBlockingQueue. And the requests can be executed again only 
> when LinkedBlockingQueue is full or some of running threads are finished.
> Thus it is better to use the same value for the core and max pool size when 
> the LinkedBlockingQueue is used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15525:
---
Attachment: HBASE-15525_V3.patch

Thanks for the test and great find [~dvdreddy].  
2nd one is an existing case.. Ya better we handle that also. Fixed the comments.

> OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
> --
>
> Key: HBASE-15525
> URL: https://issues.apache.org/jira/browse/HBASE-15525
> Project: HBase
>  Issue Type: Sub-task
>  Components: IPC/RPC
>Reporter: deepankar
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15525_V1.patch, HBASE-15525_V2.patch, 
> HBASE-15525_V3.patch, HBASE-15525_WIP.patch, WIP.patch
>
>
> After HBASE-13819 the system some times run out of direct memory whenever 
> there is some network congestion or some client side issues.
> This was because of pending RPCs in the RPCServer$Connection.responseQueue 
> and since all the responses in this queue hold a buffer for cellblock from 
> BoundedByteBufferPool this could takeup a lot of memory if the 
> BoundedByteBufferPool's moving average settles down towards a higher value 
> See the discussion here 
> [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15785) Unnecessary lock in ByteBufferArray#asSubByteBuff

[Anoop Sam John (JIRA)]

Anoop Sam John created HBASE-15785:
--

 Summary: Unnecessary lock in ByteBufferArray#asSubByteBuff
 Key: HBASE-15785
 URL: https://issues.apache.org/jira/browse/HBASE-15785
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 2.0.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Fix For: 2.0.0


{code}
 Lock lock = locks[i];
  lock.lock();
  try {
ByteBuffer bb = buffers[i];
if (i == startBuffer) {
  cnt = bufferSize - startBufferOffset;
  if (cnt > len) cnt = len;
  ByteBuffer dup = bb.duplicate();
  dup.limit(startBufferOffset + cnt).position(startBufferOffset);
  mbb[j] = dup.slice();
{code}
In asSubByteBuff, we work on the duplicate BB and set limit and position on 
that.. The locking is not needed here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15757) Reverse scan fails with no obvious cause

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274083#comment-15274083
 ] 

Anoop Sam John commented on HBASE-15757:


bq.After reducing the start and stop row getting results is pretty fast.
Means when u have less rows to scan, there is no issue..  So it is because of 
time outs only.  Another comment u said abt increasing the scan caching.  That 
is not correct in this respect.  U should reduce it.. Every RPC will try to 
fetch more rows.. And reverse scan is not same performing as the normal forward 
scan. It will take more time and when u have more rows to be scanned in one 
RPC, it is sure that the time out happening at client side and it gives a 
retry.  When retry comes, at server, already the old scan request moved the row 
position and so we throw seqNo out of sync exception. Either you have to 
increase the timeout or decrease the #rows to scan in one RPC..  You dont have 
to reduce the range. You have to reduce the scan caching value,

> Reverse scan fails with no obvious cause
> 
>
> Key: HBASE-15757
> URL: https://issues.apache.org/jira/browse/HBASE-15757
> Project: HBase
>  Issue Type: Bug
>  Components: Client, Scanners
>Affects Versions: 0.98.12
> Environment: ubuntu 14.04, amazon cloud; 10 datanodes d2.4xlarge - 
> 16cores, 12x200GB HDD, 122GB RAM
>Reporter: Robert Fiser
>
> related issue on stackoverflow: 
> http://stackoverflow.com/questions/37001169/hbase-reverse-scan-error?noredirect=1#comment61558097_37001169
> this works well:
> scan = new Scan(startRow, stopRow);
> this throws exception sometimes:
> scan = new Scan(stopRow, startRow);
>   scan.setReversed(true);
> throwing exception while traffic is at least 100 req/s. there are actually no 
> timeouts, exception is fired immediately for 1-10% requests
> hbase: 0.98.12-hadoop2;
> hadoop: 2.7.0;
> cluster in AWS, 10 datanodes: d2.4xlarge
> I think it's maybe related with this issue but I'm not using any filters 
> http://apache-hbase.679495.n3.nabble.com/Exception-during-a-reverse-scan-with-filter-td4069721.html
>   java.lang.RuntimeException: 
> org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of 
> OutOfOrderScannerNextException: was there a rpc timeout?
>   at 
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:94)
>   at 
> com.socialbakers.broker.client.hbase.htable.AbstractHtableListScanner.scanToList(AbstractHtableListScanner.java:30)
>   at 
> com.socialbakers.broker.client.hbase.htable.AbstractHtableListSingleScanner.invokeOperation(AbstractHtableListSingleScanner.java:23)
>   at 
> com.socialbakers.broker.client.hbase.htable.AbstractHtableListSingleScanner.invokeOperation(AbstractHtableListSingleScanner.java:11)
>   at 
> com.socialbakers.broker.client.hbase.AbstractHbaseApi.endPointMethod(AbstractHbaseApi.java:40)
>   at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown 
> Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> com.socialbakers.broker.client.Route.invoke(Route.java:241)
>   at 
> com.socialbakers.broker.client.handler.EndpointHandler.invoke(EndpointHandler.java:173)
>   at 
> com.socialbakers.broker.client.handler.EndpointHandler.process(EndpointHandler.java:69)
>   at 
> com.thetransactioncompany.jsonrpc2.server.Dispatcher.process(Dispatcher.java:196)
>   at 
> com.socialbakers.broker.client.RejectableRunnable.run(RejectableRunnable.java:38)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
>   Caused by: 
> org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of 
> OutOfOrderScannerNextException: was there a rpc timeout?
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:430)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:333)
>   at 
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:91)
>   ... 15 more
>   Caused by: 
> org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: 
> org.apache.hadoop.hbase.exceptions.OutOfOrderSca

[jira] [Updated] (HBASE-15785) Unnecessary lock in ByteBufferArray#asSubByteBuff

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-15785:
---
Attachment: HBASE-15785.patch

> Unnecessary lock in ByteBufferArray#asSubByteBuff
> -
>
> Key: HBASE-15785
> URL: https://issues.apache.org/jira/browse/HBASE-15785
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver, Scanners
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15785.patch
>
>
> {code}
>  Lock lock = locks[i];
>   lock.lock();
>   try {
> ByteBuffer bb = buffers[i];
> if (i == startBuffer) {
>   cnt = bufferSize - startBufferOffset;
>   if (cnt > len) cnt = len;
>   ByteBuffer dup = bb.duplicate();
>   dup.limit(startBufferOffset + cnt).position(startBufferOffset);
>   mbb[j] = dup.slice();
> {code}
> In asSubByteBuff, we work on the duplicate BB and set limit and position on 
> that.. The locking is not needed here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)