[jira] [Commented] (HBASE-14921) Memory optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209811#comment-15209811 ] stack commented on HBASE-14921: --- bq. The suggested structure uses 4 times less memory than the previous ConcurrentSkipListMap, provides no fragmentation and it is ready for off-heaping. Excellent. bq. Thanks for taking a look. It looks like the name “CellBlock" made a confusion. Yeah, needs another name... or maybe better to say, there is no defined 'CellBlock' currently -- which is a problem to be addressed -- but if we defined it, it would not be what is here... an implementation of a Map; it would be more basic (A CellScanner only?). I went back to the nice doc. bq. The reference is just a machine word and thus occupies less memory than the Cell Object. What is the 'machine word' and how does it preserve sort? I need to read this doc a few times... On the patch.. Its fun. I like what is going on in here. CellBlockObjectArray presumes passed in array of Cells already sorted? Calls to createCellBlocks are only when making subMaps? Who else would call them? Let me give more review tomorrow. This is interesting... > Memory optimizations > > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15389: -- Attachment: HBASE-15389-0.98.v2.patch > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 0.98.19, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 0.98.19, 1.4.0 > > Attachments: HBASE-15389-0.98.patch, HBASE-15389-0.98.v1.patch, > HBASE-15389-0.98.v2.patch, HBASE-15389-branch-1.patch, HBASE-15389-uc.patch, > HBASE-15389-v1.patch, HBASE-15389-v10.patch, HBASE-15389-v11.patch, > HBASE-15389-v12.patch, HBASE-15389-v13.patch, HBASE-15389-v14.patch, > HBASE-15389-v2.patch, HBASE-15389-v3.patch, HBASE-15389-v4.patch, > HBASE-15389-v5.patch, HBASE-15389-v6.patch, HBASE-15389-v7.patch, > HBASE-15389-v8.patch, HBASE-15389-v9.patch, HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209793#comment-15209793 ] stack commented on HBASE-13819: --- New JIRA! This one is resolved (smlle) [~anoop.hbase] > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209792#comment-15209792 ] stack commented on HBASE-13819: --- bq. Client side can do? Server has to do it. In this case, we did not write the client (asynchbase). Thanks for your #1 and #2. Nice summary. bq. its really difficult for the user to predict how much max off heap size he need to give. Yeah. Lets fix. We could multiply the max queue size in listener * maximum request size to get max offheap buffer allocated Thats kinda ugly. Need the server throttling. bq. (If we go out of space in off heap area)!!! Yes. Need to bound offheap usage. What GC does when offheap up against the limit is probably not pretty. bq. So that is why my thinking abt changing these temp BB creation when happens, those should be HBBs. You still need to do accounting of the spike though. But yeah, probably easier taking the spike onheap Lets discuss in new issue. > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15524) Fix NPE in client-side metrics
[
https://issues.apache.org/jira/browse/HBASE-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209788#comment-15209788
]
Hadoop QA commented on HBASE-15524:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s {color}
| {color:red} HBASE-15524 does not apply to branch-1.3. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.2.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795149/HBASE-15524-branch-1.3.v2.patch
|
| JIRA Issue | HBASE-15524 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1163/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Fix NPE in client-side metrics
> --
>
> Key: HBASE-15524
> URL: https://issues.apache.org/jira/browse/HBASE-15524
> Project: HBase
> Issue Type: Bug
> Components: Client, metrics
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: HBASE-15524-branch-1.3.v1.patch,
> HBASE-15524-branch-1.3.v2.patch
>
>
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.client.MetricsConnection.incrCacheDroppingExceptions(MetricsConnection.java:458)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.updateCachedLocations(ConnectionManager.java:2256)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1179)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:600)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:743)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209785#comment-15209785 ] Anoop Sam John commented on HBASE-15525: For writing the CellBlock response, we really dont need the whole of it in one BB. While writing the response to socket, we have a BufferChain. The Cell block can also be backed by N BBs rather than 1. Even within BufferChain, can see that the actual write to socket happens as chunks of 64 KB size. So if we have a cell block of 512 KB and instead of one BB, we have 4 BBs of size 128 KB each, the socket write wont take more time as the total #writes in both case will be same. With a model of writing cell block to N smaller sized BBs, we can avoid the need for regrow (In BBOutputStream) and copy from old smaller sized BB to new BB. We need a new BBOS which can do this way of write. This one will passed to Encoder for writing the Cells to form cell block. > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209775#comment-15209775 ] Anoop Sam John commented on HBASE-15525: The issue why we go out of off heap memory is because of 2 main reasons 1. The pool has already reached its max capacity of #BBs. And at a given point of time, all are in use. Again other Calls ask pool for BBs for their cell block creation. The pool happily make new BBs which are off heap with each having size of avg running length. And all these cell blocks are tied to Call until the Responder write them to socket. Ya we wont be keeping them in pool. But it is kept as is for loner time specially when the response Q is growing. 2. Even when the response CellBlock need is very low like 12 KB or so, we waste 512 MB per response. Waste in the sense that all the 500 MB is not usable at all. And even the new BBs which pool create on demand (These might not pooled at all as we reach max #BBs in pool) also takes 512 MB per BB. So in a simple way we can say that its really difficult for the user to predict how much max off heap size he need to give. With deepankar case, he is applying some calc based on the max #BBs in pool and max BB size + some additional GBs and set the max off heap size as 5 GB. But this is going wrong.. To explain it with an eg: Consider one configured the max #BBs in pool as 100. And max per item size as 1MB. Means max can have 100 MB off heap consumption by this pool.. Now consider there are lots of reqs and the response Q is big.. Say the 1st 100 response use all BBs from pool. Now again reqs are there and say there are like 100 more adding to Q.. Each one req to pool. It makes BB off heap for those. Means out of the pool we have made double the total max size what we thought it will take.. I agree that we wont store those all BBs in pool and ya the GC may be able to clean it also.. But for some time (untill we clear these response Q) the usage is more. And one more thing for GC is that the full GC only can clean the off heap area? So this in other words cause more full GCs? (If we go out of space in off heap area)!!! So that is why my thinking abt changing these temp BB creation when happens, those should be HBBs. We need to make pool such that we will give a BB back if it is having a free one. When it is not having a free one and capacity is not reached, it makes a new DBB and return. If that is also not the case it wont return any. The BBBPool will make and take back offf heap BBs only. If it can not give, let the caller do what they want (Make on heap BB and make sure dont give back to pool) And abt fixing the size of BBs from pool.. Will write in another comment. This is too big already > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209768#comment-15209768 ] stack commented on HBASE-13819: --- bq, .what do you think ? Thinking on it, a manual config that put upper size on BBs at say 32k and that allowed 4x the amount of buffers to be created might get you over your spike. As to what the heuristic should be, so you don't have to do these configs. lets work that out in the new issue you filed. > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14613) Remove MemStoreChunkPool?
[ https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208316#comment-15208316 ] ramkrishna.s.vasudevan edited comment on HBASE-14613 at 3/24/16 5:03 AM: - bqThat means that if you're promoting a decent amount of objects but not getting enough mixed gen collections, it's possible to get pauses even though there is a lot of memory left. [~eclark] I repeated the experiment and I can see that the case where we have No MSLAB and no chunk pool the GC pause is infact more and the case where we have only MSLAB and no chunk pool the GC pause is lesser but the through put is higher in case the NO MSLAB case. What could be the reason for that? Does your results also go with this justification? was (Author: ram_krish): bqThat means that if you're promoting a decent amount of objects but not getting enough mixed gen collections, it's possible to get pauses even though there is a lot of memory left. [~eclark] I repeated the experiment and I can see that the case where we have No MSLAB and no chunk pool the GC pause is infact more and the case where we have only MSLAB and no chunk pool the GC pause is lesser but the through put is higher than the NO MSLAB case. What could be the reason for that? Does your results also go with this justification? > Remove MemStoreChunkPool? > - > > Key: HBASE-14613 > URL: https://issues.apache.org/jira/browse/HBASE-14613 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 14613-0.98.txt, gc.png, writes.png > > > I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks > of allocations rather than letting the GC handle this. > Now, it's off by default, and it seems to me to be of dubious value. I'd > recommend just removing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209757#comment-15209757 ] Anoop Sam John commented on HBASE-13819: Ya am not saying this on heap create is enough. IMO we should fix the other problem also. The waste of off heap area for responding with a very small sized cell block. I did not add those related comment in the previous one as that was on one item and it was too big already :-)Let me continue in next comment. Or we can switch to new Jira > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15389: -- Attachment: HBASE-15389-v14.patch > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 0.98.19, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 0.98.19, 1.4.0 > > Attachments: HBASE-15389-0.98.patch, HBASE-15389-0.98.v1.patch, > HBASE-15389-branch-1.patch, HBASE-15389-uc.patch, HBASE-15389-v1.patch, > HBASE-15389-v10.patch, HBASE-15389-v11.patch, HBASE-15389-v12.patch, > HBASE-15389-v13.patch, HBASE-15389-v14.patch, HBASE-15389-v2.patch, > HBASE-15389-v3.patch, HBASE-15389-v4.patch, HBASE-15389-v5.patch, > HBASE-15389-v6.patch, HBASE-15389-v7.patch, HBASE-15389-v8.patch, > HBASE-15389-v9.patch, HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15389: -- Attachment: (was: HBASE-15389-v14.patch) > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 0.98.19, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 0.98.19, 1.4.0 > > Attachments: HBASE-15389-0.98.patch, HBASE-15389-0.98.v1.patch, > HBASE-15389-branch-1.patch, HBASE-15389-uc.patch, HBASE-15389-v1.patch, > HBASE-15389-v10.patch, HBASE-15389-v11.patch, HBASE-15389-v12.patch, > HBASE-15389-v13.patch, HBASE-15389-v14.patch, HBASE-15389-v2.patch, > HBASE-15389-v3.patch, HBASE-15389-v4.patch, HBASE-15389-v5.patch, > HBASE-15389-v6.patch, HBASE-15389-v7.patch, HBASE-15389-v8.patch, > HBASE-15389-v9.patch, HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209753#comment-15209753 ] Anoop Sam John commented on HBASE-15525: I made it critical because it make it very difficult for the user to work out what should be the max off heap size > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15525) Fix OutOfMemory that could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15525: --- Priority: Critical (was: Major) > Fix OutOfMemory that could occur when using BoundedByteBufferPool during RPC > bursts > --- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] deepankar updated HBASE-15525: -- Summary: OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts (was: Fix OutOfMemory that could occur when using BoundedByteBufferPool during RPC bursts) > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-15525) Fix OutOfMemory that could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John reassigned HBASE-15525: -- Assignee: Anoop Sam John > Fix OutOfMemory that could occur when using BoundedByteBufferPool during RPC > bursts > --- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209749#comment-15209749 ] deepankar commented on HBASE-13819: --- I totally agree with your idea, but I think practically it would be hard to exhaust out the offheap stuff if the allocation is proportional to the expected response size, one reason why we did not decrease the buffer size from BoundedByteBufferPool is that these large size requests are not that uncommon, they do occur occasionally and we did not want them having to allocate a full size buffer at that time with our initial assumption being with BoundedByteBufferPool there might not be any more allocations at all. May be we should try out internally by restricting max size from BBBP to a much lower value, may be its ok for large RPCs to have to create their own buffers. bq. And one more thing for GC is that the full GC only can clean the off heap area? I think young could also clear them, the problem is that Bits class when there is not enough offheap space calls the System.GC which tr iggers full GC I think but any way I agree this GC is wastefull bq. We need to make pool such that we will give a BB back if it is having a free one. When it is not having a free one and capacity is not reached, it makes a new DBB and return This would be nice, we had a hot patch internally which just fails the request when you see the Bits is going to call System.GC(), this was just temporarily to stop the RegionServer from crashing. > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15493) Default ArrayList size may not be optimal for Mutation
[
https://issues.apache.org/jira/browse/HBASE-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209746#comment-15209746
]
Anoop Sam John commented on HBASE-15493:
[~ram_krish] When Mutation comes to server, we know the total #cells in that.
Not per family count. We dont PB that info. And may be we should not also?
Ya it will be best if we can have a new kind of Mutation only for server. At
server we have 2 advs. We can know the #cells in a mutation. (At least total)
and all the cells are in order. So we dont need a TreeMap at least. If u see
a Mutation , the over head heap size is really big. But we expose Mutation in
our CPs and so FamilyMap which is Map.. We can discuss more on how to reduce
these.. For a major version we may be able to break even some things in CP
side.
[~vrodionov]
When I say addImmutable(byte[] fam, List) am also thinking user should
call it only once.
So if all cells are passed, we may have to split them into family map and again
create new Lists. My idea was to reuse the List created by user. And ya we
say it should be used only when user is read to give the List as immutable.
So they may be to do like
{code}
List f1Cells = new ArrayList<>(3);
f1Cells.add( CellUtil.createCell(...));
f1Cells.add( CellUtil.createCell(...));
f1Cells.add( CellUtil.createCell(...));// 3 cells in f1
put.addImmutable (f1, f1Cells);
List f2Cells = new ArrayList<>(2);
f2Cells.add( CellUtil.createCell(...));
f2Cells.add( CellUtil.createCell(...)); // 2 cells in f2
put.addImmutable (f2, f2Cells);
{code}
If it is one API with taking all family cells,they can reduce 2 lines. Within
mutation we need to create List for each of the cf and we need to use size= 5?
> Default ArrayList size may not be optimal for Mutation
> --
>
> Key: HBASE-15493
> URL: https://issues.apache.org/jira/browse/HBASE-15493
> Project: HBase
> Issue Type: Improvement
> Components: Client, regionserver
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-15493-v1.patch, HBASE-15493-v2.patch
>
>
> {code}
> List getCellList(byte[] family) {
> List list = this.familyMap.get(family);
> if (list == null) {
> list = new ArrayList();
> }
> return list;
> }
> {code}
> Creates list of size 10, this is up to 80 bytes per column family in mutation
> object.
> Suggested:
> {code}
> List getCellList(byte[] family) {
> List list = this.familyMap.get(family);
> if (list == null) {
> list = new ArrayList(CELL_LIST_INITIAL_CAPACITY);
> }
> return list;
> }
> {code}
> CELL_LIST_INITIAL_CAPACITY = 2 in the patch, this is debatable. For mutation
> where every CF has 1 cell, this gives decent reduction in memory allocation
> rate in both client and server during write workload. ~2%, not a big number,
> but as I said, already, memory optimization will include many small steps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14256) Flush task message may be confusing when region is recovered
[
https://issues.apache.org/jira/browse/HBASE-14256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209742#comment-15209742
]
Jerry He commented on HBASE-14256:
--
The timeout test TestAsyncIPC has been flaky, with timeout happening in recent
builds.
> Flush task message may be confusing when region is recovered
>
>
> Key: HBASE-14256
> URL: https://issues.apache.org/jira/browse/HBASE-14256
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0, 1.2.0
>Reporter: Lars George
>Assignee: Gabor Liptak
> Labels: beginner
> Attachments: HBASE-14256.1.patch
>
>
> In {{HRegion.setRecovering()}} we have this code:
> {code}
> // force a flush only if region replication is set up for this region.
> Otherwise no need.
> boolean forceFlush = getTableDesc().getRegionReplication() > 1;
> // force a flush first
> MonitoredTask status = TaskMonitor.get().createStatus(
> "Flushing region " + this + " because recovery is finished");
> try {
> if (forceFlush) {
> internalFlushcache(status);
> }
> {code}
> So we only optionally force flush after a recovery of a region, but the
> message always is set to "Flushing...", which might be confusing. We should
> change the message based on {{forceFlush}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209739#comment-15209739 ] deepankar commented on HBASE-13819: --- I agree with you but what I feel is in the ideal scenario the increase in the heap usage should be proportional to the number of RPCs coming right ?, when we initially allocated the heap size for BBB we accounted for the anticipated burst, but the issue that came was when the size use on the server was couple of orders of magnitude more than this (the actual response waiting to respond was around 80 MB when the total heap usage was around 3.1 GB). what do you think ? > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209736#comment-15209736 ] deepankar commented on HBASE-13819: --- bq. What you think of Anoop's idea of the BB being allocated onheap rather than offheap if we can't get it from the pool? The allocation would be faster... I feel both onheap / offheap allocation could suffer from the same problem as long as we are not tight in the allocation (with less wastage compared to the anticipated response), as an example we somewhat made this error rare by increasing the MaxDirectMemory to a higher value, but in the onheap case also if a user allocated his memory by tightly accounting stuff he could may as well face the issue of unnecessary GCs I think. bq. deepankar Would you mind opening a new issue describing how you would like this to work? created a jira HBASE-15525, we would definitely help in whatever way we can on this. > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14613) Remove MemStoreChunkPool?
[ https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209735#comment-15209735 ] Vladimir Rodionov commented on HBASE-14613: --- I am curios have anybody tried to run HBase with heap > 100GB? The server with 1TB RAM can be found today for less than 10K. > Remove MemStoreChunkPool? > - > > Key: HBASE-14613 > URL: https://issues.apache.org/jira/browse/HBASE-14613 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 14613-0.98.txt, gc.png, writes.png > > > I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks > of allocations rather than letting the GC handle this. > Now, it's off by default, and it seems to me to be of dubious value. I'd > recommend just removing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209732#comment-15209732 ] Anoop Sam John commented on HBASE-13819: Its not case of leak Stack.. Ya I agree that when there is async reqs coming in and the response is some how delayed, there should be some sort of throttling. Client side can do? The issue why we go out of off heap memory is because of 2 main reasons 1. The pool has already reached its max capacity of #BBs. And at a given point of time, all are in use. Again other Calls ask pool for BBs for their cell block creation. The pool happily make new BBs which are off heap with each having size of avg running length. And all these cell blocks are tied to Call until the Responder write them to socket. Ya we wont be keeping them in pool. But it is kept as is for loner time specially when the response Q is growing. 2. Even when the response CellBlock need is very low like 12 KB or so, we waste 512 MB per response. Waste in the sense that all the 500 MB is not usable at all. And even the new BBs which pool create on demand (These might not pooled at all as we reach max #BBs in pool) also takes 512 MB per BB. So in a simple way we can say that its really difficult for the user to predict how much max off heap size he need to give. With [~dvdreddy] case, he is applying some calc based on the max #BBs in pool and max BB size + some additional GBs and set the max off heap size as 5 GB. But this is going wrong.. To explain it with an eg: Consider one configured the max #BBs in pool as 100. And max per item size as 1MB. Means max can have 100 MB off heap consumption by this pool.. Now consider there are lots of reqs and the response Q is big.. Say the 1st 100 response use all BBs from pool. Now again reqs are there and say there are like 100 more adding to Q.. Each one req to pool. It makes BB off heap for those. Means out of the pool we have made double the total max size what we thought it will take.. I agree that we wont store those all BBs in pool and ya the GC may be able to clean it also.. But for some time (untill we clear these response Q) the usage is more. And one more thing for GC is that the full GC only can clean the off heap area? So this in other words cause more full GCs? (If we go out of space in off heap area)!!! So that is why my thinking abt changing these temp BB creation when happens, those should be HBBs. We need to make pool such that we will give a BB back if it is having a free one. When it is not having a free one and capacity is not reached, it makes a new DBB and return. If that is also not the case it wont return any. The BBBPool will make and take back offf heap BBs only. If it can not give, let the caller do what they want (Make on heap BB and make sure dont give back to pool) And abt fixing the size of BBs from pool.. Will write in another comment. This is too big already :-) > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15525) Fix OutOfMemory that could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] deepankar updated HBASE-15525: -- Description: After HBASE-13819 the system some times run out of direct memory whenever there is some network congestion or some client side issues. This was because of pending RPCs in the RPCServer$Connection.responseQueue and since all the responses in this queue hold a buffer for cellblock from BoundedByteBufferPool this could takeup a lot of memory if the BoundedByteBufferPool's moving average settles down towards a higher value See the discussion here [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] was: After HBASE-13819 the system some times run out of direct memory whenever there is some network congestion or some client side issues. This was because of pending RPCs in the RPCServer$Connection.responseQueue and since all the responses in this queue hold a buffer for cellblock from BoundedByteBufferPool this could takeup a lot of memory if the BoundedByteBufferPool's moving average settles down towards a higher value See the discussion here https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822 > Fix OutOfMemory that could occur when using BoundedByteBufferPool during RPC > bursts > --- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Reporter: deepankar > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15525) Fix OutOfMemory that could occur when using BoundedByteBufferPool during RPC bursts
deepankar created HBASE-15525: - Summary: Fix OutOfMemory that could occur when using BoundedByteBufferPool during RPC bursts Key: HBASE-15525 URL: https://issues.apache.org/jira/browse/HBASE-15525 Project: HBase Issue Type: Bug Components: IPC/RPC Reporter: deepankar After HBASE-13819 the system some times run out of direct memory whenever there is some network congestion or some client side issues. This was because of pending RPCs in the RPCServer$Connection.responseQueue and since all the responses in this queue hold a buffer for cellblock from BoundedByteBufferPool this could takeup a lot of memory if the BoundedByteBufferPool's moving average settles down towards a higher value See the discussion here https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15493) Default ArrayList size may not be optimal for Mutation
[
https://issues.apache.org/jira/browse/HBASE-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209725#comment-15209725
]
Vladimir Rodionov commented on HBASE-15493:
---
{quote}
Maybe we should add something like add(byte[] cf, List cells)
{quote}
How about addImmutable(List cells)?
It is allowed to mix different column families, but if this method is called
*only once* (and it is supposed to be called once) we always will have right
sizes in family map?
> Default ArrayList size may not be optimal for Mutation
> --
>
> Key: HBASE-15493
> URL: https://issues.apache.org/jira/browse/HBASE-15493
> Project: HBase
> Issue Type: Improvement
> Components: Client, regionserver
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-15493-v1.patch, HBASE-15493-v2.patch
>
>
> {code}
> List getCellList(byte[] family) {
> List list = this.familyMap.get(family);
> if (list == null) {
> list = new ArrayList();
> }
> return list;
> }
> {code}
> Creates list of size 10, this is up to 80 bytes per column family in mutation
> object.
> Suggested:
> {code}
> List getCellList(byte[] family) {
> List list = this.familyMap.get(family);
> if (list == null) {
> list = new ArrayList(CELL_LIST_INITIAL_CAPACITY);
> }
> return list;
> }
> {code}
> CELL_LIST_INITIAL_CAPACITY = 2 in the patch, this is debatable. For mutation
> where every CF has 1 cell, this gives decent reduction in memory allocation
> rate in both client and server during write workload. ~2%, not a big number,
> but as I said, already, memory optimization will include many small steps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15510) Reuse buffer inside RpcServer$Connection.readAndProcess
[ https://issues.apache.org/jira/browse/HBASE-15510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209721#comment-15209721 ] ramkrishna.s.vasudevan commented on HBASE-15510: bq.But the avg pause time is increased. We are working in this area now. The recent analysis indicates that this avg pause time is increasing for cases where we have huge number of cells (approx 1k cells) in a row. There is something that happens here such that the cells are getting promoted to the old gen and hence the ParNEw allocation failure time is more. > Reuse buffer inside RpcServer$Connection.readAndProcess > --- > > Key: HBASE-15510 > URL: https://issues.apache.org/jira/browse/HBASE-15510 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > ByteBuffer is created on every call. > Server-side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14613) Remove MemStoreChunkPool?
[ https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209719#comment-15209719 ] ramkrishna.s.vasudevan commented on HBASE-14613: Thanks [~eclark]. I think it would be worth spending time here before deciding on using the offheap BBPool and offheap memstore chunking for the offheap write path POC, considering that it should work with G1GC. > Remove MemStoreChunkPool? > - > > Key: HBASE-14613 > URL: https://issues.apache.org/jira/browse/HBASE-14613 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl >Priority: Minor > Attachments: 14613-0.98.txt, gc.png, writes.png > > > I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks > of allocations rather than letting the GC handle this. > Now, it's off by default, and it seems to me to be of dubious value. I'd > recommend just removing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15493) Default ArrayList size may not be optimal for Mutation
[
https://issues.apache.org/jira/browse/HBASE-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209717#comment-15209717
]
ramkrishna.s.vasudevan commented on HBASE-15493:
IMHO this operation of finding the exact size is useful in the SErver side too.
Because with large cells per mutations I could see this list#add() taking more
time. I thought of creating some thing like ServerMutation extending Mutations
(and it is private) but the problem is that it becomes complicated when we have
Cells backed by Offheap buffer.
May be this new API we can add only on server side something like that? I have
not worked on end to end to take that patch to completion.
> Default ArrayList size may not be optimal for Mutation
> --
>
> Key: HBASE-15493
> URL: https://issues.apache.org/jira/browse/HBASE-15493
> Project: HBase
> Issue Type: Improvement
> Components: Client, regionserver
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-15493-v1.patch, HBASE-15493-v2.patch
>
>
> {code}
> List getCellList(byte[] family) {
> List list = this.familyMap.get(family);
> if (list == null) {
> list = new ArrayList();
> }
> return list;
> }
> {code}
> Creates list of size 10, this is up to 80 bytes per column family in mutation
> object.
> Suggested:
> {code}
> List getCellList(byte[] family) {
> List list = this.familyMap.get(family);
> if (list == null) {
> list = new ArrayList(CELL_LIST_INITIAL_CAPACITY);
> }
> return list;
> }
> {code}
> CELL_LIST_INITIAL_CAPACITY = 2 in the patch, this is debatable. For mutation
> where every CF has 1 cell, this gives decent reduction in memory allocation
> rate in both client and server during write workload. ~2%, not a big number,
> but as I said, already, memory optimization will include many small steps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209716#comment-15209716 ] Heng Chen commented on HBASE-14703: --- Oh, thanks for your reminder, [~busbey] > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-branch-1.patch, HBASE-14703-branch-1.v1.patch, > HBASE-14703-branch-1_v1.patch, HBASE-14703-start.patch, > HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209714#comment-15209714
]
Sean Busbey commented on HBASE-14703:
-
{quote}
oh, ok. my bad. please ignore me
{quote}
no worries, perfectly reasonable question to have. AFAICT from a glance through
the relevant patches, the conflicts were in fact resolved.
> HTable.mutateRow does not collect stats
> ---
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-branch-1.patch, HBASE-14703-branch-1.v1.patch,
> HBASE-14703-branch-1_v1.patch, HBASE-14703-start.patch,
> HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch,
> HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch,
> HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch,
> HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch,
> HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch,
> HBASE-14703_v9.patch
>
>
> We are trying to fix the stats implementation, by moving it out of the Result
> object and into an Rpc payload (but not the 'cell payload', just as part of
> the values returned from the request). This change will also us use easily
> switch to AsyncProcess as the executor, and support stats, for nearly all the
> rpc calls. However, that means when you upgrade the client or server, you
> will lose stats visibility until the other side is upgraded. We could keep
> around the Result based stats storage to accommodate the old api and send
> both stats back from the server (in each result and in the rpc payload).
> Note that we will still be wire compatible - protobufs mean we can just ride
> over the lack of information.
> The other tricky part of this is that Result has a
> non-InterfaceAudience.Private getStatistics() method (along with two
> InterfaceAudience.Private addResults and setStatistics methods), so we might
> need a release to deprecate the getStats() method before throwing it out?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209711#comment-15209711
]
Sean Busbey commented on HBASE-14703:
-
one thing for next time, [~chenheng], please use the {{--signoff}} flag in git
to add a line that looks like:
{code}
Signed-off-by: chenheng
{code}
to the commit message rather than include your name as a parenthetical (that's
how we indicate the original author if they aren't the author on the commit
itself).
> HTable.mutateRow does not collect stats
> ---
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-branch-1.patch, HBASE-14703-branch-1.v1.patch,
> HBASE-14703-branch-1_v1.patch, HBASE-14703-start.patch,
> HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch,
> HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch,
> HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch,
> HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch,
> HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch,
> HBASE-14703_v9.patch
>
>
> We are trying to fix the stats implementation, by moving it out of the Result
> object and into an Rpc payload (but not the 'cell payload', just as part of
> the values returned from the request). This change will also us use easily
> switch to AsyncProcess as the executor, and support stats, for nearly all the
> rpc calls. However, that means when you upgrade the client or server, you
> will lose stats visibility until the other side is upgraded. We could keep
> around the Result based stats storage to accommodate the old api and send
> both stats back from the server (in each result and in the rpc payload).
> Note that we will still be wire compatible - protobufs mean we can just ride
> over the lack of information.
> The other tricky part of this is that Result has a
> non-InterfaceAudience.Private getStatistics() method (along with two
> InterfaceAudience.Private addResults and setStatistics methods), so we might
> need a release to deprecate the getStats() method before throwing it out?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209709#comment-15209709 ] Jesse Yates commented on HBASE-14703: - oh, ok. my bad. please ignore me :) > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-branch-1.patch, HBASE-14703-branch-1.v1.patch, > HBASE-14703-branch-1_v1.patch, HBASE-14703-start.patch, > HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209708#comment-15209708 ] Sean Busbey commented on HBASE-14703: - usually, when we cherry pick things that end up having conflicts we leave the list of conflicted paths in the commit message. that way, if things went wrong somehow those that follow have some pointers at which files weren't cleanly changed to begin with. > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-branch-1.patch, HBASE-14703-branch-1.v1.patch, > HBASE-14703-branch-1_v1.patch, HBASE-14703-start.patch, > HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15481) Add pre/post roll to WALObserver
[ https://issues.apache.org/jira/browse/HBASE-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209706#comment-15209706 ] Sean Busbey commented on HBASE-15481: - reasoning makes sense to me. > Add pre/post roll to WALObserver > > > Key: HBASE-15481 > URL: https://issues.apache.org/jira/browse/HBASE-15481 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.3.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15481-v0.patch, HBASE-15481-v1.patch, > HBASE-15481-v1.patch > > > currently the WALObserver has only a pre/post Write. It will be useful to > have a pre/post Roll too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209705#comment-15209705 ] Jesse Yates commented on HBASE-14703: - Yeah, the conflicts. Generally, you should resolve those and cleanup the commit, AFAIK. > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-branch-1.patch, HBASE-14703-branch-1.v1.patch, > HBASE-14703-branch-1_v1.patch, HBASE-14703-start.patch, > HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15520) Fix broken TestAsyncIPC
[
https://issues.apache.org/jira/browse/HBASE-15520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209704#comment-15209704
]
Sean Busbey commented on HBASE-15520:
-
{quote}
Sean Busbey Can we run Pre-Commit Admin job on the jenkins server machine
directly? It is not a heavy job I suppose? Thanks.
{quote}
I haven't historically been a maintainer for the admin job. I'm not sure what
its requirements are.
> Fix broken TestAsyncIPC
> ---
>
> Key: HBASE-15520
> URL: https://issues.apache.org/jira/browse/HBASE-15520
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15520-v1.patch, HBASE-15520.patch
>
>
> There are two problems
> 1. AsyncIPC will throw IOException when connection reset so we need to change
> the catch type in testRpcMaxRequestSize.
> 2. AsyncRpcChannel does not deal with channelInactive event in netty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209701#comment-15209701 ] Heng Chen commented on HBASE-14703: --- Yeah, [~jesse_yates], i cherry-pick it from master. Is there any problem? > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-branch-1.patch, HBASE-14703-branch-1.v1.patch, > HBASE-14703-branch-1_v1.patch, HBASE-14703-start.patch, > HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209691#comment-15209691
]
Jesse Yates commented on HBASE-14703:
-
did you cherry pick that over? The commit message is... concerning:
{code}
commit ee33bf0c7d539a63e2bf84c888ff8cf8bb57b7f6
Author: Jesse Yates
Date: Fri Mar 4 19:07:59 2016 -0800
HBASE-14703 HTable.mutateRow does not collect stats (Heng Chen)
Conflicts:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerFactory.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerImpl.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/StatsTrackingRpcRetryingCaller.java
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
{code}
> HTable.mutateRow does not collect stats
> ---
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-branch-1.patch, HBASE-14703-branch-1.v1.patch,
> HBASE-14703-branch-1_v1.patch, HBASE-14703-start.patch,
> HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch,
> HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch,
> HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch,
> HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch,
> HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch,
> HBASE-14703_v9.patch
>
>
> We are trying to fix the stats implementation, by moving it out of the Result
> object and into an Rpc payload (but not the 'cell payload', just as part of
> the values returned from the request). This change will also us use easily
> switch to AsyncProcess as the executor, and support stats, for nearly all the
> rpc calls. However, that means when you upgrade the client or server, you
> will lose stats visibility until the other side is upgraded. We could keep
> around the Result based stats storage to accommodate the old api and send
> both stats back from the server (in each result and in the rpc payload).
> Note that we will still be wire compatible - protobufs mean we can just ride
> over the lack of information.
> The other tricky part of this is that Result has a
> non-InterfaceAudience.Private getStatistics() method (along with two
> InterfaceAudience.Private addResults and setStatistics methods), so we might
> need a release to deprecate the getStats() method before throwing it out?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-14703: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-branch-1.patch, HBASE-14703-branch-1.v1.patch, > HBASE-14703-branch-1_v1.patch, HBASE-14703-start.patch, > HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209680#comment-15209680 ] Heng Chen edited comment on HBASE-14703 at 3/24/16 3:46 AM: push to branch-1 and branch-1.3 was (Author: chenheng): push to branch-1 > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-branch-1.patch, HBASE-14703-branch-1.v1.patch, > HBASE-14703-branch-1_v1.patch, HBASE-14703-start.patch, > HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15524) Fix NPE in client-side metrics
[ https://issues.apache.org/jira/browse/HBASE-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209683#comment-15209683 ] Elliott Clark commented on HBASE-15524: --- +1 > Fix NPE in client-side metrics > -- > > Key: HBASE-15524 > URL: https://issues.apache.org/jira/browse/HBASE-15524 > Project: HBase > Issue Type: Bug > Components: Client, metrics >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > Attachments: HBASE-15524-branch-1.3.v1.patch, > HBASE-15524-branch-1.3.v2.patch > > > java.lang.NullPointerException > at > org.apache.hadoop.hbase.client.MetricsConnection.incrCacheDroppingExceptions(MetricsConnection.java:458) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.updateCachedLocations(ConnectionManager.java:2256) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1179) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:600) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:743) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15524) Fix NPE in client-side metrics
[ https://issues.apache.org/jira/browse/HBASE-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15524: Status: Patch Available (was: Open) > Fix NPE in client-side metrics > -- > > Key: HBASE-15524 > URL: https://issues.apache.org/jira/browse/HBASE-15524 > Project: HBase > Issue Type: Bug > Components: Client, metrics >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > Attachments: HBASE-15524-branch-1.3.v1.patch, > HBASE-15524-branch-1.3.v2.patch > > > java.lang.NullPointerException > at > org.apache.hadoop.hbase.client.MetricsConnection.incrCacheDroppingExceptions(MetricsConnection.java:458) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.updateCachedLocations(ConnectionManager.java:2256) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1179) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:600) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:743) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) HTable.mutateRow does not collect stats
[ https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209680#comment-15209680 ] Heng Chen commented on HBASE-14703: --- push to branch-1 > HTable.mutateRow does not collect stats > --- > > Key: HBASE-14703 > URL: https://issues.apache.org/jira/browse/HBASE-14703 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-14702_v5.2_addendum-addendum.patch, > HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch, > HBASE-14703-branch-1.patch, HBASE-14703-branch-1.v1.patch, > HBASE-14703-branch-1_v1.patch, HBASE-14703-start.patch, > HBASE-14703-v4.1.patch, HBASE-14703-v4.patch, > HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch, > HBASE-14703_v1.patch, HBASE-14703_v10.patch, HBASE-14703_v10.patch, > HBASE-14703_v11.patch, HBASE-14703_v12.patch, HBASE-14703_v13.patch, > HBASE-14703_v2.patch, HBASE-14703_v3.patch, HBASE-14703_v5.1.patch, > HBASE-14703_v5.2.patch, HBASE-14703_v5.patch, HBASE-14703_v6-addendum.patch, > HBASE-14703_v6.patch, HBASE-14703_v7.patch, HBASE-14703_v8.patch, > HBASE-14703_v9.patch > > > We are trying to fix the stats implementation, by moving it out of the Result > object and into an Rpc payload (but not the 'cell payload', just as part of > the values returned from the request). This change will also us use easily > switch to AsyncProcess as the executor, and support stats, for nearly all the > rpc calls. However, that means when you upgrade the client or server, you > will lose stats visibility until the other side is upgraded. We could keep > around the Result based stats storage to accommodate the old api and send > both stats back from the server (in each result and in the rpc payload). > Note that we will still be wire compatible - protobufs mean we can just ride > over the lack of information. > The other tricky part of this is that Result has a > non-InterfaceAudience.Private getStatistics() method (along with two > InterfaceAudience.Private addResults and setStatistics methods), so we might > need a release to deprecate the getStats() method before throwing it out? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15524) Fix NPE in client-side metrics
[ https://issues.apache.org/jira/browse/HBASE-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15524: Status: Open (was: Patch Available) > Fix NPE in client-side metrics > -- > > Key: HBASE-15524 > URL: https://issues.apache.org/jira/browse/HBASE-15524 > Project: HBase > Issue Type: Bug > Components: Client, metrics >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > Attachments: HBASE-15524-branch-1.3.v1.patch, > HBASE-15524-branch-1.3.v2.patch > > > java.lang.NullPointerException > at > org.apache.hadoop.hbase.client.MetricsConnection.incrCacheDroppingExceptions(MetricsConnection.java:458) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.updateCachedLocations(ConnectionManager.java:2256) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1179) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:600) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:743) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15524) Fix NPE in client-side metrics
[
https://issues.apache.org/jira/browse/HBASE-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209679#comment-15209679
]
Hadoop QA commented on HBASE-15524:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
5s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 59s
{color} | {color:green} branch-1.3 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 39s
{color} | {color:green} branch-1.3 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
6s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s
{color} | {color:green} branch-1.3 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s
{color} | {color:green} branch-1.3 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
52m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 29s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 0s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795118/HBASE-15524-branch-1.3.v1.patch
|
| JIRA Issue | HBASE-15524 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux priapus.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT
Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-1.3 / 396e4f1 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /home/jenkins/tools/java/jdk1.8.0:1.8.0
[jira] [Work started] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.
[ https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-15523 started by yi liang. > enhance hbase-daemon.sh to enable autorestart. > -- > > Key: HBASE-15523 > URL: https://issues.apache.org/jira/browse/HBASE-15523 > Project: HBase > Issue Type: Improvement >Reporter: yi liang >Assignee: yi liang >Priority: Minor > > enhance hbase-daemon.sh to enable autorestart. > component(like master, region server) will auto-start when terminated/killed > abnormally if >(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. > export HBASE_AUTORESTART=true > (b) Then add the following 3 simple lines(59-61) to > /bin/hbase-daemon.sh > > 51 # get arguments > 52 startStop=$1 > 53 shift > 54 > 55 command=$1 > 56 shift > 57 > 58 #make sure the auto-restart are default settings > 59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; > then > 60 startStop="autorestart" > 61 fi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15411) Rewrite backup with Procedure V2
[
https://issues.apache.org/jira/browse/HBASE-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209640#comment-15209640
]
Ted Yu edited comment on HBASE-15411 at 3/24/16 3:29 AM:
-
One comment is for not going through Connection when doing snapshot.
Here is code from MasterRpcServices#snapshot() :
{code}
master.snapshotManager.takeSnapshot(snapshot);
{code}
One option is for HMaster to pass snapshotManager to FullTableBackupProcedure.
But this won't work because FullTableBackupProcedure#executeFromState() may
stop anywhere.
After which:
{code}
public FullTableBackupProcedure() {
// Required by the Procedure framework to create the procedure on replay
{code}
Meaning we should be prepared that full back continue from any state.
Suggestion for other approach is welcome.
was (Author: yuzhih...@gmail.com):
One comment is for not going through Connection when doing snapshot.
Here is code from MasterRpcServices#snapshot() :
{code}
master.snapshotManager.takeSnapshot(snapshot);
{code}
Option is for HMaster to pass snapshotManager to FullTableBackupProcedure.
I want to confirm this is acceptable.
Otherwise, suggestion for other approach is welcome.
> Rewrite backup with Procedure V2
>
>
> Key: HBASE-15411
> URL: https://issues.apache.org/jira/browse/HBASE-15411
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15411-v1.txt, 15411-v11.txt, 15411-v12.txt,
> 15411-v13.txt, 15411-v14.txt, 15411-v15.txt, 15411-v16.txt, 15411-v18.txt,
> 15411-v22.txt, 15411-v3.txt, 15411-v5.txt, 15411-v6.txt, 15411-v7.txt,
> 15411-v9.txt, FullTableBackupProcedure.java
>
>
> Currently full / incremental backup is driven by BackupHandler (see call()
> method for flow).
> This issue is to rewrite the flow using Procedure V2.
> States (enum) for full / incremental backup would be introduced in
> Backup.proto which correspond to the steps performed in BackupHandler#call().
> executeFromState() would pace the backup based on the current state.
> serializeStateData() / deserializeStateData() would be used to persist state
> into procedure WAL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15508) Add command for exporting snapshot in hbase command script
[ https://issues.apache.org/jira/browse/HBASE-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209667#comment-15209667 ] Hudson commented on HBASE-15508: FAILURE: Integrated in HBase-1.1-JDK7 #1688 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1688/]) HBASE-15508 Add command for exporting snapshot in hbase command script (matteo.bertozzi: rev 7032c0cb76941b47229b56c00c4637b70630e007) * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java * bin/hbase * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java > Add command for exporting snapshot in hbase command script > -- > > Key: HBASE-15508 > URL: https://issues.apache.org/jira/browse/HBASE-15508 > Project: HBase > Issue Type: Improvement > Components: hbase, scripts, snapshots >Affects Versions: 1.2.0, 1.1.2, 1.1.3 >Reporter: Yufeng Jiang >Assignee: Yufeng Jiang >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.2.2 > > Attachments: HBASE-15508-branch-1.1-and-1.2.patch, > HBASE-15508-branch-1.1-and-1.2.patch.v2, HBASE-15508-branch-2.0.patch, > HBASE-15508-branch-master.patch > > > Users of the hbase command script can now use the 'hbase snapshot export’ > command. This replaces the need to type the full class name of > 'org.apache.hadoop.hbase.snapshot.ExportSnapshot' > In addition to adding command 'snapshot export', we are also grouping > snapshot related commands together as subcommands under 'hbase snapshot'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15520) Fix broken TestAsyncIPC
[ https://issues.apache.org/jira/browse/HBASE-15520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15520: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-1.3+. Thanks [~stack] for reviewing. > Fix broken TestAsyncIPC > --- > > Key: HBASE-15520 > URL: https://issues.apache.org/jira/browse/HBASE-15520 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15520-v1.patch, HBASE-15520.patch > > > There are two problems > 1. AsyncIPC will throw IOException when connection reset so we need to change > the catch type in testRpcMaxRequestSize. > 2. AsyncRpcChannel does not deal with channelInactive event in netty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15524) Fix NPE in client-side metrics
[ https://issues.apache.org/jira/browse/HBASE-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15524: Status: Open (was: Patch Available) > Fix NPE in client-side metrics > -- > > Key: HBASE-15524 > URL: https://issues.apache.org/jira/browse/HBASE-15524 > Project: HBase > Issue Type: Bug > Components: Client, metrics >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > Attachments: HBASE-15524-branch-1.3.v1.patch, > HBASE-15524-branch-1.3.v2.patch > > > java.lang.NullPointerException > at > org.apache.hadoop.hbase.client.MetricsConnection.incrCacheDroppingExceptions(MetricsConnection.java:458) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.updateCachedLocations(ConnectionManager.java:2256) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1179) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:600) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:743) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15520) Fix broken TestAsyncIPC
[ https://issues.apache.org/jira/browse/HBASE-15520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209659#comment-15209659 ] Duo Zhang commented on HBASE-15520: --- Checked locally, the checkstyle errors in https://builds.apache.org/job/PreCommit-HBASE-Build/1153/artifact/patchprocess/diff-checkstyle-hbase-client.txt were all gone. Seems the Pre-Commit Admin job has been waiting for an executor for a long time, so let's commit this first. [~busbey] Can we run Pre-Commit Admin job on the jenkins server machine directly? It is not a heavy job I suppose? Thanks. > Fix broken TestAsyncIPC > --- > > Key: HBASE-15520 > URL: https://issues.apache.org/jira/browse/HBASE-15520 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15520-v1.patch, HBASE-15520.patch > > > There are two problems > 1. AsyncIPC will throw IOException when connection reset so we need to change > the catch type in testRpcMaxRequestSize. > 2. AsyncRpcChannel does not deal with channelInactive event in netty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15524) Fix NPE in client-side metrics
[ https://issues.apache.org/jira/browse/HBASE-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15524: Status: Patch Available (was: Open) > Fix NPE in client-side metrics > -- > > Key: HBASE-15524 > URL: https://issues.apache.org/jira/browse/HBASE-15524 > Project: HBase > Issue Type: Bug > Components: Client, metrics >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > Attachments: HBASE-15524-branch-1.3.v1.patch, > HBASE-15524-branch-1.3.v2.patch > > > java.lang.NullPointerException > at > org.apache.hadoop.hbase.client.MetricsConnection.incrCacheDroppingExceptions(MetricsConnection.java:458) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.updateCachedLocations(ConnectionManager.java:2256) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1179) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:600) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:743) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15524) Fix NPE in client-side metrics
[ https://issues.apache.org/jira/browse/HBASE-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15524: Attachment: HBASE-15524-branch-1.3.v2.patch > Fix NPE in client-side metrics > -- > > Key: HBASE-15524 > URL: https://issues.apache.org/jira/browse/HBASE-15524 > Project: HBase > Issue Type: Bug > Components: Client, metrics >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > Attachments: HBASE-15524-branch-1.3.v1.patch, > HBASE-15524-branch-1.3.v2.patch > > > java.lang.NullPointerException > at > org.apache.hadoop.hbase.client.MetricsConnection.incrCacheDroppingExceptions(MetricsConnection.java:458) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.updateCachedLocations(ConnectionManager.java:2256) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1179) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:600) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:743) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15360) Fix flaky TestSimpleRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-15360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209642#comment-15209642 ] Hudson commented on HBASE-15360: SUCCESS: Integrated in HBase-1.3-IT #578 (See [https://builds.apache.org/job/HBase-1.3-IT/578/]) HBASE-15360 Fix flaky TestSimpleRpcScheduler; ADDENDUM (stack: rev a3d550fbcaf2001588b42a2404e73e9864e385d6) * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/AdaptiveLifoCoDelCallQueue.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java > Fix flaky TestSimpleRpcScheduler > > > Key: HBASE-15360 > URL: https://issues.apache.org/jira/browse/HBASE-15360 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Mikhail Antonov >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15360-addendum.patch, HBASE-15360-addendum.patch, > HBASE-15360-addendum.patch, HBASE-15360.patch > > > There were several flaky tests added there as part of HBASE-15306 and likely > HBASE-15136. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15392) Single Cell Get reads two HFileBlocks
[
https://issues.apache.org/jira/browse/HBASE-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209641#comment-15209641
]
Hudson commented on HBASE-15392:
FAILURE: Integrated in HBase-Trunk_matrix #802 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/802/])
HBASE-15392 Single Cell Get reads two HFileBlocks (stack: rev
d7a4499dfc8b3936a0eca867589fc2b23b597866)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueScanFixture.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/KeyValueScanFixture.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java
> Single Cell Get reads two HFileBlocks
> -
>
> Key: HBASE-15392
> URL: https://issues.apache.org/jira/browse/HBASE-15392
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 15392-0.98-looksee.txt, 15392.wip.patch,
> 15392v2.wip.patch, 15392v3.wip.patch, 15392v4.patch, 15392v5.patch,
> 15392v6.patch, 15392v7 (1).patch, 15392v7.patch, 15392v7.patch,
> 15392v7.patch, 15392v7.patch, 15392v7.patch, 15392v9.patch,
> HBASE-15265-v8.patch, HBASE-15392_suggest.patch, gc.png, gc.png, io.png,
> no_optimize.patch, no_optimize.patch, reads.png, reads.png, two_seeks.txt
>
>
> As found by Daniel "SystemTap" Pol, a simple Get results in our reading two
> HFileBlocks, the one that contains the wanted Cell, and the block that
> follows.
> Here is a bit of custom logging that logs a stack trace on each HFileBlock
> read so you can see the call stack responsible:
> {code}
> 2016-03-03 22:20:30,191 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> START LOOP
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner:
> QCODE SEEK_NEXT_COL
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex:
> STARTED WHILE
> 2016-03-03 22:20:30,192 INFO
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.CombinedBlockCache:
> OUT OF L2
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.BucketCache: Read
> offset=31409152, len=2103
> 2016-03-03 22:20:30,192 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.FileIOEngine:
> offset=31409152, length=2103
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> From Cache [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> 2016-03-03 22:20:30,193 TRACE
> [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl:
> Cache hit return [blockType=DATA, fileOffset=2055421, headerSize=33,
> onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020,
> prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053,
> getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true,
> buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54],
> dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231,
> fileContext=[usesHBaseChecksum=true, checksumType=CRC32C,
> bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true,
> includesTags=false, compressAlgo=NONE, compressTags=false,
> cryptoContext=[cipher=NONE keyHash=NONE]]]
> java.lang.Throwable
> at
>
[jira] [Commented] (HBASE-15475) Allow TimestampsFilter to provide a seek hint
[ https://issues.apache.org/jira/browse/HBASE-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209643#comment-15209643 ] Hudson commented on HBASE-15475: SUCCESS: Integrated in HBase-1.3-IT #578 (See [https://builds.apache.org/job/HBase-1.3-IT/578/]) HBASE-15475 Allow TimestampsFilter to provide a seek hint (eclark: rev 604415e82739d049180c8c2624a84b185b50244d) * hbase-protocol/src/main/protobuf/Filter.proto * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/FilterProtos.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTimestampFilterSeekHint.java * hbase-client/src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java > Allow TimestampsFilter to provide a seek hint > - > > Key: HBASE-15475 > URL: https://issues.apache.org/jira/browse/HBASE-15475 > Project: HBase > Issue Type: Improvement > Components: Client, Filters, regionserver >Affects Versions: 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15475-branch-1.patch, HBASE-15475-v1.patch, > HBASE-15475-v2.patch, HBASE-15475-v3.patch, HBASE-15475-v4.patch, > HBASE-15475-v5.patch, HBASE-15475.patch > > > If a user wants to read specific timestamps currently it's a very linear > scan. This is so that all deletes can be respected. However if the user > doesn't care about deletes it can dramatically speed up the request to seek. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15411) Rewrite backup with Procedure V2
[
https://issues.apache.org/jira/browse/HBASE-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209640#comment-15209640
]
Ted Yu commented on HBASE-15411:
One comment is for not going through Connection when doing snapshot.
Here is code from MasterRpcServices#snapshot() :
{code}
master.snapshotManager.takeSnapshot(snapshot);
{code}
Option is for HMaster to pass snapshotManager to FullTableBackupProcedure.
I want to confirm this is acceptable.
Otherwise, suggestion for other approach is welcome.
> Rewrite backup with Procedure V2
>
>
> Key: HBASE-15411
> URL: https://issues.apache.org/jira/browse/HBASE-15411
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15411-v1.txt, 15411-v11.txt, 15411-v12.txt,
> 15411-v13.txt, 15411-v14.txt, 15411-v15.txt, 15411-v16.txt, 15411-v18.txt,
> 15411-v22.txt, 15411-v3.txt, 15411-v5.txt, 15411-v6.txt, 15411-v7.txt,
> 15411-v9.txt, FullTableBackupProcedure.java
>
>
> Currently full / incremental backup is driven by BackupHandler (see call()
> method for flow).
> This issue is to rewrite the flow using Procedure V2.
> States (enum) for full / incremental backup would be introduced in
> Backup.proto which correspond to the steps performed in BackupHandler#call().
> executeFromState() would pace the backup based on the current state.
> serializeStateData() / deserializeStateData() would be used to persist state
> into procedure WAL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15475) Allow TimestampsFilter to provide a seek hint
[ https://issues.apache.org/jira/browse/HBASE-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15475: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.4.0 Status: Resolved (was: Patch Available) > Allow TimestampsFilter to provide a seek hint > - > > Key: HBASE-15475 > URL: https://issues.apache.org/jira/browse/HBASE-15475 > Project: HBase > Issue Type: Improvement > Components: Client, Filters, regionserver >Affects Versions: 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15475-branch-1.patch, HBASE-15475-v1.patch, > HBASE-15475-v2.patch, HBASE-15475-v3.patch, HBASE-15475-v4.patch, > HBASE-15475-v5.patch, HBASE-15475.patch > > > If a user wants to read specific timestamps currently it's a very linear > scan. This is so that all deletes can be respected. However if the user > doesn't care about deletes it can dramatically speed up the request to seek. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15520) Fix broken TestAsyncIPC
[ https://issues.apache.org/jira/browse/HBASE-15520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15520: -- Attachment: HBASE-15520-v1.patch Fix the checkstyle issues > Fix broken TestAsyncIPC > --- > > Key: HBASE-15520 > URL: https://issues.apache.org/jira/browse/HBASE-15520 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15520-v1.patch, HBASE-15520.patch > > > There are two problems > 1. AsyncIPC will throw IOException when connection reset so we need to change > the catch type in testRpcMaxRequestSize. > 2. AsyncRpcChannel does not deal with channelInactive event in netty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15411) Rewrite backup with Procedure V2
[ https://issues.apache.org/jira/browse/HBASE-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209603#comment-15209603 ] Ted Yu commented on HBASE-15411: git is quite smart: when BackupHandler is removed from git, it shows BackupHandler being renamed to FullTableBackupProcedure > Rewrite backup with Procedure V2 > > > Key: HBASE-15411 > URL: https://issues.apache.org/jira/browse/HBASE-15411 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15411-v1.txt, 15411-v11.txt, 15411-v12.txt, > 15411-v13.txt, 15411-v14.txt, 15411-v15.txt, 15411-v16.txt, 15411-v18.txt, > 15411-v22.txt, 15411-v3.txt, 15411-v5.txt, 15411-v6.txt, 15411-v7.txt, > 15411-v9.txt, FullTableBackupProcedure.java > > > Currently full / incremental backup is driven by BackupHandler (see call() > method for flow). > This issue is to rewrite the flow using Procedure V2. > States (enum) for full / incremental backup would be introduced in > Backup.proto which correspond to the steps performed in BackupHandler#call(). > executeFromState() would pace the backup based on the current state. > serializeStateData() / deserializeStateData() would be used to persist state > into procedure WAL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15389) Write out multiple files when compaction
[ https://issues.apache.org/jira/browse/HBASE-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209575#comment-15209575 ] Duo Zhang commented on HBASE-15389: --- Oh, the patch for 0.98 should be stale. I will provide a new one soon after I finish HBASE-15520. All pre-commit builds are broken due to that issue. > Write out multiple files when compaction > > > Key: HBASE-15389 > URL: https://issues.apache.org/jira/browse/HBASE-15389 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 0.98.19, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 0.98.19, 1.4.0 > > Attachments: HBASE-15389-0.98.patch, HBASE-15389-0.98.v1.patch, > HBASE-15389-branch-1.patch, HBASE-15389-uc.patch, HBASE-15389-v1.patch, > HBASE-15389-v10.patch, HBASE-15389-v11.patch, HBASE-15389-v12.patch, > HBASE-15389-v13.patch, HBASE-15389-v14.patch, HBASE-15389-v2.patch, > HBASE-15389-v3.patch, HBASE-15389-v4.patch, HBASE-15389-v5.patch, > HBASE-15389-v6.patch, HBASE-15389-v7.patch, HBASE-15389-v8.patch, > HBASE-15389-v9.patch, HBASE-15389.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.
[ https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He reopened HBASE-15523: -- Could you provide a patch, [~easyliangjob]? > enhance hbase-daemon.sh to enable autorestart. > -- > > Key: HBASE-15523 > URL: https://issues.apache.org/jira/browse/HBASE-15523 > Project: HBase > Issue Type: Improvement >Reporter: yi liang >Assignee: yi liang >Priority: Minor > > enhance hbase-daemon.sh to enable autorestart. > component(like master, region server) will auto-start when terminated/killed > abnormally if >(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. > export HBASE_AUTORESTART=true > (b) Then add the following 3 simple lines(59-61) to > /bin/hbase-daemon.sh > > 51 # get arguments > 52 startStop=$1 > 53 shift > 54 > 55 command=$1 > 56 shift > 57 > 58 #make sure the auto-restart are default settings > 59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; > then > 60 startStop="autorestart" > 61 fi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15524) Fix NPE in client-side metrics
[ https://issues.apache.org/jira/browse/HBASE-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209569#comment-15209569 ] Elliott Clark commented on HBASE-15524: --- Can we do the same try/catch-log for receiveMultiAction calling updateServerStats. While it wasn't the cause this time it would be good to protect against the same issue. Other than that looks good to me. We need to clean up the client so much. > Fix NPE in client-side metrics > -- > > Key: HBASE-15524 > URL: https://issues.apache.org/jira/browse/HBASE-15524 > Project: HBase > Issue Type: Bug > Components: Client, metrics >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > Attachments: HBASE-15524-branch-1.3.v1.patch > > > java.lang.NullPointerException > at > org.apache.hadoop.hbase.client.MetricsConnection.incrCacheDroppingExceptions(MetricsConnection.java:458) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.updateCachedLocations(ConnectionManager.java:2256) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1179) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:600) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:743) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15360) Fix flaky TestSimpleRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-15360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209565#comment-15209565 ] Hudson commented on HBASE-15360: FAILURE: Integrated in HBase-1.3 #620 (See [https://builds.apache.org/job/HBase-1.3/620/]) HBASE-15360 Fix flaky TestSimpleRpcScheduler; ADDENDUM (stack: rev 76a71084f92ae1c082c276c02a6998a65b819f15) * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/AdaptiveLifoCoDelCallQueue.java > Fix flaky TestSimpleRpcScheduler > > > Key: HBASE-15360 > URL: https://issues.apache.org/jira/browse/HBASE-15360 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Mikhail Antonov >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15360-addendum.patch, HBASE-15360-addendum.patch, > HBASE-15360-addendum.patch, HBASE-15360.patch > > > There were several flaky tests added there as part of HBASE-15306 and likely > HBASE-15136. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14256) Flush task message may be confusing when region is recovered
[
https://issues.apache.org/jira/browse/HBASE-14256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209536#comment-15209536
]
Hadoop QA commented on HBASE-14256:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 44s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 24s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 59s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
60m 17s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 54m 8s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 164m 28s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.ipc.TestAsyncIPC |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12774946/HBASE-14256.1.patch |
| JIRA Issue | HBASE-14256 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d7a4499 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions |
[jira] [Commented] (HBASE-15360) Fix flaky TestSimpleRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-15360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209524#comment-15209524 ] Hudson commented on HBASE-15360: FAILURE: Integrated in HBase-1.4 #48 (See [https://builds.apache.org/job/HBase-1.4/48/]) HBASE-15360 Fix flaky TestSimpleRpcScheduler; ADDENDUM (stack: rev a3d550fbcaf2001588b42a2404e73e9864e385d6) * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/AdaptiveLifoCoDelCallQueue.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java > Fix flaky TestSimpleRpcScheduler > > > Key: HBASE-15360 > URL: https://issues.apache.org/jira/browse/HBASE-15360 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Mikhail Antonov >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15360-addendum.patch, HBASE-15360-addendum.patch, > HBASE-15360-addendum.patch, HBASE-15360.patch > > > There were several flaky tests added there as part of HBASE-15306 and likely > HBASE-15136. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15333) Enhance the filter to handle short, integer, long, float and double
[
https://issues.apache.org/jira/browse/HBASE-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209527#comment-15209527
]
Hadoop QA commented on HBASE-15333:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 28s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
35m 33s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 32s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 36s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795079/HBASE-15333-2.patch |
| JIRA Issue | HBASE-15333 |
| Optional Tests | asflicense scalac scaladoc unit compile |
| uname | Linux asf910.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d7a4499 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/1160/testReport/ |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1160/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Enhance the filter to handle short, integer, long, float and double
> ---
>
> Key: HBASE-15333
> URL: https://issues.apache.org/jira/browse/HBASE-15333
> Project: HBase
> Issue Type: Sub-task
>Reporter: Zhan Zhang
>Assignee: Zhan Zhang
> Attachments: HBASE-15333-1.patch, HBASE-15333-2.patch
>
>
> Currently, the range filter is based on the order of bytes. But for java
> primitive type, such as short, int, long, double, float, etc, their order is
> not consistent with their byte order, extra manipulation has to be in place
> to take care of them correctly.
> For example, for the integer range (-100, 100), the filter <= 1, the current
> filter will return 0 and 1, and the right return value should be (-100, 1]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.
[ https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209522#comment-15209522 ] Dima Spivak commented on HBASE-15523: - Hm? > enhance hbase-daemon.sh to enable autorestart. > -- > > Key: HBASE-15523 > URL: https://issues.apache.org/jira/browse/HBASE-15523 > Project: HBase > Issue Type: Improvement >Reporter: yi liang >Assignee: yi liang >Priority: Minor > > enhance hbase-daemon.sh to enable autorestart. > component(like master, region server) will auto-start when terminated/killed > abnormally if >(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. > export HBASE_AUTORESTART=true > (b) Then add the following 3 simple lines(59-61) to > /bin/hbase-daemon.sh > > 51 # get arguments > 52 startStop=$1 > 53 shift > 54 > 55 command=$1 > 56 shift > 57 > 58 #make sure the auto-restart are default settings > 59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; > then > 60 startStop="autorestart" > 61 fi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15524) Fix NPE in client-side metrics
[ https://issues.apache.org/jira/browse/HBASE-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15524: Status: Patch Available (was: Open) > Fix NPE in client-side metrics > -- > > Key: HBASE-15524 > URL: https://issues.apache.org/jira/browse/HBASE-15524 > Project: HBase > Issue Type: Bug > Components: Client, metrics >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > Attachments: HBASE-15524-branch-1.3.v1.patch > > > java.lang.NullPointerException > at > org.apache.hadoop.hbase.client.MetricsConnection.incrCacheDroppingExceptions(MetricsConnection.java:458) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.updateCachedLocations(ConnectionManager.java:2256) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1179) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:600) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:743) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15508) Add command for exporting snapshot in hbase command script
[ https://issues.apache.org/jira/browse/HBASE-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209492#comment-15209492 ] Hudson commented on HBASE-15508: FAILURE: Integrated in HBase-1.1-JDK8 #1774 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1774/]) HBASE-15508 Add command for exporting snapshot in hbase command script (matteo.bertozzi: rev 7032c0cb76941b47229b56c00c4637b70630e007) * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * bin/hbase > Add command for exporting snapshot in hbase command script > -- > > Key: HBASE-15508 > URL: https://issues.apache.org/jira/browse/HBASE-15508 > Project: HBase > Issue Type: Improvement > Components: hbase, scripts, snapshots >Affects Versions: 1.2.0, 1.1.2, 1.1.3 >Reporter: Yufeng Jiang >Assignee: Yufeng Jiang >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.2.2 > > Attachments: HBASE-15508-branch-1.1-and-1.2.patch, > HBASE-15508-branch-1.1-and-1.2.patch.v2, HBASE-15508-branch-2.0.patch, > HBASE-15508-branch-master.patch > > > Users of the hbase command script can now use the 'hbase snapshot export’ > command. This replaces the need to type the full class name of > 'org.apache.hadoop.hbase.snapshot.ExportSnapshot' > In addition to adding command 'snapshot export', we are also grouping > snapshot related commands together as subcommands under 'hbase snapshot'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15524) Fix NPE in client-side metrics
[ https://issues.apache.org/jira/browse/HBASE-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15524: Attachment: HBASE-15524-branch-1.3.v1.patch > Fix NPE in client-side metrics > -- > > Key: HBASE-15524 > URL: https://issues.apache.org/jira/browse/HBASE-15524 > Project: HBase > Issue Type: Bug > Components: Client, metrics >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > Attachments: HBASE-15524-branch-1.3.v1.patch > > > java.lang.NullPointerException > at > org.apache.hadoop.hbase.client.MetricsConnection.incrCacheDroppingExceptions(MetricsConnection.java:458) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.updateCachedLocations(ConnectionManager.java:2256) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1179) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:600) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:743) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15524) Fix NPE in client-side metrics
[ https://issues.apache.org/jira/browse/HBASE-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15524: Description: java.lang.NullPointerException at org.apache.hadoop.hbase.client.MetricsConnection.incrCacheDroppingExceptions(MetricsConnection.java:458) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.updateCachedLocations(ConnectionManager.java:2256) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1179) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:600) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:743) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) > Fix NPE in client-side metrics > -- > > Key: HBASE-15524 > URL: https://issues.apache.org/jira/browse/HBASE-15524 > Project: HBase > Issue Type: Bug > Components: Client, metrics >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Critical > Fix For: 1.3.0 > > > java.lang.NullPointerException > at > org.apache.hadoop.hbase.client.MetricsConnection.incrCacheDroppingExceptions(MetricsConnection.java:458) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.updateCachedLocations(ConnectionManager.java:2256) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1179) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1200(AsyncProcess.java:600) > at > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:743) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15524) Fix NPE in client-side metrics
Mikhail Antonov created HBASE-15524: --- Summary: Fix NPE in client-side metrics Key: HBASE-15524 URL: https://issues.apache.org/jira/browse/HBASE-15524 Project: HBase Issue Type: Bug Components: Client, metrics Affects Versions: 1.3.0 Reporter: Mikhail Antonov Assignee: Mikhail Antonov Priority: Critical Fix For: 1.3.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15360) Fix flaky TestSimpleRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-15360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209448#comment-15209448 ] stack commented on HBASE-15360: --- Pushed addendum to branch-1 and branch-1.3 too. > Fix flaky TestSimpleRpcScheduler > > > Key: HBASE-15360 > URL: https://issues.apache.org/jira/browse/HBASE-15360 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Mikhail Antonov >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15360-addendum.patch, HBASE-15360-addendum.patch, > HBASE-15360-addendum.patch, HBASE-15360.patch > > > There were several flaky tests added there as part of HBASE-15306 and likely > HBASE-15136. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15508) Add command for exporting snapshot in hbase command script
[ https://issues.apache.org/jira/browse/HBASE-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209442#comment-15209442 ] Hudson commented on HBASE-15508: FAILURE: Integrated in HBase-Trunk_matrix #801 (See [https://builds.apache.org/job/HBase-Trunk_matrix/801/]) HBASE-15508 Add command for exporting snapshot in hbase command script (matteo.bertozzi: rev 7177e4be64d134144ddc64b8635b06c7d3141f0c) * bin/hbase * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java > Add command for exporting snapshot in hbase command script > -- > > Key: HBASE-15508 > URL: https://issues.apache.org/jira/browse/HBASE-15508 > Project: HBase > Issue Type: Improvement > Components: hbase, scripts, snapshots >Affects Versions: 1.2.0, 1.1.2, 1.1.3 >Reporter: Yufeng Jiang >Assignee: Yufeng Jiang >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.2.2 > > Attachments: HBASE-15508-branch-1.1-and-1.2.patch, > HBASE-15508-branch-1.1-and-1.2.patch.v2, HBASE-15508-branch-2.0.patch, > HBASE-15508-branch-master.patch > > > Users of the hbase command script can now use the 'hbase snapshot export’ > command. This replaces the need to type the full class name of > 'org.apache.hadoop.hbase.snapshot.ExportSnapshot' > In addition to adding command 'snapshot export', we are also grouping > snapshot related commands together as subcommands under 'hbase snapshot'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15300) Upgrade to zookeeper 3.4.8
[ https://issues.apache.org/jira/browse/HBASE-15300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209441#comment-15209441 ] Hudson commented on HBASE-15300: FAILURE: Integrated in HBase-Trunk_matrix #801 (See [https://builds.apache.org/job/HBase-Trunk_matrix/801/]) HBASE-15300 Upgrade to zookeeper 3.4.8 (tedyu: rev 95e6d2276b710e386cf010bb9072af7293fdb890) * pom.xml > Upgrade to zookeeper 3.4.8 > -- > > Key: HBASE-15300 > URL: https://issues.apache.org/jira/browse/HBASE-15300 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Labels: zookeeper > Fix For: 2.0.0 > > Attachments: HBASE-15300.v1.patch, HBASE-15300.v2.patch, > HBASE-15300.v2.patch, HBASE-15300.v2.patch, HBASE-1533.v1.patch > > > zookeeper 3.4.8 has been released. > Among the fixes the following are desirable: > ZOOKEEPER-706 large numbers of watches can cause session re-establishment to > fail > ZOOKEEPER-1797 PurgeTxnLog may delete data logs during roll > This issue upgrades zookeeper dependency to 3.4.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209439#comment-15209439 ] stack commented on HBASE-13819: --- What would you like [~deenarang]? The burst to show onheap or offheap? Even if we did fixed size, you'd still get a graph like this, right when the traffic comes on hard? (Traffic is up when you see the above spike, is it?) > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209432#comment-15209432 ] stack commented on HBASE-13819: --- We do seem to be bottlenecking on responding. I'm not sure if it this or client not pulling the responses fast enough. This is an area to dig in on. > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[
https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209427#comment-15209427
]
stack commented on HBASE-13819:
---
Good to hear you don't think there leaking going on.
bq. No concrete evidence that responder is not able to keep up, but the bound
in pool does not help this case because we create a new BB when one is not
present in the pool and occasionally (we are observing once in 2 - 3 days)
there will be spew when returns to pool grows above the configured threshold.
Messages saying ... LOG.warn("Overflowed total capacity."); or
LOG.warn("At capacity: " + countOfBuffers); ?
They don't look that useful. Might be good though to report detail around the
burst in traffic so could adjust sizing
What you think of Anoop's idea of the BB being allocated onheap rather than
offheap if we can't get it from the pool? The allocation would be faster...
So, growing because of a burst in traffic.. but no leaking... because when the
burst passes, we seem to shrink down again (if noisy about it...)
[~dvdreddy] Would you mind opening a new issue describing how you would like
this to work? Its great that you fellas are running it and uncovering this
stuff. It would be easy enough to do you up patches for you to try since it so
contained. Thanks.
> Make RPC layer CellBlock buffer a DirectByteBuffer
> --
>
> Key: HBASE-13819
> URL: https://issues.apache.org/jira/browse/HBASE-13819
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch,
> HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png
>
>
> In RPC layer, when we make a cellBlock to put as RPC payload, we will make an
> on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto
> certain number of buffers. This jira aims at testing possibility for making
> this buffers off heap ones. (DBB) The advantages
> 1. Unsafe based writes to off heap is faster than that to on heap. Now we are
> not using unsafe based writes at all. Even if we add, DBB will be better
> 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap
> writes will be better
> 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer
> to the socket channel, it will create a temp DBB and copy data to there and
> only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can
> avoid this one more level of copying.
> Will do different perf testing with changed and report back.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209425#comment-15209425 ] deepankar commented on HBASE-13819: --- Attached image of the metric over time. we are running bucket cache of size around 69427MB and the parameters of BBBP are 2048 max entries and max size is 1MB, trace showed current moving avg is 512 kb !q.png! > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] deepankar updated HBASE-13819: -- Attachment: q.png > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209414#comment-15209414 ] deepankar commented on HBASE-13819: --- bq. In description above though, talk is of Responder queue backed up which seems to say we are not clearing the server of finished responses fast enough Is it possible that the issues on client side we are not able to push the responses instead of Responder being slow ? because we do try sending the responses from the handler itself if it is possible which points somewhat the issue could be on client side ? > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209412#comment-15209412 ] stack commented on HBASE-13819: --- bq. What Anoop Sam John said is correct, in our case avg response sizes are around 10 -16 kb but the max size is 512kb if you run for significantly longer time the running avg goest toward 512kb. I'd think this a good thing, no? You are seeing some traffic that is 512kb... and the BBBP can accommodate without a resize. If you do find it a problem, change "hbase.ipc.server.reservoir.max.buffer.size". Its default is 1024 * 1024. bq. Another point its not like that there are lot of request if you have around 200 clients and lets say for these 4000 requests is only 82 MB its atleast 1 - 2 orders of magnitude of space getting occupied extra. This sounds like we are leaking? The JVM publishes how much memory is being used in the jmx metric java.nio.type=BufferPool,name=direct Does this grow over time? > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209409#comment-15209409 ] deepankar commented on HBASE-13819: --- bq. Which Q we talking? The is config to put bounds on the call queue where we will reject calls... The one that reject calls only accounts for incoming request sizes and there is default 2GB limit I think. The response sizes are not accounted in this I think. The Queue is RpcServer$Connection.responseQueue. bq. Yeah, there is heuristic and we grow till we hit an average. Are we saying we grew to 512k and then afterward, all calls were 16k only? Is this a problem? In our use case our worst case response size is 512 kb (hard capped from client side) and our avg response size is between 12kb, what we observe is after 3 - 4 days of running almost always the moving avg is at 512 kb and in the heap dump all the response buffers is of size 512 kb > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209401#comment-15209401 ] deepankar commented on HBASE-13819: --- bq. Because we are not returning BB to the pool? The pool is growing w/o bound? I think there are no leaks in BB from the analysis on the heap dump, all the objects were accounted bq. We should add these if not present at TRACE level. Sorry My comment was misled, by debug statements I meant by enabling TRACE (these loggings were most useful stuff in many debugging scenarios) bq. So 2G instead of 1G? But the pool is bounded? bq. The responder is not keeping up? It is not moving stuff out of the server fast enough? bq. I am interested in leaks; how they are happening. No concrete evidence that responder is not able to keep up, but the bound in pool does not help this case because we create a new BB when one is not present in the pool and occasionally (we are observing once in 2 - 3 days) there will be spew when returns to pool grows above the configured threshold. >From the analysis we did there are no leaks bq. Where is pendingCallsQueue? The queue per connection object RpcServer$Connection.responseQueue bq. Did you observe the offheap size used growing? There s a metric IIRC. Yes we saw this in the metric (hbase.regionserver.direct.MemoryUsed) bq. Where would the fixed size be? In BBBP they eventually reach fixed size? Yes they eventually reach fixed size, but the size is that of the larger response sizes rather than median or some smaller number > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209398#comment-15209398 ] stack commented on HBASE-13819: --- The BBBP is configurable. You can set upper bound on BBBP member size. bq. Still IMO these on demand BB creation should not be on off heap area. Interesting. Why you say? Because it slower doing it offheap? What is wrong doing it offheap otherwise? Why is the offheap BB not discarded? So, I think this the issue... the server is overrun and so burst our bounds on the BBBP and when at limit, new BB's are allocated offheap... so we go outside of our expected offheap usage. Is there leaking going on here? But we have means of bounding the queues of inbound traffic. In description above though, talk is of Responder queue backed up which seems to say we are not clearing the server of finished responses fast enough > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209388#comment-15209388 ] stack commented on HBASE-13819: --- Which Q we talking? The is config to put bounds on the call queue where we will reject calls... bq. The case is row reads. So normally the response size is much lesser than this 512 KB. Still we reserve those many for each of the BB. Yeah, there is heuristic and we grow till we hit an average. Are we saying we grew to 512k and then afterward, all calls were 16k only? Is this a problem? Yeah, I like the idea for the BBBP made of fixed-size BBs. Move the code for BucketCache out here. > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15493) Default ArrayList size may not be optimal for Mutation
[
https://issues.apache.org/jira/browse/HBASE-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209385#comment-15209385
]
Vladimir Rodionov commented on HBASE-15493:
---
{quote}
Lets do it 'right' or not bother.
{quote}
Brilliant words, somebody should have said that back in 2006-2007 :)
> Default ArrayList size may not be optimal for Mutation
> --
>
> Key: HBASE-15493
> URL: https://issues.apache.org/jira/browse/HBASE-15493
> Project: HBase
> Issue Type: Improvement
> Components: Client, regionserver
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-15493-v1.patch, HBASE-15493-v2.patch
>
>
> {code}
> List getCellList(byte[] family) {
> List list = this.familyMap.get(family);
> if (list == null) {
> list = new ArrayList();
> }
> return list;
> }
> {code}
> Creates list of size 10, this is up to 80 bytes per column family in mutation
> object.
> Suggested:
> {code}
> List getCellList(byte[] family) {
> List list = this.familyMap.get(family);
> if (list == null) {
> list = new ArrayList(CELL_LIST_INITIAL_CAPACITY);
> }
> return list;
> }
> {code}
> CELL_LIST_INITIAL_CAPACITY = 2 in the patch, this is debatable. For mutation
> where every CF has 1 cell, this gives decent reduction in memory allocation
> rate in both client and server during write workload. ~2%, not a big number,
> but as I said, already, memory optimization will include many small steps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.
yi liang created HBASE-15523: Summary: enhance hbase-daemon.sh to enable autorestart. Key: HBASE-15523 URL: https://issues.apache.org/jira/browse/HBASE-15523 Project: HBase Issue Type: Improvement Reporter: yi liang Assignee: yi liang Priority: Minor enhance hbase-daemon.sh to enable autorestart. component(like master, region server) will auto-start when terminated/killed abnormally if (a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. export HBASE_AUTORESTART=true (b) Then add the following 3 simple lines(59-61) to /bin/hbase-daemon.sh 51 # get arguments 52 startStop=$1 53 shift 54 55 command=$1 56 shift 57 58 #make sure the auto-restart are default settings 59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; then 60 startStop="autorestart" 61 fi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.
[ https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yi liang resolved HBASE-15523. -- Resolution: Fixed > enhance hbase-daemon.sh to enable autorestart. > -- > > Key: HBASE-15523 > URL: https://issues.apache.org/jira/browse/HBASE-15523 > Project: HBase > Issue Type: Improvement >Reporter: yi liang >Assignee: yi liang >Priority: Minor > > enhance hbase-daemon.sh to enable autorestart. > component(like master, region server) will auto-start when terminated/killed > abnormally if >(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. > export HBASE_AUTORESTART=true > (b) Then add the following 3 simple lines(59-61) to > /bin/hbase-daemon.sh > > 51 # get arguments > 52 startStop=$1 > 53 shift > 54 > 55 command=$1 > 56 shift > 57 > 58 #make sure the auto-restart are default settings > 59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; > then > 60 startStop="autorestart" > 61 fi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209378#comment-15209378 ] Walter Koetke commented on HBASE-12148: --- Lars, can you be more specific about what the problem is with having just this one implementation, as opposed to one that's synchronized and one that is not? Unless there is a mistake in this patched TimeRangeTracker as implemented, the intent is that it would be thread safe and support both usages, with the AtomicReference, the compareAndSet, etc. > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: Walter Koetke > Fix For: 2.0.0, 1.3.0, 0.98.19 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.txt, 12148.txt, 12148v2.txt, 12148v2.txt, > HBASE-12148-V3.patch, HBASE-12148.txt, HBASE-12148V2.txt, Screen Shot > 2014-10-01 at 3.39.46 PM.png, Screen Shot 2014-10-01 at 3.41.07 PM.png, > TimeRangeTracker.tiff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209371#comment-15209371 ] stack commented on HBASE-13819: --- bq. We recently pulled this patch internally and are seeing some significant side effects of BoundedByteBufferPool where the system runs out of direct memory whenever there is some network congestion or some client side issues. Because we are not returning BB to the pool? The pool is growing w/o bound? bq. Buffer size settles to 512 kb over time(from debug statements we put in) We should add these if not present at TRACE level. bq. number of rpcs that were queued in the responder was around ~4000 and this leads to exhaustion of the direct buffer space, So 2G instead of 1G? But the pool is bounded? The responder is not keeping up? It is not moving stuff out of the server fast enough? Where is pendingCallsQueue? Did you observe the offheap size used growing? There s a metric IIRC. I am interested in leaks; how they are happening. bq. ...and having a fixed size buffers instead Where would the fixed size be? In BBBP they eventually reach fixed size? Thanks. > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15475) Allow TimestampsFilter to provide a seek hint
[ https://issues.apache.org/jira/browse/HBASE-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209338#comment-15209338 ] Mikhail Antonov commented on HBASE-15475: - +1 > Allow TimestampsFilter to provide a seek hint > - > > Key: HBASE-15475 > URL: https://issues.apache.org/jira/browse/HBASE-15475 > Project: HBase > Issue Type: Improvement > Components: Client, Filters, regionserver >Affects Versions: 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15475-branch-1.patch, HBASE-15475-v1.patch, > HBASE-15475-v2.patch, HBASE-15475-v3.patch, HBASE-15475-v4.patch, > HBASE-15475-v5.patch, HBASE-15475.patch > > > If a user wants to read specific timestamps currently it's a very linear > scan. This is so that all deletes can be respected. However if the user > doesn't care about deletes it can dramatically speed up the request to seek. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15453) [Performance] Considering reverting HBASE-10015 - reinstate synchronized in StoreScanner
[ https://issues.apache.org/jira/browse/HBASE-15453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209318#comment-15209318 ] Andrew Purtell commented on HBASE-15453: Agreed, make the change. We should all be advocating 8 runtimes. > [Performance] Considering reverting HBASE-10015 - reinstate synchronized in > StoreScanner > > > Key: HBASE-15453 > URL: https://issues.apache.org/jira/browse/HBASE-15453 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Attachments: 15453-0.98.txt > > > In HBASE-10015 back then I found that intrinsic locks (synchronized) in > StoreScanner are slower that explicit locks. > I was surprised by this. To make sure I added a simple perf test and many > folks ran it on their machines. All found that explicit locks were faster. > Now... I just ran that test again. On the latest JDK8 I find that now the > intrinsic locks are significantly faster: > (OpenJDK Runtime Environment (build 1.8.0_72-b15)) > Explicit locks: > 10 runs mean:2223.6 sigma:72.29412147609237 > Intrinsic locks: > 10 runs mean:1865.3 sigma:32.63755505548784 > I confirmed the same with timing some Phoenix scans. We can save a bunch of > time by changing this back > Arrghhh... So maybe it's time to revert this now...? > (Note that in trunk due to [~ram_krish]'s work, we do not lock in > StoreScanner anymore) > I'll attach the perf test and a patch that changes lock to synchronized, if > some folks could run this on 0.98, that'd be great. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15333) Enhance the filter to handle short, integer, long, float and double
[ https://issues.apache.org/jira/browse/HBASE-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhan Zhang updated HBASE-15333: --- Attachment: HBASE-15333-2.patch > Enhance the filter to handle short, integer, long, float and double > --- > > Key: HBASE-15333 > URL: https://issues.apache.org/jira/browse/HBASE-15333 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: HBASE-15333-1.patch, HBASE-15333-2.patch > > > Currently, the range filter is based on the order of bytes. But for java > primitive type, such as short, int, long, double, float, etc, their order is > not consistent with their byte order, extra manipulation has to be in place > to take care of them correctly. > For example, for the integer range (-100, 100), the filter <= 1, the current > filter will return 0 and 1, and the right return value should be (-100, 1] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15508) Add command for exporting snapshot in hbase command script
[ https://issues.apache.org/jira/browse/HBASE-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209287#comment-15209287 ] Hudson commented on HBASE-15508: SUCCESS: Integrated in HBase-1.2 #585 (See [https://builds.apache.org/job/HBase-1.2/585/]) HBASE-15508 Add command for exporting snapshot in hbase command script (matteo.bertozzi: rev 9725a5bd737116198795f35259aee2531779e610) * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * bin/hbase * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java > Add command for exporting snapshot in hbase command script > -- > > Key: HBASE-15508 > URL: https://issues.apache.org/jira/browse/HBASE-15508 > Project: HBase > Issue Type: Improvement > Components: hbase, scripts, snapshots >Affects Versions: 1.2.0, 1.1.2, 1.1.3 >Reporter: Yufeng Jiang >Assignee: Yufeng Jiang >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.2.2 > > Attachments: HBASE-15508-branch-1.1-and-1.2.patch, > HBASE-15508-branch-1.1-and-1.2.patch.v2, HBASE-15508-branch-2.0.patch, > HBASE-15508-branch-master.patch > > > Users of the hbase command script can now use the 'hbase snapshot export’ > command. This replaces the need to type the full class name of > 'org.apache.hadoop.hbase.snapshot.ExportSnapshot' > In addition to adding command 'snapshot export', we are also grouping > snapshot related commands together as subcommands under 'hbase snapshot'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15515) Improve LocalityBasedCandidateGenerator in Balancer
[
https://issues.apache.org/jira/browse/HBASE-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209302#comment-15209302
]
Ted Yu commented on HBASE-15515:
{code}
+ * Returns a least loaded server which have better locality for this region
+ * than the current server.
{code}
which have -> which has
Is it possible to add test for the issues described above ?
> Improve LocalityBasedCandidateGenerator in Balancer
> ---
>
> Key: HBASE-15515
> URL: https://issues.apache.org/jira/browse/HBASE-15515
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15515-v1.patch, HBASE-15515.patch
>
>
> There are some problems which need to fix.
> 1. LocalityBasedCandidateGenerator.getLowestLocalityRegionOnServer should
> skip empty region.
> 2. When use LocalityBasedCandidateGenerator to generate Cluster.Action, it
> should add random operation instead of pickLowestLocalityServer(cluster).
> Because the search function may stuck here if it always generate the same
> Cluster.Action.
> 3. getLeastLoadedTopServerForRegion should get least loaded server which have
> better locality than current server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15508) Add command for exporting snapshot in hbase command script
[ https://issues.apache.org/jira/browse/HBASE-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209286#comment-15209286 ] Hudson commented on HBASE-15508: FAILURE: Integrated in HBase-1.3 #619 (See [https://builds.apache.org/job/HBase-1.3/619/]) HBASE-15508 Add command for exporting snapshot in hbase command script (matteo.bertozzi: rev fc966c840d7fce71e3875a895d2a75d746464b48) * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java * bin/hbase * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java > Add command for exporting snapshot in hbase command script > -- > > Key: HBASE-15508 > URL: https://issues.apache.org/jira/browse/HBASE-15508 > Project: HBase > Issue Type: Improvement > Components: hbase, scripts, snapshots >Affects Versions: 1.2.0, 1.1.2, 1.1.3 >Reporter: Yufeng Jiang >Assignee: Yufeng Jiang >Priority: Minor > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.2.2 > > Attachments: HBASE-15508-branch-1.1-and-1.2.patch, > HBASE-15508-branch-1.1-and-1.2.patch.v2, HBASE-15508-branch-2.0.patch, > HBASE-15508-branch-master.patch > > > Users of the hbase command script can now use the 'hbase snapshot export’ > command. This replaces the need to type the full class name of > 'org.apache.hadoop.hbase.snapshot.ExportSnapshot' > In addition to adding command 'snapshot export', we are also grouping > snapshot related commands together as subcommands under 'hbase snapshot'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15493) Default ArrayList size may not be optimal for Mutation
[
https://issues.apache.org/jira/browse/HBASE-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209295#comment-15209295
]
stack commented on HBASE-15493:
---
This would be preferable. Client could pay the price of the calculation and
volunteer the info to the serverside. Instantiating new object, could use this
hint if present.
> Default ArrayList size may not be optimal for Mutation
> --
>
> Key: HBASE-15493
> URL: https://issues.apache.org/jira/browse/HBASE-15493
> Project: HBase
> Issue Type: Improvement
> Components: Client, regionserver
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-15493-v1.patch, HBASE-15493-v2.patch
>
>
> {code}
> List getCellList(byte[] family) {
> List list = this.familyMap.get(family);
> if (list == null) {
> list = new ArrayList();
> }
> return list;
> }
> {code}
> Creates list of size 10, this is up to 80 bytes per column family in mutation
> object.
> Suggested:
> {code}
> List getCellList(byte[] family) {
> List list = this.familyMap.get(family);
> if (list == null) {
> list = new ArrayList(CELL_LIST_INITIAL_CAPACITY);
> }
> return list;
> }
> {code}
> CELL_LIST_INITIAL_CAPACITY = 2 in the patch, this is debatable. For mutation
> where every CF has 1 cell, this gives decent reduction in memory allocation
> rate in both client and server during write workload. ~2%, not a big number,
> but as I said, already, memory optimization will include many small steps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
