[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13836022#comment-13836022 ] Nicolas Liochon commented on HBASE-9535: bq. You have diff of GC use? Not really. I've looked at the gc through the profile, it was around 4 minor gc/s with CMS, 1/s with G1, w/ or w/o the patch. I don't think the patch makes a difference in this scenario, as with YCSB puts we have a few large allocations, instead of ismultiple tiny ones. So for this scenario I think the benefit comes from the array copy suppressed. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: Nicolas Liochon Attachments: 9535.v1.patch, 9535.v2-trunk.patch, 9535.v2.patch, 9535.v3.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[
https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835330#comment-13835330
]
Hadoop QA commented on HBASE-9535:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616360/9535.v2-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.ipc.TestIPC
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8029//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8029//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8029//console
This message is automatically generated.
Try a pool of direct byte buffers handling incoming ipc requests
Key: HBASE-9535
URL: https://issues.apache.org/jira/browse/HBASE-9535
Project: HBase
Issue Type: Brainstorming
Reporter: stack
Assignee: Nicolas Liochon
Attachments: 9535.v1.patch, 9535.v2-trunk.patch, 9535.v2.patch,
9535.v3.patch
ipc takes in a query by allocating a ByteBuffer of the size of the request
and then reading off the socket into this on-heap BB.
Experiment with keeping a pool of BBs so we have some buffer reuse to cut on
garbage generated. Could checkout from pool in RpcServer#Reader. Could
check back into the pool when Handler is done just before it queues the
response on the Responder's queue. We should be good since, at least for
now, kvs get copied up into MSLAB (not references) when data gets stuffed
into MemStore; this should make it so no references left over when we check
the BB back into the pool for use next time around.
If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes
time so if already allocated, should be good. GC of DBBs is a pain but if in
a pool, we shouldn't be wanting this to happen). The copy from socket to the
DBB will be off-heap (should be fast).
Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by
size (we might want to bypass the pool if an object is size N).
DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when
short-circuit reading. It'd be a bummer if we had to allocate big
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835487#comment-13835487 ] Nicolas Liochon commented on HBASE-9535: My tests shows a ~4% improvement with the last version. I will need to check why we lost 3 points. the test with ycsb and and empty table, on the same machine, shows: trunk: 165K/s trunk with patch: 173K/s 0.96: 172K/s 0.96 with patch: 178K/s It's not a huge difference. In a real cluster, it should be even less important: ycsb sends huge messages (6 mb), and we have a 100% hit ratio on the pool. But it's a step in the good direction I would say. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: Nicolas Liochon Attachments: 9535.v1.patch, 9535.v2-trunk.patch, 9535.v2.patch, 9535.v3.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835535#comment-13835535 ] stack commented on HBASE-9535: -- You have diff of GC use? Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: Nicolas Liochon Attachments: 9535.v1.patch, 9535.v2-trunk.patch, 9535.v2.patch, 9535.v3.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834693#comment-13834693 ] Liang Xie commented on HBASE-9535: -- [~nkeywal] It would be great to add some mem used counters arond ElasticByteBufferPool, that will be useful while we encourntered a OOM Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch, 9535.v2.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834707#comment-13834707 ] Nicolas Liochon commented on HBASE-9535: bq. It would be great to add some mem used counters arond ElasticByteBufferPool, that will be useful while we encourntered a OOM Yeah, I agree. I'm actually wondering if I have to integrate it to the metrics from day one or if I can wait. Thanks for looking at the patch. It's still early stage, yes. If you can try it from a performance point of view, I'm interested. I would like to have a good enough (i.e. fully implemented with the right configuration parameters available) version for tomorrow. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch, 9535.v2.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[
https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835020#comment-13835020
]
Hadoop QA commented on HBASE-9535:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12616295/9535.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8026//console
This message is automatically generated.
Try a pool of direct byte buffers handling incoming ipc requests
Key: HBASE-9535
URL: https://issues.apache.org/jira/browse/HBASE-9535
Project: HBase
Issue Type: Brainstorming
Reporter: stack
Assignee: Nicolas Liochon
Attachments: 9535.v1.patch, 9535.v2.patch, 9535.v3.patch
ipc takes in a query by allocating a ByteBuffer of the size of the request
and then reading off the socket into this on-heap BB.
Experiment with keeping a pool of BBs so we have some buffer reuse to cut on
garbage generated. Could checkout from pool in RpcServer#Reader. Could
check back into the pool when Handler is done just before it queues the
response on the Responder's queue. We should be good since, at least for
now, kvs get copied up into MSLAB (not references) when data gets stuffed
into MemStore; this should make it so no references left over when we check
the BB back into the pool for use next time around.
If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes
time so if already allocated, should be good. GC of DBBs is a pain but if in
a pool, we shouldn't be wanting this to happen). The copy from socket to the
DBB will be off-heap (should be fast).
Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by
size (we might want to bypass the pool if an object is size N).
DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when
short-circuit reading. It'd be a bummer if we had to allocate big objects
on-heap. Would still be an improvement.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833622#comment-13833622 ] Nicolas Liochon commented on HBASE-9535: bq. do you have any gc perf numbers before and after the patch? I am interesting to know is there any scenarios existing could observe the gc improvement after pooling ipc memory. I will do a test... when it works ;-) I'm aiming more at the array copy here. But it could have a nice effect on the gc as well when the system is loaded; the query could get promoted to older generations (may be). bq. Do we need a BB of the size of the request? Would be better if we allocated fix sized small'ish (8-128k) BBs and read in chunks. I don't know actually. I'm trying to minimize the changes. I suppose I would have to create a specific InputStream to hide this for the prototobuf parser. The idea is nice. I don't know if we have lock issues, that would justify doing a single read to help to free the lock sooner. bq. In that case I would posit that we won't need a pool. Do you have some insight on this: there is a lot of it depends on the jvm and the hardware when it comes to the cost of doing a direct allocate. If we can make more assumptions on the allocation speed on jvm 1.7+ on modern hardware, it would help for sure. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[
https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833641#comment-13833641
]
Nicolas Liochon commented on HBASE-9535:
fwiw, it looked at the code called when you're doing a direct allocate, and I
saw, in java.nio.Bits
{code}
// These methods should be called whenever direct memory is allocated or
// freed. They allow the user to control the amount of direct memory
// which a process may access. All sizes are specified in bytes.
static void reserveMemory(long size) {
synchronized (Bits.class) { hum
if (!memoryLimitSet VM.isBooted()) {
maxMemory = VM.maxDirectMemory();
memoryLimitSet = true;
}
if (size = maxMemory - reservedMemory) {
reservedMemory += size;
return;
}
}
System.gc(); = ok...
try {
Thread.sleep(100); =
wow
} catch (InterruptedException x) {
// Restore interrupt status
Thread.currentThread().interrupt();
}
synchronized (Bits.class) {
if (reservedMemory + size maxMemory)
throw new OutOfMemoryError(Direct buffer memory);
reservedMemory += size;
}
}
{code}
Try a pool of direct byte buffers handling incoming ipc requests
Key: HBASE-9535
URL: https://issues.apache.org/jira/browse/HBASE-9535
Project: HBase
Issue Type: Brainstorming
Reporter: stack
Assignee: stack
Attachments: 9535.v1.patch
ipc takes in a query by allocating a ByteBuffer of the size of the request
and then reading off the socket into this on-heap BB.
Experiment with keeping a pool of BBs so we have some buffer reuse to cut on
garbage generated. Could checkout from pool in RpcServer#Reader. Could
check back into the pool when Handler is done just before it queues the
response on the Responder's queue. We should be good since, at least for
now, kvs get copied up into MSLAB (not references) when data gets stuffed
into MemStore; this should make it so no references left over when we check
the BB back into the pool for use next time around.
If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes
time so if already allocated, should be good. GC of DBBs is a pain but if in
a pool, we shouldn't be wanting this to happen). The copy from socket to the
DBB will be off-heap (should be fast).
Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by
size (we might want to bypass the pool if an object is size N).
DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when
short-circuit reading. It'd be a bummer if we had to allocate big objects
on-heap. Would still be an improvement.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834018#comment-13834018 ] Lars Hofhansl commented on HBASE-9535: -- Meh... :) DBBs are hard to get right. The JDK makes DBBs for BBs, when used in IO, under the hood, though. For each requested size it makes a new one and (in JDK6 at least) keeps 3 of them around. So it is quite important to use BBs of the same size only. You're right we should measure it. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834219#comment-13834219 ] Nicolas Liochon commented on HBASE-9535: v2 is still incomplete, but does work now. I think I have to use some guava stuff to use a fifo policy on the pool however. I've got a 7% improvement on this one, with my YCSB + empty table test. It's not bad, and there is another advantage: as we allocate the memory off heap, it does not interfere with the GC, so it's good when we have a server with a lot of memory that is so difficult to use in Java. It's not a full test, I need to look more at what the gc is doing. bq. So it is quite important to use BBs of the same size only. The hadoop pool has a very simple policy for this: it takes the first buffer big enough instead of at the same size. This seems smart. It could be even smarter to use the smallest big enough one. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch, 9535.v2.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834443#comment-13834443 ] Liang Xie commented on HBASE-9535: -- [~nkeywal], good to see there's an improvement ! I am planning to take a look at your patch to learn today:) Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch, 9535.v2.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832703#comment-13832703 ] Nicolas Liochon commented on HBASE-9535: fwiw, I'm trying some stuff around this currently. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13832920#comment-13832920 ] stack commented on HBASE-9535: -- bq. One of the difficult point is that we need to have a pool for the ByteBuffer ... yeah. And this pool will contend w/ the unbounded one we have in place doing SSR. bq. If we give it back at the end of the readAndProcess, the rpc doesn't workanymore, so someone must depend on it somewhere. What do you mean when you say rpc doesn't work anymore? Oh, you are saying it is being held on to past readAndProcess? Yeah, would be coolio to have a DBB per inbound-request. Good on you N. On patch, you pull in classes just for your test? Looks good so far Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833023#comment-13833023 ] Nicolas Liochon commented on HBASE-9535: bq. What do you mean when you say rpc doesn't work anymore? The server gets stuck somewhere. I don't really know where nor why. bq. On patch, you pull in classes just for your test? I plan to keep them. The pool is only in hadoop 2, so we need it for hadoop 1 compat. The ByteBufferInputStream is a technical class. It's public in ZK, but not really part of their API... And I've done some changes already. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833036#comment-13833036 ] stack commented on HBASE-9535: -- ok. We need to put any bounded DBB pool back in hdfs when we have a working one... later. Good on you N. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833407#comment-13833407 ] Liang Xie commented on HBASE-9535: -- [~nkeywal], do you have any gc perf numbers before and after the patch? I am interesting to know is there any scenarios existing could observe the gc improvement after pooling ipc memory. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13833526#comment-13833526 ] Lars Hofhansl commented on HBASE-9535: -- Do we need a BB of the size of the request? Would be better if we allocated fix sized small'ish (8-128k) BBs and read in chunks. In that case I would posit that we won't need a pool. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack Attachments: 9535.v1.patch ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771920#comment-13771920 ] Liang Xie commented on HBASE-9535: -- If i remembered correctly, in my test result, the avg ygct is 0.2s per one sec, means the avg each ygc costs 200ms/5 = 40ms. good on you, [~stack], hope you can get a better result:) Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[
https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770622#comment-13770622
]
Liang Xie commented on HBASE-9535:
--
to me, seems the RPC's ByteBuffer.allocate is not in the top contributor list
for gc.
I added logs below
{code}
data = ByteBuffer.allocate(dataLength);
{code}
in SecureServer.java (my test env is security enabled)
for a 100%read ycsb scenario, yes, this log shows the data lengths are similar
with each other(that means it's very easier to reuse/pool the byte buffers),
but after a raw estimation:
my config: Xmn512m Xmx4096m
my result: SecureServer.readAndProcess, dataLength:162;read throughput 7000ops;
jstat -gcutil shows about 5ygc per second
7000 * 162 is just 1MB more or less, is just a little portion of Xmn size, so
per my view, seems it doesn't very helpful to improve gc...
Try a pool of direct byte buffers handling incoming ipc requests
Key: HBASE-9535
URL: https://issues.apache.org/jira/browse/HBASE-9535
Project: HBase
Issue Type: Brainstorming
Reporter: stack
Assignee: stack
ipc takes in a query by allocating a ByteBuffer of the size of the request
and then reading off the socket into this on-heap BB.
Experiment with keeping a pool of BBs so we have some buffer reuse to cut on
garbage generated. Could checkout from pool in RpcServer#Reader. Could
check back into the pool when Handler is done just before it queues the
response on the Responder's queue. We should be good since, at least for
now, kvs get copied up into MSLAB (not references) when data gets stuffed
into MemStore; this should make it so no references left over when we check
the BB back into the pool for use next time around.
If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes
time so if already allocated, should be good. GC of DBBs is a pain but if in
a pool, we shouldn't be wanting this to happen). The copy from socket to the
DBB will be off-heap (should be fast).
Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by
size (we might want to bypass the pool if an object is size N).
DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when
short-circuit reading. It'd be a bummer if we had to allocate big objects
on-heap. Would still be an improvement.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771609#comment-13771609 ] stack commented on HBASE-9535: -- Thanks for doing a bit of math [~xieliang007] I'd say 162 bytes is unusually small for a request as is the ycsb workload where queries are all the same size. What if the requests were bigger, say 1-100k and we were doing 2-5k requests a second? Lets say 5k * 10k for easy calc. That'd be 50MB a second in the YG not counting any other creations. Some of these items could get promoted if there were pressure in YG. What if server was getting up to 1MB cells and we were doing 1k or more hits a second. The alternative is that these items would not go through the YG at all. How long did you run your test for? How long were the ygcs each second? I'd still be interested in trying this out since seems easy enough to do. Thanks boss. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768128#comment-13768128 ] Nicolas Liochon commented on HBASE-9535: bq. Maybe we should purge the Reader+queuing step and just have Handlers do the read (Nicolas Liochon what you think?) I've tried something similar (I removed the handlers and kept the readers), but the performance was not visible. The responder seemed to be a bottleneck. But it was not the only issue: we also want to manage priorities between the tasks, but we need to read them to get enough information to make the right priorities. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768215#comment-13768215 ] Liang Xie commented on HBASE-9535: -- i have some times these days, will work(test) it from tomorrow and give some perf number if available, [~stack]:) Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768485#comment-13768485 ] stack commented on HBASE-9535: -- [~liochon] It'd be simpler getting rid of Handler -- or rather have Handler do the read. Could park itself if priority was inappropriate at time of read? [~xieliang007] Excellent. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767875#comment-13767875 ] Vladimir Rodionov commented on HBASE-9535: -- ThreadLocals won't work? I mean, instead of pool we can keep BB in ThreadLocal variables. I suppose all incoming RPC requests are served already by a fixed pool of threads? Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767927#comment-13767927 ] stack commented on HBASE-9535: -- [~vrodionov] Thread local could work. There is a bit of a mismatch though since we read in one thread and the processing is done in another altogether (There are 10 Reader threads. We'd need the buffer available at read time. They read the request, put it on a queue from which the Handler threads pick it up. There are usually many more handlers than Readers (50 Handlers by default but could be hundreds). Maybe we should purge the Reader+queuing step and just have Handlers do the read ([~nkeywal] what you think?) Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767931#comment-13767931 ] Vladimir Rodionov commented on HBASE-9535: -- Nope, I was wrong ThreadLocals will work only if read/handle are in the same thread. Three different pools (Readers, Handlers and Senders) in HBase RPC seem like an old legacy. NIO, Netty? Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9535) Try a pool of direct byte buffers handling incoming ipc requests
[ https://issues.apache.org/jira/browse/HBASE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767968#comment-13767968 ] stack commented on HBASE-9535: -- [~vrodionov] Yeah, legacy. Yeah to trying Netty. Try a pool of direct byte buffers handling incoming ipc requests Key: HBASE-9535 URL: https://issues.apache.org/jira/browse/HBASE-9535 Project: HBase Issue Type: Brainstorming Reporter: stack Assignee: stack ipc takes in a query by allocating a ByteBuffer of the size of the request and then reading off the socket into this on-heap BB. Experiment with keeping a pool of BBs so we have some buffer reuse to cut on garbage generated. Could checkout from pool in RpcServer#Reader. Could check back into the pool when Handler is done just before it queues the response on the Responder's queue. We should be good since, at least for now, kvs get copied up into MSLAB (not references) when data gets stuffed into MemStore; this should make it so no references left over when we check the BB back into the pool for use next time around. If on-heap BBs work, we could then try direct BBs (Allocation of DBBs takes time so if already allocated, should be good. GC of DBBs is a pain but if in a pool, we shouldn't be wanting this to happen). The copy from socket to the DBB will be off-heap (should be fast). Could start w/ the HDFS DirectBufferPool. It is unbounded and keeps items by size (we might want to bypass the pool if an object is size N). DBBs for this task would contend w/ offheap BBs used in BlockReadLocal when short-circuit reading. It'd be a bummer if we had to allocate big objects on-heap. Would still be an improvement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
