[jira] [Commented] (HBASE-10374) add jemalloc into the choice of memstore allocation
[ https://issues.apache.org/jira/browse/HBASE-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877326#comment-13877326 ] stack commented on HBASE-10374: --- [~jianbginglover] You suggest jemalloc for memstore block allocations. What are you thinking we'll see by way of benefit? I'd think that isolating the MemStore and then in a simple testing harness trying various options would be the way to go (MemStore can be stood up outside of a HStore IIRC). Try netty implementation first since code is done. See if you can get any speedup in your testing rig. If improvement, then lets talk. We'll have to see about what [~ndimiduk] reminds us of, that netty implemenation is ByteBuf-based (as opposed to ByteBuffer). On pulling in netty4, it might not be too bad since they changed the package from org.jboss to io.netty. > add jemalloc into the choice of memstore allocation > > > Key: HBASE-10374 > URL: https://issues.apache.org/jira/browse/HBASE-10374 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.1.1 >Reporter: Bing Jiang >Priority: Minor > > https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead > introduced that Netty used jemalloc to gain benefits from GC. > It can be a good choice for memstore block allocation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10374) add jemalloc into the choice of memstore allocation
[ https://issues.apache.org/jira/browse/HBASE-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875587#comment-13875587 ] Bing Jiang commented on HBASE-10374: [~stack] I have a question, if we try to use the feature which Netty provides, I believe there are two choices, the one is to add dependence upon Netty, the other one is to implements the jemalloc by the help of their codes. I want to take your suggestion. > add jemalloc into the choice of memstore allocation > > > Key: HBASE-10374 > URL: https://issues.apache.org/jira/browse/HBASE-10374 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.1.1 >Reporter: Bing Jiang >Priority: Minor > > https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead > introduced that Netty used jemalloc to gain benefits from GC. > It can be a good choice for memstore block allocation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10374) add jemalloc into the choice of memstore allocation
[ https://issues.apache.org/jira/browse/HBASE-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875319#comment-13875319 ] Andrew Purtell commented on HBASE-10374: This is also directly related to HBASE-10191 > add jemalloc into the choice of memstore allocation > > > Key: HBASE-10374 > URL: https://issues.apache.org/jira/browse/HBASE-10374 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.1.1 >Reporter: Bing Jiang >Priority: Minor > > https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead > introduced that Netty used jemalloc to gain benefits from GC. > It can be a good choice for memstore block allocation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10374) add jemalloc into the choice of memstore allocation
[ https://issues.apache.org/jira/browse/HBASE-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875265#comment-13875265 ] Nick Dimiduk commented on HBASE-10374: -- Premature submit. As I was a saying. I don't have conclusive reasons to pursue Netty ByteBufs vs nio ByteBuffers. The compositional structure Netty provides is really nice, if we can use it, but I think it could be implemented on top of stock ByteBuffers as well. > add jemalloc into the choice of memstore allocation > > > Key: HBASE-10374 > URL: https://issues.apache.org/jira/browse/HBASE-10374 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.1.1 >Reporter: Bing Jiang >Priority: Minor > > https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead > introduced that Netty used jemalloc to gain benefits from GC. > It can be a good choice for memstore block allocation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10374) add jemalloc into the choice of memstore allocation
[ https://issues.apache.org/jira/browse/HBASE-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875262#comment-13875262 ] Nick Dimiduk commented on HBASE-10374: -- Yeah, I tried this once and it didn't go very well. I am keen to pursue Stack's suggestion though. I haven't come to a conclusive decision for why we'd pursue Netty's ByteBuf vs nio > add jemalloc into the choice of memstore allocation > > > Key: HBASE-10374 > URL: https://issues.apache.org/jira/browse/HBASE-10374 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.1.1 >Reporter: Bing Jiang >Priority: Minor > > https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead > introduced that Netty used jemalloc to gain benefits from GC. > It can be a good choice for memstore block allocation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10374) add jemalloc into the choice of memstore allocation
[ https://issues.apache.org/jira/browse/HBASE-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875181#comment-13875181 ] Andrew Purtell commented on HBASE-10374: Someone should try pulling in netty4 while excluding earlier versions to see what happens. I bet it will be a mess :-) given our dependencies on Jetty 6 and ZooKeeper. > add jemalloc into the choice of memstore allocation > > > Key: HBASE-10374 > URL: https://issues.apache.org/jira/browse/HBASE-10374 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.1.1 >Reporter: Bing Jiang >Priority: Minor > > https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead > introduced that Netty used jemalloc to gain benefits from GC. > It can be a good choice for memstore block allocation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10374) add jemalloc into the choice of memstore allocation
[ https://issues.apache.org/jira/browse/HBASE-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875179#comment-13875179 ] Andrew Purtell commented on HBASE-10374: I found it here: http://netty.io/4.0/api/io/netty/buffer/ByteBufAllocator.html > add jemalloc into the choice of memstore allocation > > > Key: HBASE-10374 > URL: https://issues.apache.org/jira/browse/HBASE-10374 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.1.1 >Reporter: Bing Jiang >Priority: Minor > > https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead > introduced that Netty used jemalloc to gain benefits from GC. > It can be a good choice for memstore block allocation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HBASE-10374) add jemalloc into the choice of memstore allocation
[ https://issues.apache.org/jira/browse/HBASE-10374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875022#comment-13875022 ] stack commented on HBASE-10374: --- [~jianbginglover] What you thinking? We pull in netty4, redo our core on top of it (smile), then as a side benefit, use their jemalloc implementation for block cache? Poking around, I don't see it here http://netty.io/4.0/api/index.html Thanks. > add jemalloc into the choice of memstore allocation > > > Key: HBASE-10374 > URL: https://issues.apache.org/jira/browse/HBASE-10374 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.96.1.1 >Reporter: Bing Jiang >Priority: Minor > > https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead > introduced that Netty used jemalloc to gain benefits from GC. > It can be a good choice for memstore block allocation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
