[ https://issues.apache.org/jira/browse/HBASE-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011452#comment-13011452 ]
Jonathan Gray commented on HBASE-3694: -------------------------------------- Do we really want to put things like this into RegionServerMetrics? That class is a mess and is currently only used for the publishing of our metrics (not used for internal state tracking). And we should avoid the hadoop Metrics* classes like the plague... heavily synchronized and generally confusing. My vote would be to add a new class, maybe {{RegionServerHeapManager}} or something like that... might be a good opportunity to cleanup and centralize the code related to that. But could just hold this one AtomicLong for now. Agree that adding a new interface method just for the long is not ideal since it buys us nothing down the road. Better to add something new that we can use later. > high multiput latency due to checking global mem store size in a synchronized > function > -------------------------------------------------------------------------------------- > > Key: HBASE-3694 > URL: https://issues.apache.org/jira/browse/HBASE-3694 > Project: HBase > Issue Type: Improvement > Reporter: Liyin Tang > Assignee: Liyin Tang > Attachments: Hbase-3694[r1085306], Hbase-3694[r1085306]_2.patch, > Hbase-3694[r1085306]_3.patch, Hbase-3694[r1085508]_4.patch > > > The problem is we found the multiput latency is very high. > In our case, we have almost 22 Regions in each RS and there are no flush > happened during these puts. > After investigation, we believe that the root cause is the function > getGlobalMemStoreSize, which is to check the high water mark of mem store. > This function takes almost 40% of total execution time of multiput when > instrumenting some metrics in the code. > The actual percentage may be more higher. The execution time is spent on > synchronize contention. > One solution is to keep a static var in HRegion to keep the global MemStore > size instead of calculating them every time. > Why using static variable? > Since all the HRegion objects in the same JVM share the same memory heap, > they need to share fate as well. > The static variable, globalMemStroeSize, naturally shows the total mem usage > in this shared memory heap for this JVM. > If multiple RS need to run in the same JVM, they still need only one > globalMemStroeSize. > If multiple RS run on different JVMs, everything is fine. > After changing, in our cases, the avg multiput latency decrease from 60ms to > 10ms. > I will submit a patch based on the current trunk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira