[
https://issues.apache.org/jira/browse/HBASE-14613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14613:
--------------------------
Attachment: writes.png
gc.png
Here are some diagrams. 25 clients YCSB writing a single regionserver. I ran
four test runs. The first and last were WITHOUT memstorechunkpool. The second
and third were WITH memstorechunkpool enabled. I set it to be .25 the size of
memstore and I set initial size to 0.
The diagrams are GC and write rates (the fourth write run is missing from the
diagram...)
So, high-level, write rates are same with or without patch. No surprise.
The CMS GCs are less when the pool is in place. No surprise given we are not
recreating chunks but reusing them.
What is surprising is that the new gen seems to be doing more collections. I'd
have to dig in why.... (Guess would be that this pool is throwing off the
ergonomic sizing of new vs old... would have to check).
The pool prints stats and the stats look nice:
{code}
2015-10-21 11:04:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=9,created chunk
count=39,reused chunk count=3170,reuseRatio=98.78%
2015-10-21 11:09:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=8,created chunk
count=39,reused chunk count=3322,reuseRatio=98.84%
2015-10-21 11:14:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=14,created chunk
count=39,reused chunk count=3584,reuseRatio=98.92%
2015-10-21 11:19:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=30,created chunk
count=39,reused chunk count=3829,reuseRatio=98.99%
2015-10-21 11:24:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=37,created chunk
count=39,reused chunk count=4092,reuseRatio=99.06%
2015-10-21 11:29:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=12,created chunk
count=39,reused chunk count=4352,reuseRatio=99.11%
2015-10-21 11:34:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=14,created chunk
count=39,reused chunk count=4609,reuseRatio=99.16%
2015-10-21 11:39:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=20,created chunk
count=39,reused chunk count=4865,reuseRatio=99.20%
2015-10-21 11:44:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=17,created chunk
count=39,reused chunk count=5128,reuseRatio=99.25%
2015-10-21 11:49:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=20,created chunk
count=39,reused chunk count=5377,reuseRatio=99.28%
2015-10-21 11:54:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=17,created chunk
count=39,reused chunk count=5630,reuseRatio=99.31%
2015-10-21 11:59:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=23,created chunk
count=39,reused chunk count=5871,reuseRatio=99.34%
2015-10-21 12:04:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=22,created chunk
count=39,reused chunk count=6122,reuseRatio=99.37%
2015-10-21 12:09:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=36,created chunk
count=39,reused chunk count=6216,reuseRatio=99.38%
2015-10-21 12:14:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=36,created chunk
count=39,reused chunk count=6216,reuseRatio=99.38%
2015-10-21 12:19:07,026 DEBUG
[StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1-MemStoreChunkPool Statistics]
regionserver.MemStoreChunkPool: Stats: current pool size=36,created chunk
count=39,reused chunk count=6216,reuseRatio=99.38%
{code}
The reuse is impressive.
So, the pool could be good. I'd change it so the stats ran out of an executor
rather than as a dedicated thread as a Chore. We'd have to figure the sizing
too. The initial sizing can be zero. Thats fine. Let it ramp up even if it
means chunks have to traverse survivor space a few times.... (would be sweet if
could allocated direct on old gen but that seems hard to do reliably -- could
spend some more time digging in on this). What for the max size? Currently
it is a proportion of memstore size. Could study running machine to see how
many chunks are outstanding when a write load going on. Could then set the
pool to be this size. But, we need to have a pool that can grow and shrink.
There is the ergonomic resize of the memstore that will resize memstore if we
are doing many more writes than reads. The pool should adjust accordingly. The
pool should also adjust if it has not been accessed in a long time. We do not
want the case where we have all memstores holding on to big pools because we
just underwent a furious write load but now it has abated...
> Remove MemStoreChunkPool?
> -------------------------
>
> Key: HBASE-14613
> URL: https://issues.apache.org/jira/browse/HBASE-14613
> Project: HBase
> Issue Type: Brainstorming
> Reporter: Lars Hofhansl
> Priority: Minor
> Attachments: 14613-0.98.txt, gc.png, writes.png
>
>
> I just stumbled across MemStoreChunkPool. The idea behind is to reuse chunks
> of allocations rather than letting the GC handle this.
> Now, it's off by default, and it seems to me to be of dubious value. I'd
> recommend just removing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
