[
https://issues.apache.org/jira/browse/HBASE-6312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405973#comment-13405973
]
Zhihong Ted Yu commented on HBASE-6312:
---------------------------------------
Pasting from a message on dev@:
Ideally those shouldn't be configurable, we should just set to a level
that makes more sense. If we do make it configurable and let it like
that then we'll have questions like "what acceptable/min factor should
I use?" and we'll spend hours doing back and forth on the ML to get
minimal results.
Currently having the acceptable factor set to where it is just means
that we're using less memory than configured eg if you need to cache
2GB per machine, set hfile.block.cache.size to ~2.35GB and you'll have
it.
The real issue is the minimum factor. The idea is that we don't want
to overflow the configured maximum size while we're evicting. The
problems:
- Evicting 10% of the cache (85-75) is pretty hardcore, it means that
if you evict often then you're never close to using 85% of your cache.
- Evictions are purely CPU-bound and in my tests are almost never
likely to be so slow that you reach 100% utilization (whereas loading
the cache usually means you need to read from disk). It was too slow
for caches of up to 32MB with data generated in-memory.
- Considering the previous two problems it seems we should set the
minimum factor close to the acceptable one, but on big caches this
would waste a lot of CPU cycles (I haven't quantified that yet, I'm
just stating this from experience).
So back to HBASE-6312, at the moment I think we should just set the
minimum factor 5% closer to the acceptable one. Jie Huang doesn't
mention if in their tests their customers compared the caching ratio
for caches of the same size but with different acceptable factor or if
they tried to compare apples to apples. What I'm trying to say, going
back to my earlier example, is that if they compared two caches with
hfile.block.cache.size=0.2 but with different acceptable factors then
well yes the one with the bigger acceptable factor will win... because
it's using a bigger cache.
J-D
> Make BlockCache eviction thresholds configurable
> ------------------------------------------------
>
> Key: HBASE-6312
> URL: https://issues.apache.org/jira/browse/HBASE-6312
> Project: HBase
> Issue Type: Improvement
> Components: io
> Affects Versions: 0.94.0
> Reporter: Jie Huang
> Priority: Minor
> Attachments: hbase-6312.patch
>
>
> Some of our customers found that tuning the BlockCache eviction thresholds
> made test results different in their test environment. However, those
> thresholds are not configurable in the current implementation. The only way
> to change those values is to re-compile the HBase source code. We wonder if
> it is possible to make them configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
