[
https://issues.apache.org/jira/browse/HBASE-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Gray updated HBASE-1460:
---------------------------------
Attachment: HBASE-1460-v3.patch
Addresses all issues from last comment, stack.
Renamed things to be proper. Replaces LruBlockCache instead of adding a new
class. Added more javadocs, integrated with metrics, etc.
Regarding the timing issue in the tests, it will take a bit of a rework so that
the eviction thread "reports" back when it's done or some such thing. Rather
complex. I've changed things around a bit so that I can disable the thread and
run them from the context of the test rather than in the background thread. So
the tests that are testing the evictions themselves, heap size, etc.. will not
be tied to any timing issues. However, the issue there is we aren't testing
the operation of the background thread anymore. So I added a single test that
just checks if the background thread works (it contains the same kind of timing
issue). However, in this case I really don't think we'll have a problem on
hudson, the eviction is very fast and is executed directly.
I also added a scheduled thread inside that outputs statistics on a fixed
period. It's really just in there for now while I do some heavy debugging, can
be taken out before commit (or left in if people think it's useful, probably a
better way to do it though).
This breaks erik's new TestHeapSize. Need to talk to him tomorrow about how we
get that working. The cache heap size that it reports includes ALL overhead
not actually part of the blocks (the pre-allocated Map, it's entries, etc).
TestLruBlockCache does a bunch of validation of heapSize() reporting, but is
more to ensure it's tracking block sizes rather than verifying our static
overheads.
Not ready for commit. I want to add a few more tests and do some load testing.
Will report more tomorrow.
> Concurrent LRU Block Cache
> --------------------------
>
> Key: HBASE-1460
> URL: https://issues.apache.org/jira/browse/HBASE-1460
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: io
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Fix For: 0.20.0
>
> Attachments: HBASE-1460-v1.patch, HBASE-1460-v2.patch,
> HBASE-1460-v3.patch
>
>
> The LRU-based block cache that will be committed in HBASE-1192 is thread-safe
> but contains a big lock on the hash map. Under high load, the block cache
> will be hit very heavily from a number of threads, so it needs to be built to
> handle massive concurrency.
> This issue aims to implement a new block cache with LRU eviction, but backed
> by a ConcurrentHashMap and a separate eviction thread. Influence will be
> drawn from Solr's ConcurrentLRUCache, however there are major differences
> because solr treats all cached elements as equal size whereas we are
> dependent on our HeapSize interface with realistic (though approximate) heap
> usage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
