[
https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113492#comment-16113492
]
Chen Liang commented on HDFS-12196:
-----------------------------------
Thanks [~cheersyang] for the reply and the updated patch!
I'm still concerned about the result caching though...I agree with you that it
can save RPC calls, but the thing is it can be very tricky to get the caching
work properly...more specifically:
1. One potential issue is that it seems there is no purging of the cache. i.e.
once an entry gets added to the {{resultList}}, it never gets removed. So the
size of {{resultList}} will monolithically increase to the point where no more
entries can be added and then the result cache will no longer be helpful for
any further entry. Also since all entries are just being added when there is
space, if there is no more query for those cached result, without purging, we
may end up just holding a bunch of objects in memory that will never be useful.
I think ideally the cache should probably be removing entries based on factors
such as access time, frequency etc. In short, the cache needs to have purging.
2. Seems there is no easy way to check the current {{resultList}} efficiently.
Any service using it will have to check the entire cache. Namely, since it is a
list, so even if some service wants to check if the result of a call is cached
in it before making the call, it still needs to iterate over all the entries in
it, so if the list is large enough, it could be even less efficient than
contacting datanodes. In short, the cache needs to have fast lookup. Another
side thing is that the entire {{resultList}} object is exposed and returned to
caller, a bug in caller can easily mess up the list...
I'm not against adding cache here, just saying we should probably pay a little
attention. If we really want cache here, it shouldn't be hard to implement a
cache on top of either
[LRUMap|https://commons.apache.org/proper/commons-collections/apidocs/org/apache/commons/collections4/map/LRUMap.html]
OR [Guava loading cache|https://github.com/google/guava/wiki/CachesExplained],
which should make purging and fast lookup very easy, even trivial. (One example
is the cache in {{XceiverClientManager}}, just FYI).
> Ozone: DeleteKey-2: Implement container recycling service to delete stale
> blocks at background
> ----------------------------------------------------------------------------------------------
>
> Key: HDFS-12196
> URL: https://issues.apache.org/jira/browse/HDFS-12196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
> Reporter: Weiwei Yang
> Assignee: Weiwei Yang
> Attachments: HDFS-12196-HDFS-7240.001.patch,
> HDFS-12196-HDFS-7240.002.patch
>
>
> Implement a recycling service running on datanode to delete stale blocks.
> The recycling service scans staled blocks for each container and delete
> chunks and references periodically.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
