[
https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157633#comment-16157633
]
Kai Zheng commented on HDFS-7859:
---------------------------------
bq. Checking file / directory that is using this particular policy is a
potentially O(n) operation, where n = # of inodes. I feel that it is OK to
leave it in fsimage as garbage for now. In the future, we can let the fsimage
loading process to handling this garbage, as it is O(n).
Discussed with Sammi offline before, we can do this very lightly like below:
{code}
# all used polices by files/directories
usedPoliciesSet = ();
# while loading inodes from fsimage, add the following two lines
foreach (in: inodes) {
policyId = getPolicyIdFromInode(in) # a bitwise op, very minor
usedPoliciesSet.add(policyId)
}
# when inodes all loaded, add the following post step
ErasureCodingPolicyManager.getInstance().updateWithUsedPolices(usedPoliciesSet)
# in ErasureCodingPolicyManager.updateWithUsedPolices, it's a simple step to
clean up removed policies with the used polices set.
{code}
bq. Here or elsewhere, please ensure no policy can be DISABLED/REMOVED if it's
used by files, with necessary tests.
Let me correct myself. We should allow to disable/remove polices regardless
they're used or not. It would be too much overhead to track policy usages while
NN is running along with lots of files being operated. We can just do a post
clean up as above illustrated.
I'm fine to leave the policies clean up work as a future work to do, but if
sounds good maybe we can get it done before 3.0 GA. It should be OK since it
doesn't involve API change.
bq. Could this happen before BETA 1? it seems to be a breaking change. If not ,
do we have a plan to preserve both this key and the capability of
adding/removing policies?
I agree, we should get it done this time. Actually, IIRC, this was already done
but Sammi may need some double check and clean up if any.
> Erasure Coding: Persist erasure coding policies in NameNode
> -----------------------------------------------------------
>
> Key: HDFS-7859
> URL: https://issues.apache.org/jira/browse/HDFS-7859
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Kai Zheng
> Assignee: SammiChen
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-7859.001.patch, HDFS-7859.002.patch,
> HDFS-7859.004.patch, HDFS-7859.005.patch, HDFS-7859.006.patch,
> HDFS-7859.007.patch, HDFS-7859.008.patch, HDFS-7859.009.patch,
> HDFS-7859.010.patch, HDFS-7859.011.patch, HDFS-7859.012.patch,
> HDFS-7859.013.patch, HDFS-7859.014.patch, HDFS-7859.015.patch,
> HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.002.patch,
> HDFS-7859-HDFS-7285.003.patch
>
>
> In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we
> persist EC schemas in NameNode centrally and reliably, so that EC zones can
> reference them by name efficiently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
