[
https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Mackrory updated HDFS-10797:
---------------------------------
Attachment: HDFS-10797.005.patch
Now attaching a patch that should give what I think we agree are the completely
correct semantics:
- du -s in a given directory will yield the space consumed by all snapshots and
current files in the relevant directory, or below, regardless of subsequent
renames or deletions.
- du -s in a parent directory may yield a smaller value than the sum of all du
-s results in child directories, if files have been snapshotted in one child
directory and then move to another. There is overlap in the space consumed by
each directory in this case.
- In no cases is any INode counted twice in the context of a single du -s
computation.
In my opinion, these semantics are the most correct and least surprising to
users, and they are consistent. If I understand your first reply correctly, I
*think* you would agree with this, [~jingzhao]? And the implementation resolves
the inconsistency we were trying to avoid. Let me know if I've misunderstood
anything...
Attaching the patch, but going to do a little more manual testing, tinkering
and thinking about this before "submitting" again since this is a very
different approach from my previous patches.
> Disk usage summary of snapshots causes renamed blocks to get counted twice
> --------------------------------------------------------------------------
>
> Key: HDFS-10797
> URL: https://issues.apache.org/jira/browse/HDFS-10797
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Sean Mackrory
> Assignee: Sean Mackrory
> Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch,
> HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch
>
>
> DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how
> much disk usage is used by a snapshot by tallying up the files in the
> snapshot that have since been deleted (that way it won't overlap with regular
> files whose disk usage is computed separately). However that is determined
> from a diff that shows moved (to Trash or otherwise) or renamed files as a
> deletion and a creation operation that may overlap with the list of blocks.
> Only the deletion operation is taken into consideration, and this causes
> those blocks to get represented twice in the disk usage tallying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
