Srinivasu Majeti created HDFS-17323:
---------------------------------------
Summary: Uncontrolled fsimage size due to snapshot diff meta for
file deletions
Key: HDFS-17323
URL: https://issues.apache.org/jira/browse/HDFS-17323
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Affects Versions: 3.1.1
Reporter: Srinivasu Majeti
We have seen quite a good number of customer cases w.r.t fsimage size increased
drastically while storing snapshot meta for fileDiff entries. Here is an
example fsimage meta storing entire inode info after deleting a file. I'm not
sure about any restrictions on why the entire inode meta needs to be stored in
fileDiff entry when there is no change w.r.t actual inode meta and it's just a
delete file operation.
The fileDiffEntry for the inode 1860467 seems redundant for a simple file
delete operation.
{code:java}
<snapshot><id>43</id><root><id>1860465</id><type>DIRECTORY</type><name>s3</name><mtime>1704197935903</mtime><permission>hdfs:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></root></snapshot>
<snapshot><id>44</id><root><id>1860465</id><type>DIRECTORY</type><name>s4</name><mtime>1704197951829</mtime><permission>hdfs:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></root></snapshot>
<inode><id>1860467</id><type>FILE</type><name>file1</name><replication>3</replication><mtime>1704197917315</mtime><atime>1704197917031</atime><preferredBlockSize>134217728</preferredBlockSize><permission>hdfs:supergroup:0644</permission><blocks><block><id>1074008442</id><genstamp>267653</genstamp><numBytes>418</numBytes></block>
<ref><referredId>1860467</referredId><name>file1</name><dstSnapshotId>0</dstSnapshotId><lastSnapshotId>43</lastSnapshotId></ref>
<ref><referredId>1860467</referredId><name></name><dstSnapshotId>2147483646</dstSnapshotId><lastSnapshotId>0</lastSnapshotId></ref>
</dirDiffEntry><fileDiffEntry><inodeId>1860467</inodeId><count>1</count><fileDiff><snapshotId>43</snapshotId><size>418</size><name>file1</name><snapshotCopy><replication>3</replication><mtime>1704197917315</mtime><atime>1704197917031</atime><preferredBlockSize>134217728</preferredBlockSize><permission>hdfs:supergroup:0644</permission><storagePolicyId>0</storagePolicyId></snapshotCopy>
</fileDiff>
</fileDiffEntry>{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
