Srinivasu Majeti created HDFS-17323: ---------------------------------------
Summary: Uncontrolled fsimage size due to snapshot diff meta for file deletions Key: HDFS-17323 URL: https://issues.apache.org/jira/browse/HDFS-17323 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Affects Versions: 3.1.1 Reporter: Srinivasu Majeti We have seen quite a good number of customer cases w.r.t fsimage size increased drastically while storing snapshot meta for fileDiff entries. Here is an example fsimage meta storing entire inode info after deleting a file. I'm not sure about any restrictions on why the entire inode meta needs to be stored in fileDiff entry when there is no change w.r.t actual inode meta and it's just a delete file operation. The fileDiffEntry for the inode 1860467 seems redundant for a simple file delete operation. {code:java} <snapshot><id>43</id><root><id>1860465</id><type>DIRECTORY</type><name>s3</name><mtime>1704197935903</mtime><permission>hdfs:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></root></snapshot> <snapshot><id>44</id><root><id>1860465</id><type>DIRECTORY</type><name>s4</name><mtime>1704197951829</mtime><permission>hdfs:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></root></snapshot> <inode><id>1860467</id><type>FILE</type><name>file1</name><replication>3</replication><mtime>1704197917315</mtime><atime>1704197917031</atime><preferredBlockSize>134217728</preferredBlockSize><permission>hdfs:supergroup:0644</permission><blocks><block><id>1074008442</id><genstamp>267653</genstamp><numBytes>418</numBytes></block> <ref><referredId>1860467</referredId><name>file1</name><dstSnapshotId>0</dstSnapshotId><lastSnapshotId>43</lastSnapshotId></ref> <ref><referredId>1860467</referredId><name></name><dstSnapshotId>2147483646</dstSnapshotId><lastSnapshotId>0</lastSnapshotId></ref> </dirDiffEntry><fileDiffEntry><inodeId>1860467</inodeId><count>1</count><fileDiff><snapshotId>43</snapshotId><size>418</size><name>file1</name><snapshotCopy><replication>3</replication><mtime>1704197917315</mtime><atime>1704197917031</atime><preferredBlockSize>134217728</preferredBlockSize><permission>hdfs:supergroup:0644</permission><storagePolicyId>0</storagePolicyId></snapshotCopy> </fileDiff> </fileDiffEntry>{code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org