jojochuang commented on code in PR #10119:
URL: https://github.com/apache/ozone/pull/10119#discussion_r3149960845


##########
hadoop-hdds/docs/content/feature/SnapshotDefragmentation.md:
##########
@@ -42,109 +42,163 @@ Note: Snapshot Defragmentation was previously called 
Snapshot Compaction earlier
 
 1. ### Introducing last defragmentation time
 
-   A new boolean flag (`needsDefrag`), timestamp (`lastDefragTime`), int 
`version` would be added to snapshot metadata.
-   `needsDefrag` tells the system whether a snapshot is pending defrag 
(`true`) or if it is already defragged and up to date (`false`). This helps 
manage and automate the defrag workflow, ensuring snapshots are efficiently 
stored and maintained. `needsDefrag` defaults to `false` during initialization 
and when absent.
-   A new list of Map\<String, List\<Longs\>\> (`notDefraggedSstFileList`) also 
would be added to snapshot meta as part of snapshot create operation; this 
would be storing the original list of SST files in the not defragged copy of 
the snapshot corresponding to keyTable/fileTable/DirectoryTable. This should be 
done as part of the snapshot create operation.
-   Since this is not going to be consistent across all OMs this would have to 
be written to a local yaml file inside the snapshot directory and this can be 
maintained in the SnapshotChainManager in memory on startup. So all updates 
should not go through Ratis.  
-   An additional Map\<Integer, Map\<String, List\<Long\>\>\> 
(`defraggedSstFileList`) also would be added to snapshotMeta. This will be 
maintaining a list of sstFiles of different versions of defragged snapshots. 
The key here would be the version number of snapshot DBs.
+   The implemented metadata is stored in local `OmSnapshotLocalData` YAML 
files,
+   not in Ratis-replicated `SnapshotInfo`. The YAML is created on every 
snapshot
+   creation even when the periodic defrag service is disabled.
+
+   The important YAML fields are `snapshotId`, `previousSnapshotId`, `version`,
+   `needsDefrag`, `versionSstFileInfos`, `dbTxSequenceNumber`,
+   `transactionInfo`, `lastDefragTime`, `checksum`, and `isSSTFiltered`.
+   `lastDefragTime` is serialized, but current defrag decisions are based on

Review Comment:
   maybe somewhere in the code we want to mark `lastDefragTime` deprecated. 



##########
hadoop-hdds/docs/content/feature/SnapshotDefragmentation.md:
##########
@@ -42,109 +42,163 @@ Note: Snapshot Defragmentation was previously called 
Snapshot Compaction earlier
 
 1. ### Introducing last defragmentation time
 
-   A new boolean flag (`needsDefrag`), timestamp (`lastDefragTime`), int 
`version` would be added to snapshot metadata.
-   `needsDefrag` tells the system whether a snapshot is pending defrag 
(`true`) or if it is already defragged and up to date (`false`). This helps 
manage and automate the defrag workflow, ensuring snapshots are efficiently 
stored and maintained. `needsDefrag` defaults to `false` during initialization 
and when absent.
-   A new list of Map\<String, List\<Longs\>\> (`notDefraggedSstFileList`) also 
would be added to snapshot meta as part of snapshot create operation; this 
would be storing the original list of SST files in the not defragged copy of 
the snapshot corresponding to keyTable/fileTable/DirectoryTable. This should be 
done as part of the snapshot create operation.
-   Since this is not going to be consistent across all OMs this would have to 
be written to a local yaml file inside the snapshot directory and this can be 
maintained in the SnapshotChainManager in memory on startup. So all updates 
should not go through Ratis.  
-   An additional Map\<Integer, Map\<String, List\<Long\>\>\> 
(`defraggedSstFileList`) also would be added to snapshotMeta. This will be 
maintaining a list of sstFiles of different versions of defragged snapshots. 
The key here would be the version number of snapshot DBs.
+   The implemented metadata is stored in local `OmSnapshotLocalData` YAML 
files,
+   not in Ratis-replicated `SnapshotInfo`. The YAML is created on every 
snapshot
+   creation even when the periodic defrag service is disabled.
+
+   The important YAML fields are `snapshotId`, `previousSnapshotId`, `version`,
+   `needsDefrag`, `versionSstFileInfos`, `dbTxSequenceNumber`,
+   `transactionInfo`, `lastDefragTime`, `checksum`, and `isSSTFiltered`.
+   `lastDefragTime` is serialized, but current defrag decisions are based on
+   `version`, `needsDefrag`, and `versionSstFileInfos`.
+
+   The earlier proposal's `notDefraggedSstFileList` and `defraggedSstFileList`
+   became `versionSstFileInfos`, a map from local snapshot version to
+   `VersionMeta`. Each `VersionMeta` records the previous snapshot version it
+   depends on and the tracked SST files for `keyTable`, `fileTable`, and
+   `directoryTable`.
 
 2. ### Snapshot Cache Lock for Read Prevention
 
-   A snapshot lock will be introduced in the snapshot cache to prevent reads 
on a specific snapshot during the last step of defragmentation. This ensures no 
active reads occur while we are replacing the underlying RocksDB instance. The 
swap should be instantaneous.
+   The implemented lock path uses `SNAPSHOT_DB_CONTENT_LOCK` to prevent
+   concurrent snapshot content changes while defrag reloads non-incremental
+   tables and switches the current version. Older checkpoint directory deletion
+   also acquires the snapshot DB cache write lock so stale cached handles 
cannot
+   write to an old version while it is being removed.
 
 3. ### Directory Structure Changes
 
-   Snapshots currently reside under `db.snapshots/checkpointState/` directory. 
The proposal introduces a `db.snapshots/checkpointStateDefragged/` directory 
for defragged snapshots. The directory format should be as follows:
+   Snapshots reside under the OM metadata directory in
+   `db.snapshots/checkpointState/`. The implementation does not use a separate
+   `checkpointStateDefragged` directory. Defragged versions are sibling

Review Comment:
   btw, this is no longer used checkpointStateDefragged / 
OM_SNAPSHOT_CHECKPOINT_DEFRAGGED_DIR
   consider removing it in the future.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to