[
https://issues.apache.org/jira/browse/HDDS-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-15027:
-----------------------------------
Description:
A very common question users ask is: when a user issues a delete request, how
does it eventually get deleted? Due to Ozone's distributed nature, it can be
tricky to understand all the moving parts, so this page gives a holistic
overview of delete.
---
Client side:
✦ Client-side deletion in Ozone depends on the interface and bucket layout,
with trash operations implemented as a rename to .Trash/Current by both the
Ozone CLI and FileSystem.
1. Ozone CLI Deletion (DeleteKeyHandler)
* FSO Buckets:
* If Trash is enabled (fs.trash.interval > 0), the CLI renames the key
to .Trash/<user>/Current/<keyName>.
* If Trash is disabled, it calls bucket.deleteKey(keyName) for direct
deletion.
* OBS/Legacy Buckets:
* Directly calls bucket.deleteKey(keyName), which triggers a logical
delete in OM. Trash is not natively supported for non-FSO buckets via the ozone
sh key delete command.
2. Ozone FileSystem Deletion (OzoneFileSystem)
The FileSystem interface (o3fs:// or ofs://) typically uses the standard
Hadoop Trash Policy:
* When you run hadoop fs -rm <path>, the Hadoop client code checks the
configuration.
* If trash is enabled, the client performs a Rename operation:
* Source: /path/to/file
* Destination: /.Trash/Current/path/to/file
* If -skipTrash is used or trash is disabled, it calls FileSystem.delete(),
which maps to a direct deleteKey RPC to the Ozone Manager.
3. OM Logic for Deletion vs. Trash
* Rename (Trash): To the OM, moving a file to trash is just a metadata
update in the keyTable or fileTable. The object ID remains the same, and no
blocks
are reclaimed.
* Direct Delete: Triggered by bucket.deleteKey(). As previously explained,
OM moves the metadata to the deletedTable, starting the asynchronous block
reclamation process.
4. Trash Emptier
Because trash is just a normal directory (.Trash), it is emptied by a Trash
Emptier (usually a background thread in the Hadoop client or a scheduled
service):
1. It renames .Trash/Current to a timestamped checkpoint like
.Trash/<timestamp>.
2. After the retention period expires, it issues a recursive delete on the
timestamped directory.
3. This recursive delete finally moves the keys to the OM's deletedTable for
physical reclamation.
Summary Flow
┌───────────────┬──────────────┬───────────────────────┬─────────────────────────────────────────────────────────┐
│ Client Action │ Trash Status │ OM Operation │ Result
│
├───────────────┼──────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤
│ rm <file> │ Disabled │ deleteKey │ Moved to
deletedTable (Asynchronous reclamation begins) │
│ rm <file> │ Enabled │ renameKey │ Moved to
.Trash/Current (No space reclaimed yet) │
│ rm -skipTrash │ Enabled │ deleteKey │ Moved to
deletedTable (Immediate reclamation begins) │
│ Trash Emptier │ N/A │ deleteKey (recursive) │ Moved to
deletedTable (Space finally reclaimed) │
└───────────────┴──────────────┴───────────────────────┴─────────────────────────────────────────────────────────┘
Key takeaway: Trash in Ozone is a Client-Side simulation achieved via
renames. The Ozone Manager itself does not have a "Trash Table"; it only sees
renames or final deletes.
---
Inside OM:
✦ The deletion process in Apache Ozone is a multi-phase, asynchronous workflow
designed to handle massive scale and File System Optimized (FSO) structures
efficiently.
Based on the provided source files, here is the lifecycle of a deleted file
or directory as it moves through the Ozone Manager (OM) tables:
1. Initial Deletion (Synchronous Phase)
When a client issues a delete request, OMKeyDeleteRequest handles the
transaction:
* For Files: The entry is removed from the fileTable (or keyTable in non-FSO
buckets) and a new entry is added to the deletedTable.
* For Directories (FSO): The directory is removed from the directoryTable
and moved to the deletedDirectoryTable.
* Quota Update: The bucket's used bytes and namespace quota are updated at
this stage to reflect the "logical" deletion.
2. Directory Expansion (DirectoryDeletingService)
For FSO buckets, deleting a directory does not immediately move all its
children. Instead, the DirectoryDeletingService processes them asynchronously:
* Scanning: It scans the deletedDirectoryTable for pending deletions.
* Recursive Movement:
* Sub-files: It finds all files within the deleted directory and moves
them from the fileTable to the deletedTable.
* Sub-directories: It finds sub-directories and moves them from the
directoryTable to the deletedDirectoryTable so they can be processed in the next
iteration.
* Purging the Directory: Once a directory is empty (all children have been
moved to the deleted tables), the service sends a PurgePathRequest to remove
the directory itself from the deletedDirectoryTable.
3. Block Reclamation (KeyDeletingService)
The KeyDeletingService is responsible for the physical reclamation of storage
space occupied by files in the deletedTable:
* SCM Coordination: It scans the deletedTable, extracts the block list
(BlockGroups) for each file, and sends a request to the Storage Container
Manager
(SCM) to delete these blocks.
* Wait for ACK: It waits for SCM to acknowledge that the block deletion
intent has been persisted.
* Trigger Purge: Once SCM acknowledges, the service initiates the final
cleanup by submitting an OMKeyPurgeRequest.
4. Final Metadata Cleanup (OMKeyPurgeRequest)
This is the terminal state for a file's metadata:
* Purging: The OMKeyPurgeRequest removes the entries from the deletedTable.
* Snapshot Handling: If snapshots are involved, it may move keys to a
"previous" snapshot's deleted table instead of purging them, ensuring data
consistency across the snapshot chain.
Summary Table Flow
┌──────┬────────────┬───────────────────────┬───────────────────────┬──────────────────────────────────────────────┐
│ Step │ Entity │ Source Table │ Destination Table │ Handled
By │
├──────┼────────────┼───────────────────────┼───────────────────────┼──────────────────────────────────────────────┤
│ 1 │ File │ fileTable │ deletedTable │
OMKeyDeleteRequest │
│ 1 │ Directory │ directoryTable │ deletedDirectoryTable │
OMKeyDeleteRequest (FSO) │
│ 2 │ Child File │ fileTable │ deletedTable │
DirectoryDeletingService │
│ 2 │ Child Dir │ directoryTable │ deletedDirectoryTable │
DirectoryDeletingService │
│ 3 │ Final File │ deletedTable │ Removed │
KeyDeletingService -> OMKeyPurgeRequest │
│ 4 │ Final Dir │ deletedDirectoryTable │ Removed │
DirectoryDeletingService -> PurgePathRequest │
└──────┴────────────┴───────────────────────┴───────────────────────┴──────────────────────────────────────────────┘
---
SCM:
✦ The block reclamation process in SCM (Storage Container Manager) is an
asynchronous, transaction-based workflow that ensures data is physically deleted
from Datanodes while maintaining consistency across replicas.
Here is the step-by-step implementation of how SCM handles block reclamation:
1. Transaction Persistence (DeletedBlockLog)
When SCM receives a deleteKeyBlocks request from the Ozone Manager (OM), it
doesn't delete the blocks immediately. Instead, it creates a durable record:
* Grouping: SCM groups the blocks by their Container ID.
* Logging: It creates a DeletedBlocksTransaction for each container. Each
transaction is assigned a unique Transaction ID (TXID).
* Storage: These transactions are persisted in the SCM metadata database in
the DeletedBlocksTXTable. This ensures that even if SCM restarts, it won't
"forget" to delete these blocks.
2. Throttled Command Dispatch (SCMBlockDeletingService)
A background service called SCMBlockDeletingService periodically scans the
DeletedBlocksTXTable to move the process forward:
* Node Selection: It identifies healthy Datanodes that have room in their
command queues (to avoid overwhelming them).
* Transaction Matching: For each pending transaction, it identifies which
Datanodes hold a replica of that container.
* Command Creation: It bundles multiple transactions into a
DeleteBlocksCommand for a specific Datanode.
* Dispatch: The command is sent to the Datanode, typically as a payload in a
heartbeat response.
3. Physical Deletion (Datanode Side)
Once a Datanode receives the DeleteBlocksCommand:
* It iterates through the transactions and physically deletes the chunks
associated with the listed Block IDs from its local disks.
* It records the success or failure of each transaction.
4. Acknowledgement and Final Cleanup
SCM needs to ensure that all replicas of a block are deleted before it
considers the task complete:
* ACK Reporting: The Datanode sends back a ContainerBlocksDeletionACKProto
(via DeleteBlockStatus report) to SCM.
* Tracking: SCM’s SCMDeletedBlockTransactionStatusManager tracks which
Datanodes have acknowledged which TXIDs.
* Removal: Only after every replica (e.g., all 3 nodes in a Ratis pipeline)
has successfully acknowledged the deletion does SCM finally remove the
transaction from the DeletedBlocksTXTable.
Why this architecture?
* Fault Tolerance: By using a persistent log, SCM ensures that deletions
happen eventually even if nodes are temporarily down.
* Consistency: SCM won't stop retrying until every single replica is
confirmed deleted, preventing "orphan" blocks from wasting space.
* Performance: Batching multiple block deletions into a single transaction
and command reduces RPC overhead between OM, SCM, and Datanodes.
---
Datanode:
✦ Block deletion in the Datanode is a two-step process: first, it marks the
metadata for deletion upon receiving a command from SCM, and then a background
service physically removes the data from the disk.
Here is the detailed implementation:
1. Metadata Marking (DeleteBlocksCommandHandler)
When the Datanode receives a DeleteBlocksCommand from SCM, it triggers the
DeleteBlocksCommandHandler:
* Queueing: The command is added to a deleteCommandQueues. A dedicated
DeleteCmdWorker thread picks up the commands.
* Transaction Processing: For each transaction in the command, a
ProcessTransactionTask is executed.
* Metadata Update:
* Schema V2/V3: It adds the transaction to a DeletedBlocksTXTable within
the container's database.
* Schema V1: It moves the block entry from the blockDataTable to the
deletedBlocksTable (effectively renaming the key to a "deleting" prefix).
* ACK: Once the metadata is updated, the Datanode sends an Acknowledgement
(ACK) back to SCM. SCM then considers its part of the transaction "done" (once
all replicas ACK).
2. Physical Deletion (BlockDeletingService)
The actual removal of chunks from the disk is handled by a background service
to avoid blocking the main heartbeat/command processing:
* Choosing Containers: The BlockDeletingService periodically scans
containers and identifies those with pending deletions.
* Task Execution: It creates a BlockDeletingTask for each container.
* Chunk Removal:
* The task iterates through the DeletedBlocksTXTable (or the deleted
blocks in Schema V1).
* It calls handler.deleteBlock(), which physically deletes the chunk
files from the local file system.
* DB Cleanup: After the files are deleted, the service:
* Removes the block entries from the blockDataTable.
* Removes the transaction from the DeletedBlocksTXTable.
* Updates the container's metadata (e.g., reduces usedBytes and
pendingDeletionBlocks count).
* Empty Container Check: If a container becomes completely empty after
deletion, it is marked as such, allowing SCM to potentially delete the container
itself later.
Summary of Component Roles
┌────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Component │ Responsibility
│
├────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ DeleteBlocksCommandHandler │ Receives RPC from SCM, persists deletion
intent in the container DB, and sends ACK. │
│ BlockDeletingService │ Background thread that throttles and schedules
physical disk cleanup tasks. │
│ BlockDeletingTask │ Performs the actual rm of chunk files and
updates the container DB counters. │
│ Handler │ Interface used by the task to interact with
the specific storage engine (e.g., KeyValueHandler). │
└────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
This decoupled architecture ensures that the Ozone Manager and SCM don't have
to wait for slow disk I/O to confirm a deletion, while still guaranteeing
that the space will eventually be reclaimed.
--
In the case of OBS buckets:
✦ OBS and LEGACY buckets use a flat key structure, simplifying the deletion
process in the Ozone Manager. Unlike FSO buckets, which require
DirectoryDeletingService to expand directory deletions, OBS buckets move keys
directly from the keyTable to the deletedTable.
1. Synchronous Phase (OMKeyDeleteRequest)
When a client deletes an object in an OBS bucket:
* Table Access: OM accesses the keyTable (specifically
getKeyTable(BucketLayout.OBJECT_STORE)).
* Tombstone: It adds a cache entry (tombstone) to the keyTable to mark the
key as deleted in memory.
* Quota: The bucket's usedBytes and usedNamespace are updated immediately.
2. Response and Batch Update (OMKeyDeleteResponse)
The addToDBBatch method in the response handles the persistent changes:
* Remove from Source: The key is deleted from the keyTable.
* Move to Deleted: The key's metadata (OmKeyInfo) is wrapped into a
RepeatedOmKeyInfo and added to the deletedTable.
* Note: Even though OBS has "directories" in the name (e.g.,
dir1/file1), these are just string prefixes. There is no directoryTable or
deletedDirectoryTable involved.
* Deleted Table Key Format: The key in the deletedTable is typically
formatted as volume/bucket/keyName/objectID.
3. Asynchronous Reclamation (KeyDeletingService)
From this point forward, the process is identical to the FSO file deletion
process:
* KeyDeletingService picks up the entry from the deletedTable.
* It coordinates with SCM to delete the physical blocks.
* Once SCM acknowledges, the entry is permanently removed from the
deletedTable via OMKeyPurgeRequest.
Comparison: FSO vs. OBS Deletion in OM
┌──────────────────────┬─────────────────────────────────────┬─────────────────────┐
│ Feature │ FSO Bucket │ OBS / LEGACY
Bucket │
├──────────────────────┼─────────────────────────────────────┼─────────────────────┤
│ Source Tables │ fileTable, directoryTable │ keyTable
│
│ Intermediate Table │ deletedDirectoryTable (for dirs) │ None
│
│ Final Deletion Table │ deletedTable (for files) │ deletedTable
│
│ Background Expansion │ Required (DirectoryDeletingService) │ Not Required
│
│ Complexity │ High (Tree traversal needed) │ Low (Direct
move) │
└──────────────────────┴─────────────────────────────────────┴─────────────────────┘
In short, OBS deletion is much simpler because it bypasses the entire
"directory expansion" logic. Every object is treated as a single independent
unit,
regardless of whether its name contains slashes.
---
was:
A very common question users ask is: when a user issues a delete request, how
does it eventually get deleted? Due to Ozone's distributed nature, it can be
tricky to understand all the moving parts, so this page gives a holistic
overview of delete.
Trace every part of a delete request from beginning to end. This includes:
- Client getting encryption keys
- Client calling OM to delete the key
- OM validating client's Kerberos principal
- OM checking permissions (Ranger or Native ACLs) - OMKeyDeleteRequest
- OM marking the key for deletion and removing it from the namespace
- OM acking to the client
- OM sending blocks to delete to SCM
- SCM sending blocks to delete to the Datanode
- Datanode removes data
Note that delete works the same regardless of replication type. Also document
timing of background services and their batch size to estimate the rate of
deletion.
---
Inside OM:
✦ The deletion process in Apache Ozone is a multi-phase, asynchronous workflow
designed to handle massive scale and File System Optimized (FSO) structures
efficiently.
Based on the provided source files, here is the lifecycle of a deleted file
or directory as it moves through the Ozone Manager (OM) tables:
1. Initial Deletion (Synchronous Phase)
When a client issues a delete request, OMKeyDeleteRequest handles the
transaction:
* For Files: The entry is removed from the fileTable (or keyTable in non-FSO
buckets) and a new entry is added to the deletedTable.
* For Directories (FSO): The directory is removed from the directoryTable
and moved to the deletedDirectoryTable.
* Quota Update: The bucket's used bytes and namespace quota are updated at
this stage to reflect the "logical" deletion.
2. Directory Expansion (DirectoryDeletingService)
For FSO buckets, deleting a directory does not immediately move all its
children. Instead, the DirectoryDeletingService processes them asynchronously:
* Scanning: It scans the deletedDirectoryTable for pending deletions.
* Recursive Movement:
* Sub-files: It finds all files within the deleted directory and moves
them from the fileTable to the deletedTable.
* Sub-directories: It finds sub-directories and moves them from the
directoryTable to the deletedDirectoryTable so they can be processed in the next
iteration.
* Purging the Directory: Once a directory is empty (all children have been
moved to the deleted tables), the service sends a PurgePathRequest to remove
the directory itself from the deletedDirectoryTable.
3. Block Reclamation (KeyDeletingService)
The KeyDeletingService is responsible for the physical reclamation of storage
space occupied by files in the deletedTable:
* SCM Coordination: It scans the deletedTable, extracts the block list
(BlockGroups) for each file, and sends a request to the Storage Container
Manager
(SCM) to delete these blocks.
* Wait for ACK: It waits for SCM to acknowledge that the block deletion
intent has been persisted.
* Trigger Purge: Once SCM acknowledges, the service initiates the final
cleanup by submitting an OMKeyPurgeRequest.
4. Final Metadata Cleanup (OMKeyPurgeRequest)
This is the terminal state for a file's metadata:
* Purging: The OMKeyPurgeRequest removes the entries from the deletedTable.
* Snapshot Handling: If snapshots are involved, it may move keys to a
"previous" snapshot's deleted table instead of purging them, ensuring data
consistency across the snapshot chain.
Summary Table Flow
┌──────┬────────────┬───────────────────────┬───────────────────────┬──────────────────────────────────────────────┐
│ Step │ Entity │ Source Table │ Destination Table │ Handled
By │
├──────┼────────────┼───────────────────────┼───────────────────────┼──────────────────────────────────────────────┤
│ 1 │ File │ fileTable │ deletedTable │
OMKeyDeleteRequest │
│ 1 │ Directory │ directoryTable │ deletedDirectoryTable │
OMKeyDeleteRequest (FSO) │
│ 2 │ Child File │ fileTable │ deletedTable │
DirectoryDeletingService │
│ 2 │ Child Dir │ directoryTable │ deletedDirectoryTable │
DirectoryDeletingService │
│ 3 │ Final File │ deletedTable │ Removed │
KeyDeletingService -> OMKeyPurgeRequest │
│ 4 │ Final Dir │ deletedDirectoryTable │ Removed │
DirectoryDeletingService -> PurgePathRequest │
└──────┴────────────┴───────────────────────┴───────────────────────┴──────────────────────────────────────────────┘
---
SCM:
✦ The block reclamation process in SCM (Storage Container Manager) is an
asynchronous, transaction-based workflow that ensures data is physically deleted
from Datanodes while maintaining consistency across replicas.
Here is the step-by-step implementation of how SCM handles block reclamation:
1. Transaction Persistence (DeletedBlockLog)
When SCM receives a deleteKeyBlocks request from the Ozone Manager (OM), it
doesn't delete the blocks immediately. Instead, it creates a durable record:
* Grouping: SCM groups the blocks by their Container ID.
* Logging: It creates a DeletedBlocksTransaction for each container. Each
transaction is assigned a unique Transaction ID (TXID).
* Storage: These transactions are persisted in the SCM metadata database in
the DeletedBlocksTXTable. This ensures that even if SCM restarts, it won't
"forget" to delete these blocks.
2. Throttled Command Dispatch (SCMBlockDeletingService)
A background service called SCMBlockDeletingService periodically scans the
DeletedBlocksTXTable to move the process forward:
* Node Selection: It identifies healthy Datanodes that have room in their
command queues (to avoid overwhelming them).
* Transaction Matching: For each pending transaction, it identifies which
Datanodes hold a replica of that container.
* Command Creation: It bundles multiple transactions into a
DeleteBlocksCommand for a specific Datanode.
* Dispatch: The command is sent to the Datanode, typically as a payload in a
heartbeat response.
3. Physical Deletion (Datanode Side)
Once a Datanode receives the DeleteBlocksCommand:
* It iterates through the transactions and physically deletes the chunks
associated with the listed Block IDs from its local disks.
* It records the success or failure of each transaction.
4. Acknowledgement and Final Cleanup
SCM needs to ensure that all replicas of a block are deleted before it
considers the task complete:
* ACK Reporting: The Datanode sends back a ContainerBlocksDeletionACKProto
(via DeleteBlockStatus report) to SCM.
* Tracking: SCM’s SCMDeletedBlockTransactionStatusManager tracks which
Datanodes have acknowledged which TXIDs.
* Removal: Only after every replica (e.g., all 3 nodes in a Ratis pipeline)
has successfully acknowledged the deletion does SCM finally remove the
transaction from the DeletedBlocksTXTable.
Why this architecture?
* Fault Tolerance: By using a persistent log, SCM ensures that deletions
happen eventually even if nodes are temporarily down.
* Consistency: SCM won't stop retrying until every single replica is
confirmed deleted, preventing "orphan" blocks from wasting space.
* Performance: Batching multiple block deletions into a single transaction
and command reduces RPC overhead between OM, SCM, and Datanodes.
---
Datanode:
✦ Block deletion in the Datanode is a two-step process: first, it marks the
metadata for deletion upon receiving a command from SCM, and then a background
service physically removes the data from the disk.
Here is the detailed implementation:
1. Metadata Marking (DeleteBlocksCommandHandler)
When the Datanode receives a DeleteBlocksCommand from SCM, it triggers the
DeleteBlocksCommandHandler:
* Queueing: The command is added to a deleteCommandQueues. A dedicated
DeleteCmdWorker thread picks up the commands.
* Transaction Processing: For each transaction in the command, a
ProcessTransactionTask is executed.
* Metadata Update:
* Schema V2/V3: It adds the transaction to a DeletedBlocksTXTable within
the container's database.
* Schema V1: It moves the block entry from the blockDataTable to the
deletedBlocksTable (effectively renaming the key to a "deleting" prefix).
* ACK: Once the metadata is updated, the Datanode sends an Acknowledgement
(ACK) back to SCM. SCM then considers its part of the transaction "done" (once
all replicas ACK).
2. Physical Deletion (BlockDeletingService)
The actual removal of chunks from the disk is handled by a background service
to avoid blocking the main heartbeat/command processing:
* Choosing Containers: The BlockDeletingService periodically scans
containers and identifies those with pending deletions.
* Task Execution: It creates a BlockDeletingTask for each container.
* Chunk Removal:
* The task iterates through the DeletedBlocksTXTable (or the deleted
blocks in Schema V1).
* It calls handler.deleteBlock(), which physically deletes the chunk
files from the local file system.
* DB Cleanup: After the files are deleted, the service:
* Removes the block entries from the blockDataTable.
* Removes the transaction from the DeletedBlocksTXTable.
* Updates the container's metadata (e.g., reduces usedBytes and
pendingDeletionBlocks count).
* Empty Container Check: If a container becomes completely empty after
deletion, it is marked as such, allowing SCM to potentially delete the container
itself later.
Summary of Component Roles
┌────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Component │ Responsibility
│
├────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ DeleteBlocksCommandHandler │ Receives RPC from SCM, persists deletion
intent in the container DB, and sends ACK. │
│ BlockDeletingService │ Background thread that throttles and schedules
physical disk cleanup tasks. │
│ BlockDeletingTask │ Performs the actual rm of chunk files and
updates the container DB counters. │
│ Handler │ Interface used by the task to interact with
the specific storage engine (e.g., KeyValueHandler). │
└────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
This decoupled architecture ensures that the Ozone Manager and SCM don't have
to wait for slow disk I/O to confirm a deletion, while still guaranteeing
that the space will eventually be reclaimed.
--
In the case of OBS buckets:
✦ OBS and LEGACY buckets use a flat key structure, simplifying the deletion
process in the Ozone Manager. Unlike FSO buckets, which require
DirectoryDeletingService to expand directory deletions, OBS buckets move keys
directly from the keyTable to the deletedTable.
1. Synchronous Phase (OMKeyDeleteRequest)
When a client deletes an object in an OBS bucket:
* Table Access: OM accesses the keyTable (specifically
getKeyTable(BucketLayout.OBJECT_STORE)).
* Tombstone: It adds a cache entry (tombstone) to the keyTable to mark the
key as deleted in memory.
* Quota: The bucket's usedBytes and usedNamespace are updated immediately.
2. Response and Batch Update (OMKeyDeleteResponse)
The addToDBBatch method in the response handles the persistent changes:
* Remove from Source: The key is deleted from the keyTable.
* Move to Deleted: The key's metadata (OmKeyInfo) is wrapped into a
RepeatedOmKeyInfo and added to the deletedTable.
* Note: Even though OBS has "directories" in the name (e.g.,
dir1/file1), these are just string prefixes. There is no directoryTable or
deletedDirectoryTable involved.
* Deleted Table Key Format: The key in the deletedTable is typically
formatted as volume/bucket/keyName/objectID.
3. Asynchronous Reclamation (KeyDeletingService)
From this point forward, the process is identical to the FSO file deletion
process:
* KeyDeletingService picks up the entry from the deletedTable.
* It coordinates with SCM to delete the physical blocks.
* Once SCM acknowledges, the entry is permanently removed from the
deletedTable via OMKeyPurgeRequest.
Comparison: FSO vs. OBS Deletion in OM
┌──────────────────────┬─────────────────────────────────────┬─────────────────────┐
│ Feature │ FSO Bucket │ OBS / LEGACY
Bucket │
├──────────────────────┼─────────────────────────────────────┼─────────────────────┤
│ Source Tables │ fileTable, directoryTable │ keyTable
│
│ Intermediate Table │ deletedDirectoryTable (for dirs) │ None
│
│ Final Deletion Table │ deletedTable (for files) │ deletedTable
│
│ Background Expansion │ Required (DirectoryDeletingService) │ Not Required
│
│ Complexity │ High (Tree traversal needed) │ Low (Direct
move) │
└──────────────────────┴─────────────────────────────────────┴─────────────────────┘
In short, OBS deletion is much simpler because it bypasses the entire
"directory expansion" logic. Every object is treated as a single independent
unit,
regardless of whether its name contains slashes.
---
> [Docs] [System Internals] Implementation of Delete Operations
> -------------------------------------------------------------
>
> Key: HDDS-15027
> URL: https://issues.apache.org/jira/browse/HDDS-15027
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: documentation
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> A very common question users ask is: when a user issues a delete request, how
> does it eventually get deleted? Due to Ozone's distributed nature, it can be
> tricky to understand all the moving parts, so this page gives a holistic
> overview of delete.
> ---
> Client side:
> ✦ Client-side deletion in Ozone depends on the interface and bucket layout,
> with trash operations implemented as a rename to .Trash/Current by both the
> Ozone CLI and FileSystem.
> 1. Ozone CLI Deletion (DeleteKeyHandler)
> * FSO Buckets:
> * If Trash is enabled (fs.trash.interval > 0), the CLI renames the key
> to .Trash/<user>/Current/<keyName>.
> * If Trash is disabled, it calls bucket.deleteKey(keyName) for direct
> deletion.
> * OBS/Legacy Buckets:
> * Directly calls bucket.deleteKey(keyName), which triggers a logical
> delete in OM. Trash is not natively supported for non-FSO buckets via the
> ozone
> sh key delete command.
> 2. Ozone FileSystem Deletion (OzoneFileSystem)
> The FileSystem interface (o3fs:// or ofs://) typically uses the standard
> Hadoop Trash Policy:
> * When you run hadoop fs -rm <path>, the Hadoop client code checks the
> configuration.
> * If trash is enabled, the client performs a Rename operation:
> * Source: /path/to/file
> * Destination: /.Trash/Current/path/to/file
> * If -skipTrash is used or trash is disabled, it calls
> FileSystem.delete(), which maps to a direct deleteKey RPC to the Ozone
> Manager.
> 3. OM Logic for Deletion vs. Trash
> * Rename (Trash): To the OM, moving a file to trash is just a metadata
> update in the keyTable or fileTable. The object ID remains the same, and no
> blocks
> are reclaimed.
> * Direct Delete: Triggered by bucket.deleteKey(). As previously explained,
> OM moves the metadata to the deletedTable, starting the asynchronous block
> reclamation process.
> 4. Trash Emptier
> Because trash is just a normal directory (.Trash), it is emptied by a Trash
> Emptier (usually a background thread in the Hadoop client or a scheduled
> service):
> 1. It renames .Trash/Current to a timestamped checkpoint like
> .Trash/<timestamp>.
> 2. After the retention period expires, it issues a recursive delete on the
> timestamped directory.
> 3. This recursive delete finally moves the keys to the OM's deletedTable
> for physical reclamation.
> Summary Flow
>
> ┌───────────────┬──────────────┬───────────────────────┬─────────────────────────────────────────────────────────┐
> │ Client Action │ Trash Status │ OM Operation │ Result
> │
>
> ├───────────────┼──────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤
> │ rm <file> │ Disabled │ deleteKey │ Moved to
> deletedTable (Asynchronous reclamation begins) │
> │ rm <file> │ Enabled │ renameKey │ Moved to
> .Trash/Current (No space reclaimed yet) │
> │ rm -skipTrash │ Enabled │ deleteKey │ Moved to
> deletedTable (Immediate reclamation begins) │
> │ Trash Emptier │ N/A │ deleteKey (recursive) │ Moved to
> deletedTable (Space finally reclaimed) │
>
> └───────────────┴──────────────┴───────────────────────┴─────────────────────────────────────────────────────────┘
> Key takeaway: Trash in Ozone is a Client-Side simulation achieved via
> renames. The Ozone Manager itself does not have a "Trash Table"; it only sees
> renames or final deletes.
> ---
> Inside OM:
> ✦ The deletion process in Apache Ozone is a multi-phase, asynchronous
> workflow designed to handle massive scale and File System Optimized (FSO)
> structures
> efficiently.
> Based on the provided source files, here is the lifecycle of a deleted file
> or directory as it moves through the Ozone Manager (OM) tables:
> 1. Initial Deletion (Synchronous Phase)
> When a client issues a delete request, OMKeyDeleteRequest handles the
> transaction:
> * For Files: The entry is removed from the fileTable (or keyTable in
> non-FSO buckets) and a new entry is added to the deletedTable.
> * For Directories (FSO): The directory is removed from the directoryTable
> and moved to the deletedDirectoryTable.
> * Quota Update: The bucket's used bytes and namespace quota are updated at
> this stage to reflect the "logical" deletion.
> 2. Directory Expansion (DirectoryDeletingService)
> For FSO buckets, deleting a directory does not immediately move all its
> children. Instead, the DirectoryDeletingService processes them asynchronously:
> * Scanning: It scans the deletedDirectoryTable for pending deletions.
> * Recursive Movement:
> * Sub-files: It finds all files within the deleted directory and moves
> them from the fileTable to the deletedTable.
> * Sub-directories: It finds sub-directories and moves them from the
> directoryTable to the deletedDirectoryTable so they can be processed in the
> next
> iteration.
> * Purging the Directory: Once a directory is empty (all children have been
> moved to the deleted tables), the service sends a PurgePathRequest to remove
> the directory itself from the deletedDirectoryTable.
> 3. Block Reclamation (KeyDeletingService)
> The KeyDeletingService is responsible for the physical reclamation of
> storage space occupied by files in the deletedTable:
> * SCM Coordination: It scans the deletedTable, extracts the block list
> (BlockGroups) for each file, and sends a request to the Storage Container
> Manager
> (SCM) to delete these blocks.
> * Wait for ACK: It waits for SCM to acknowledge that the block deletion
> intent has been persisted.
> * Trigger Purge: Once SCM acknowledges, the service initiates the final
> cleanup by submitting an OMKeyPurgeRequest.
> 4. Final Metadata Cleanup (OMKeyPurgeRequest)
> This is the terminal state for a file's metadata:
> * Purging: The OMKeyPurgeRequest removes the entries from the deletedTable.
> * Snapshot Handling: If snapshots are involved, it may move keys to a
> "previous" snapshot's deleted table instead of purging them, ensuring data
> consistency across the snapshot chain.
> Summary Table Flow
>
> ┌──────┬────────────┬───────────────────────┬───────────────────────┬──────────────────────────────────────────────┐
> │ Step │ Entity │ Source Table │ Destination Table │
> Handled By │
>
> ├──────┼────────────┼───────────────────────┼───────────────────────┼──────────────────────────────────────────────┤
> │ 1 │ File │ fileTable │ deletedTable │
> OMKeyDeleteRequest │
> │ 1 │ Directory │ directoryTable │ deletedDirectoryTable │
> OMKeyDeleteRequest (FSO) │
> │ 2 │ Child File │ fileTable │ deletedTable │
> DirectoryDeletingService │
> │ 2 │ Child Dir │ directoryTable │ deletedDirectoryTable │
> DirectoryDeletingService │
> │ 3 │ Final File │ deletedTable │ Removed │
> KeyDeletingService -> OMKeyPurgeRequest │
> │ 4 │ Final Dir │ deletedDirectoryTable │ Removed │
> DirectoryDeletingService -> PurgePathRequest │
>
> └──────┴────────────┴───────────────────────┴───────────────────────┴──────────────────────────────────────────────┘
> ---
> SCM:
> ✦ The block reclamation process in SCM (Storage Container Manager) is an
> asynchronous, transaction-based workflow that ensures data is physically
> deleted
> from Datanodes while maintaining consistency across replicas.
> Here is the step-by-step implementation of how SCM handles block
> reclamation:
> 1. Transaction Persistence (DeletedBlockLog)
> When SCM receives a deleteKeyBlocks request from the Ozone Manager (OM), it
> doesn't delete the blocks immediately. Instead, it creates a durable record:
> * Grouping: SCM groups the blocks by their Container ID.
> * Logging: It creates a DeletedBlocksTransaction for each container. Each
> transaction is assigned a unique Transaction ID (TXID).
> * Storage: These transactions are persisted in the SCM metadata database
> in the DeletedBlocksTXTable. This ensures that even if SCM restarts, it won't
> "forget" to delete these blocks.
> 2. Throttled Command Dispatch (SCMBlockDeletingService)
> A background service called SCMBlockDeletingService periodically scans the
> DeletedBlocksTXTable to move the process forward:
> * Node Selection: It identifies healthy Datanodes that have room in their
> command queues (to avoid overwhelming them).
> * Transaction Matching: For each pending transaction, it identifies which
> Datanodes hold a replica of that container.
> * Command Creation: It bundles multiple transactions into a
> DeleteBlocksCommand for a specific Datanode.
> * Dispatch: The command is sent to the Datanode, typically as a payload in
> a heartbeat response.
> 3. Physical Deletion (Datanode Side)
> Once a Datanode receives the DeleteBlocksCommand:
> * It iterates through the transactions and physically deletes the chunks
> associated with the listed Block IDs from its local disks.
> * It records the success or failure of each transaction.
> 4. Acknowledgement and Final Cleanup
> SCM needs to ensure that all replicas of a block are deleted before it
> considers the task complete:
> * ACK Reporting: The Datanode sends back a ContainerBlocksDeletionACKProto
> (via DeleteBlockStatus report) to SCM.
> * Tracking: SCM’s SCMDeletedBlockTransactionStatusManager tracks which
> Datanodes have acknowledged which TXIDs.
> * Removal: Only after every replica (e.g., all 3 nodes in a Ratis
> pipeline) has successfully acknowledged the deletion does SCM finally remove
> the
> transaction from the DeletedBlocksTXTable.
> Why this architecture?
> * Fault Tolerance: By using a persistent log, SCM ensures that deletions
> happen eventually even if nodes are temporarily down.
> * Consistency: SCM won't stop retrying until every single replica is
> confirmed deleted, preventing "orphan" blocks from wasting space.
> * Performance: Batching multiple block deletions into a single transaction
> and command reduces RPC overhead between OM, SCM, and Datanodes.
> ---
> Datanode:
> ✦ Block deletion in the Datanode is a two-step process: first, it marks the
> metadata for deletion upon receiving a command from SCM, and then a background
> service physically removes the data from the disk.
> Here is the detailed implementation:
> 1. Metadata Marking (DeleteBlocksCommandHandler)
> When the Datanode receives a DeleteBlocksCommand from SCM, it triggers the
> DeleteBlocksCommandHandler:
> * Queueing: The command is added to a deleteCommandQueues. A dedicated
> DeleteCmdWorker thread picks up the commands.
> * Transaction Processing: For each transaction in the command, a
> ProcessTransactionTask is executed.
> * Metadata Update:
> * Schema V2/V3: It adds the transaction to a DeletedBlocksTXTable
> within the container's database.
> * Schema V1: It moves the block entry from the blockDataTable to the
> deletedBlocksTable (effectively renaming the key to a "deleting" prefix).
> * ACK: Once the metadata is updated, the Datanode sends an Acknowledgement
> (ACK) back to SCM. SCM then considers its part of the transaction "done" (once
> all replicas ACK).
> 2. Physical Deletion (BlockDeletingService)
> The actual removal of chunks from the disk is handled by a background
> service to avoid blocking the main heartbeat/command processing:
> * Choosing Containers: The BlockDeletingService periodically scans
> containers and identifies those with pending deletions.
> * Task Execution: It creates a BlockDeletingTask for each container.
> * Chunk Removal:
> * The task iterates through the DeletedBlocksTXTable (or the deleted
> blocks in Schema V1).
> * It calls handler.deleteBlock(), which physically deletes the chunk
> files from the local file system.
> * DB Cleanup: After the files are deleted, the service:
> * Removes the block entries from the blockDataTable.
> * Removes the transaction from the DeletedBlocksTXTable.
> * Updates the container's metadata (e.g., reduces usedBytes and
> pendingDeletionBlocks count).
> * Empty Container Check: If a container becomes completely empty after
> deletion, it is marked as such, allowing SCM to potentially delete the
> container
> itself later.
> Summary of Component Roles
>
> ┌────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
> │ Component │ Responsibility
> │
>
> ├────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
> │ DeleteBlocksCommandHandler │ Receives RPC from SCM, persists deletion
> intent in the container DB, and sends ACK. │
> │ BlockDeletingService │ Background thread that throttles and
> schedules physical disk cleanup tasks. │
> │ BlockDeletingTask │ Performs the actual rm of chunk files and
> updates the container DB counters. │
> │ Handler │ Interface used by the task to interact with
> the specific storage engine (e.g., KeyValueHandler). │
>
> └────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
> This decoupled architecture ensures that the Ozone Manager and SCM don't
> have to wait for slow disk I/O to confirm a deletion, while still guaranteeing
> that the space will eventually be reclaimed.
> --
> In the case of OBS buckets:
> ✦ OBS and LEGACY buckets use a flat key structure, simplifying the deletion
> process in the Ozone Manager. Unlike FSO buckets, which require
> DirectoryDeletingService to expand directory deletions, OBS buckets move
> keys directly from the keyTable to the deletedTable.
> 1. Synchronous Phase (OMKeyDeleteRequest)
> When a client deletes an object in an OBS bucket:
> * Table Access: OM accesses the keyTable (specifically
> getKeyTable(BucketLayout.OBJECT_STORE)).
> * Tombstone: It adds a cache entry (tombstone) to the keyTable to mark the
> key as deleted in memory.
> * Quota: The bucket's usedBytes and usedNamespace are updated immediately.
> 2. Response and Batch Update (OMKeyDeleteResponse)
> The addToDBBatch method in the response handles the persistent changes:
> * Remove from Source: The key is deleted from the keyTable.
> * Move to Deleted: The key's metadata (OmKeyInfo) is wrapped into a
> RepeatedOmKeyInfo and added to the deletedTable.
> * Note: Even though OBS has "directories" in the name (e.g.,
> dir1/file1), these are just string prefixes. There is no directoryTable or
> deletedDirectoryTable involved.
> * Deleted Table Key Format: The key in the deletedTable is typically
> formatted as volume/bucket/keyName/objectID.
> 3. Asynchronous Reclamation (KeyDeletingService)
> From this point forward, the process is identical to the FSO file deletion
> process:
> * KeyDeletingService picks up the entry from the deletedTable.
> * It coordinates with SCM to delete the physical blocks.
> * Once SCM acknowledges, the entry is permanently removed from the
> deletedTable via OMKeyPurgeRequest.
> Comparison: FSO vs. OBS Deletion in OM
>
> ┌──────────────────────┬─────────────────────────────────────┬─────────────────────┐
> │ Feature │ FSO Bucket │ OBS / LEGACY
> Bucket │
>
> ├──────────────────────┼─────────────────────────────────────┼─────────────────────┤
> │ Source Tables │ fileTable, directoryTable │ keyTable
> │
> │ Intermediate Table │ deletedDirectoryTable (for dirs) │ None
> │
> │ Final Deletion Table │ deletedTable (for files) │ deletedTable
> │
> │ Background Expansion │ Required (DirectoryDeletingService) │ Not Required
> │
> │ Complexity │ High (Tree traversal needed) │ Low (Direct
> move) │
>
> └──────────────────────┴─────────────────────────────────────┴─────────────────────┘
> In short, OBS deletion is much simpler because it bypasses the entire
> "directory expansion" logic. Every object is treated as a single independent
> unit,
> regardless of whether its name contains slashes.
> ---
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
