[
https://issues.apache.org/jira/browse/HDDS-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-15027:
-----------------------------------
Description:
A very common question users ask is: when a user issues a delete request, how
does it eventually get deleted? Due to Ozone's distributed nature, it can be
tricky to understand all the moving parts, so this page gives a holistic
overview of delete.
Trace every part of a delete request from beginning to end. This includes:
- Client getting encryption keys
- Client calling OM to delete the key
- OM validating client's Kerberos principal
- OM checking permissions (Ranger or Native ACLs) - OMKeyDeleteRequest
- OM marking the key for deletion and removing it from the namespace
- OM acking to the client
- OM sending blocks to delete to SCM
- SCM sending blocks to delete to the Datanode
- Datanode removes data
Note that delete works the same regardless of replication type. Also document
timing of background services and their batch size to estimate the rate of
deletion.
---
Inside OM:
✦ The deletion process in Apache Ozone is a multi-phase, asynchronous workflow
designed to handle massive scale and File System Optimized (FSO) structures
efficiently.
Based on the provided source files, here is the lifecycle of a deleted file
or directory as it moves through the Ozone Manager (OM) tables:
1. Initial Deletion (Synchronous Phase)
When a client issues a delete request, OMKeyDeleteRequest handles the
transaction:
* For Files: The entry is removed from the fileTable (or keyTable in non-FSO
buckets) and a new entry is added to the deletedTable.
* For Directories (FSO): The directory is removed from the directoryTable
and moved to the deletedDirectoryTable.
* Quota Update: The bucket's used bytes and namespace quota are updated at
this stage to reflect the "logical" deletion.
2. Directory Expansion (DirectoryDeletingService)
For FSO buckets, deleting a directory does not immediately move all its
children. Instead, the DirectoryDeletingService processes them asynchronously:
* Scanning: It scans the deletedDirectoryTable for pending deletions.
* Recursive Movement:
* Sub-files: It finds all files within the deleted directory and moves
them from the fileTable to the deletedTable.
* Sub-directories: It finds sub-directories and moves them from the
directoryTable to the deletedDirectoryTable so they can be processed in the next
iteration.
* Purging the Directory: Once a directory is empty (all children have been
moved to the deleted tables), the service sends a PurgePathRequest to remove
the directory itself from the deletedDirectoryTable.
3. Block Reclamation (KeyDeletingService)
The KeyDeletingService is responsible for the physical reclamation of storage
space occupied by files in the deletedTable:
* SCM Coordination: It scans the deletedTable, extracts the block list
(BlockGroups) for each file, and sends a request to the Storage Container
Manager
(SCM) to delete these blocks.
* Wait for ACK: It waits for SCM to acknowledge that the block deletion
intent has been persisted.
* Trigger Purge: Once SCM acknowledges, the service initiates the final
cleanup by submitting an OMKeyPurgeRequest.
4. Final Metadata Cleanup (OMKeyPurgeRequest)
This is the terminal state for a file's metadata:
* Purging: The OMKeyPurgeRequest removes the entries from the deletedTable.
* Snapshot Handling: If snapshots are involved, it may move keys to a
"previous" snapshot's deleted table instead of purging them, ensuring data
consistency across the snapshot chain.
Summary Table Flow
┌──────┬────────────┬───────────────────────┬───────────────────────┬──────────────────────────────────────────────┐
│ Step │ Entity │ Source Table │ Destination Table │ Handled
By │
├──────┼────────────┼───────────────────────┼───────────────────────┼──────────────────────────────────────────────┤
│ 1 │ File │ fileTable │ deletedTable │
OMKeyDeleteRequest │
│ 1 │ Directory │ directoryTable │ deletedDirectoryTable │
OMKeyDeleteRequest (FSO) │
│ 2 │ Child File │ fileTable │ deletedTable │
DirectoryDeletingService │
│ 2 │ Child Dir │ directoryTable │ deletedDirectoryTable │
DirectoryDeletingService │
│ 3 │ Final File │ deletedTable │ Removed │
KeyDeletingService -> OMKeyPurgeRequest │
│ 4 │ Final Dir │ deletedDirectoryTable │ Removed │
DirectoryDeletingService -> PurgePathRequest │
└──────┴────────────┴───────────────────────┴───────────────────────┴──────────────────────────────────────────────┘
---
SCM:
✦ The block reclamation process in SCM (Storage Container Manager) is an
asynchronous, transaction-based workflow that ensures data is physically deleted
from Datanodes while maintaining consistency across replicas.
Here is the step-by-step implementation of how SCM handles block reclamation:
1. Transaction Persistence (DeletedBlockLog)
When SCM receives a deleteKeyBlocks request from the Ozone Manager (OM), it
doesn't delete the blocks immediately. Instead, it creates a durable record:
* Grouping: SCM groups the blocks by their Container ID.
* Logging: It creates a DeletedBlocksTransaction for each container. Each
transaction is assigned a unique Transaction ID (TXID).
* Storage: These transactions are persisted in the SCM metadata database in
the DeletedBlocksTXTable. This ensures that even if SCM restarts, it won't
"forget" to delete these blocks.
2. Throttled Command Dispatch (SCMBlockDeletingService)
A background service called SCMBlockDeletingService periodically scans the
DeletedBlocksTXTable to move the process forward:
* Node Selection: It identifies healthy Datanodes that have room in their
command queues (to avoid overwhelming them).
* Transaction Matching: For each pending transaction, it identifies which
Datanodes hold a replica of that container.
* Command Creation: It bundles multiple transactions into a
DeleteBlocksCommand for a specific Datanode.
* Dispatch: The command is sent to the Datanode, typically as a payload in a
heartbeat response.
3. Physical Deletion (Datanode Side)
Once a Datanode receives the DeleteBlocksCommand:
* It iterates through the transactions and physically deletes the chunks
associated with the listed Block IDs from its local disks.
* It records the success or failure of each transaction.
4. Acknowledgement and Final Cleanup
SCM needs to ensure that all replicas of a block are deleted before it
considers the task complete:
* ACK Reporting: The Datanode sends back a ContainerBlocksDeletionACKProto
(via DeleteBlockStatus report) to SCM.
* Tracking: SCM’s SCMDeletedBlockTransactionStatusManager tracks which
Datanodes have acknowledged which TXIDs.
* Removal: Only after every replica (e.g., all 3 nodes in a Ratis pipeline)
has successfully acknowledged the deletion does SCM finally remove the
transaction from the DeletedBlocksTXTable.
Why this architecture?
* Fault Tolerance: By using a persistent log, SCM ensures that deletions
happen eventually even if nodes are temporarily down.
* Consistency: SCM won't stop retrying until every single replica is
confirmed deleted, preventing "orphan" blocks from wasting space.
* Performance: Batching multiple block deletions into a single transaction
and command reduces RPC overhead between OM, SCM, and Datanodes.
was:
A very common question users ask is: when a user issues a delete request, how
does it eventually get deleted? Due to Ozone's distributed nature, it can be
tricky to understand all the moving parts, so this page gives a holistic
overview of delete.
Trace every part of a delete request from beginning to end. This includes:
- Client getting encryption keys
- Client calling OM to delete the key
- OM validating client's Kerberos principal
- OM checking permissions (Ranger or Native ACLs) - OMKeyDeleteRequest
- OM marking the key for deletion and removing it from the namespace
- OM acking to the client
- OM sending blocks to delete to SCM
- SCM sending blocks to delete to the Datanode
- Datanode removes data
Note that delete works the same regardless of replication type. Also document
timing of background services and their batch size to estimate the rate of
deletion.
---
Inside OM:
✦ The deletion process in Apache Ozone is a multi-phase, asynchronous workflow
designed to handle massive scale and File System Optimized (FSO) structures
efficiently.
Based on the provided source files, here is the lifecycle of a deleted file
or directory as it moves through the Ozone Manager (OM) tables:
1. Initial Deletion (Synchronous Phase)
When a client issues a delete request, OMKeyDeleteRequest handles the
transaction:
* For Files: The entry is removed from the fileTable (or keyTable in non-FSO
buckets) and a new entry is added to the deletedTable.
* For Directories (FSO): The directory is removed from the directoryTable
and moved to the deletedDirectoryTable.
* Quota Update: The bucket's used bytes and namespace quota are updated at
this stage to reflect the "logical" deletion.
2. Directory Expansion (DirectoryDeletingService)
For FSO buckets, deleting a directory does not immediately move all its
children. Instead, the DirectoryDeletingService processes them asynchronously:
* Scanning: It scans the deletedDirectoryTable for pending deletions.
* Recursive Movement:
* Sub-files: It finds all files within the deleted directory and moves
them from the fileTable to the deletedTable.
* Sub-directories: It finds sub-directories and moves them from the
directoryTable to the deletedDirectoryTable so they can be processed in the next
iteration.
* Purging the Directory: Once a directory is empty (all children have been
moved to the deleted tables), the service sends a PurgePathRequest to remove
the directory itself from the deletedDirectoryTable.
3. Block Reclamation (KeyDeletingService)
The KeyDeletingService is responsible for the physical reclamation of storage
space occupied by files in the deletedTable:
* SCM Coordination: It scans the deletedTable, extracts the block list
(BlockGroups) for each file, and sends a request to the Storage Container
Manager
(SCM) to delete these blocks.
* Wait for ACK: It waits for SCM to acknowledge that the block deletion
intent has been persisted.
* Trigger Purge: Once SCM acknowledges, the service initiates the final
cleanup by submitting an OMKeyPurgeRequest.
4. Final Metadata Cleanup (OMKeyPurgeRequest)
This is the terminal state for a file's metadata:
* Purging: The OMKeyPurgeRequest removes the entries from the deletedTable.
* Snapshot Handling: If snapshots are involved, it may move keys to a
"previous" snapshot's deleted table instead of purging them, ensuring data
consistency across the snapshot chain.
Summary Table Flow
┌──────┬────────────┬───────────────────────┬───────────────────────┬──────────────────────────────────────────────┐
│ Step │ Entity │ Source Table │ Destination Table │ Handled
By │
├──────┼────────────┼───────────────────────┼───────────────────────┼──────────────────────────────────────────────┤
│ 1 │ File │ fileTable │ deletedTable │
OMKeyDeleteRequest │
│ 1 │ Directory │ directoryTable │ deletedDirectoryTable │
OMKeyDeleteRequest (FSO) │
│ 2 │ Child File │ fileTable │ deletedTable │
DirectoryDeletingService │
│ 2 │ Child Dir │ directoryTable │ deletedDirectoryTable │
DirectoryDeletingService │
│ 3 │ Final File │ deletedTable │ Removed │
KeyDeletingService -> OMKeyPurgeRequest │
│ 4 │ Final Dir │ deletedDirectoryTable │ Removed │
DirectoryDeletingService -> PurgePathRequest │
└──────┴────────────┴───────────────────────┴───────────────────────┴──────────────────────────────────────────────┘
> [Docs] [System Internals] Implementation of Delete Operations
> -------------------------------------------------------------
>
> Key: HDDS-15027
> URL: https://issues.apache.org/jira/browse/HDDS-15027
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: documentation
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> A very common question users ask is: when a user issues a delete request, how
> does it eventually get deleted? Due to Ozone's distributed nature, it can be
> tricky to understand all the moving parts, so this page gives a holistic
> overview of delete.
> Trace every part of a delete request from beginning to end. This includes:
> - Client getting encryption keys
> - Client calling OM to delete the key
> - OM validating client's Kerberos principal
> - OM checking permissions (Ranger or Native ACLs) - OMKeyDeleteRequest
> - OM marking the key for deletion and removing it from the namespace
> - OM acking to the client
> - OM sending blocks to delete to SCM
> - SCM sending blocks to delete to the Datanode
> - Datanode removes data
> Note that delete works the same regardless of replication type. Also document
> timing of background services and their batch size to estimate the rate of
> deletion.
> ---
> Inside OM:
> ✦ The deletion process in Apache Ozone is a multi-phase, asynchronous
> workflow designed to handle massive scale and File System Optimized (FSO)
> structures
> efficiently.
> Based on the provided source files, here is the lifecycle of a deleted file
> or directory as it moves through the Ozone Manager (OM) tables:
> 1. Initial Deletion (Synchronous Phase)
> When a client issues a delete request, OMKeyDeleteRequest handles the
> transaction:
> * For Files: The entry is removed from the fileTable (or keyTable in
> non-FSO buckets) and a new entry is added to the deletedTable.
> * For Directories (FSO): The directory is removed from the directoryTable
> and moved to the deletedDirectoryTable.
> * Quota Update: The bucket's used bytes and namespace quota are updated at
> this stage to reflect the "logical" deletion.
> 2. Directory Expansion (DirectoryDeletingService)
> For FSO buckets, deleting a directory does not immediately move all its
> children. Instead, the DirectoryDeletingService processes them asynchronously:
> * Scanning: It scans the deletedDirectoryTable for pending deletions.
> * Recursive Movement:
> * Sub-files: It finds all files within the deleted directory and moves
> them from the fileTable to the deletedTable.
> * Sub-directories: It finds sub-directories and moves them from the
> directoryTable to the deletedDirectoryTable so they can be processed in the
> next
> iteration.
> * Purging the Directory: Once a directory is empty (all children have been
> moved to the deleted tables), the service sends a PurgePathRequest to remove
> the directory itself from the deletedDirectoryTable.
> 3. Block Reclamation (KeyDeletingService)
> The KeyDeletingService is responsible for the physical reclamation of
> storage space occupied by files in the deletedTable:
> * SCM Coordination: It scans the deletedTable, extracts the block list
> (BlockGroups) for each file, and sends a request to the Storage Container
> Manager
> (SCM) to delete these blocks.
> * Wait for ACK: It waits for SCM to acknowledge that the block deletion
> intent has been persisted.
> * Trigger Purge: Once SCM acknowledges, the service initiates the final
> cleanup by submitting an OMKeyPurgeRequest.
> 4. Final Metadata Cleanup (OMKeyPurgeRequest)
> This is the terminal state for a file's metadata:
> * Purging: The OMKeyPurgeRequest removes the entries from the deletedTable.
> * Snapshot Handling: If snapshots are involved, it may move keys to a
> "previous" snapshot's deleted table instead of purging them, ensuring data
> consistency across the snapshot chain.
> Summary Table Flow
>
> ┌──────┬────────────┬───────────────────────┬───────────────────────┬──────────────────────────────────────────────┐
> │ Step │ Entity │ Source Table │ Destination Table │
> Handled By │
>
> ├──────┼────────────┼───────────────────────┼───────────────────────┼──────────────────────────────────────────────┤
> │ 1 │ File │ fileTable │ deletedTable │
> OMKeyDeleteRequest │
> │ 1 │ Directory │ directoryTable │ deletedDirectoryTable │
> OMKeyDeleteRequest (FSO) │
> │ 2 │ Child File │ fileTable │ deletedTable │
> DirectoryDeletingService │
> │ 2 │ Child Dir │ directoryTable │ deletedDirectoryTable │
> DirectoryDeletingService │
> │ 3 │ Final File │ deletedTable │ Removed │
> KeyDeletingService -> OMKeyPurgeRequest │
> │ 4 │ Final Dir │ deletedDirectoryTable │ Removed │
> DirectoryDeletingService -> PurgePathRequest │
>
> └──────┴────────────┴───────────────────────┴───────────────────────┴──────────────────────────────────────────────┘
> ---
> SCM:
> ✦ The block reclamation process in SCM (Storage Container Manager) is an
> asynchronous, transaction-based workflow that ensures data is physically
> deleted
> from Datanodes while maintaining consistency across replicas.
> Here is the step-by-step implementation of how SCM handles block
> reclamation:
> 1. Transaction Persistence (DeletedBlockLog)
> When SCM receives a deleteKeyBlocks request from the Ozone Manager (OM), it
> doesn't delete the blocks immediately. Instead, it creates a durable record:
> * Grouping: SCM groups the blocks by their Container ID.
> * Logging: It creates a DeletedBlocksTransaction for each container. Each
> transaction is assigned a unique Transaction ID (TXID).
> * Storage: These transactions are persisted in the SCM metadata database
> in the DeletedBlocksTXTable. This ensures that even if SCM restarts, it won't
> "forget" to delete these blocks.
> 2. Throttled Command Dispatch (SCMBlockDeletingService)
> A background service called SCMBlockDeletingService periodically scans the
> DeletedBlocksTXTable to move the process forward:
> * Node Selection: It identifies healthy Datanodes that have room in their
> command queues (to avoid overwhelming them).
> * Transaction Matching: For each pending transaction, it identifies which
> Datanodes hold a replica of that container.
> * Command Creation: It bundles multiple transactions into a
> DeleteBlocksCommand for a specific Datanode.
> * Dispatch: The command is sent to the Datanode, typically as a payload in
> a heartbeat response.
> 3. Physical Deletion (Datanode Side)
> Once a Datanode receives the DeleteBlocksCommand:
> * It iterates through the transactions and physically deletes the chunks
> associated with the listed Block IDs from its local disks.
> * It records the success or failure of each transaction.
> 4. Acknowledgement and Final Cleanup
> SCM needs to ensure that all replicas of a block are deleted before it
> considers the task complete:
> * ACK Reporting: The Datanode sends back a ContainerBlocksDeletionACKProto
> (via DeleteBlockStatus report) to SCM.
> * Tracking: SCM’s SCMDeletedBlockTransactionStatusManager tracks which
> Datanodes have acknowledged which TXIDs.
> * Removal: Only after every replica (e.g., all 3 nodes in a Ratis
> pipeline) has successfully acknowledged the deletion does SCM finally remove
> the
> transaction from the DeletedBlocksTXTable.
> Why this architecture?
> * Fault Tolerance: By using a persistent log, SCM ensures that deletions
> happen eventually even if nodes are temporarily down.
> * Consistency: SCM won't stop retrying until every single replica is
> confirmed deleted, preventing "orphan" blocks from wasting space.
> * Performance: Batching multiple block deletions into a single transaction
> and command reduces RPC overhead between OM, SCM, and Datanodes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]