[
https://issues.apache.org/jira/browse/HDDS-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-15027:
-----------------------------------
Description:
A very common question users ask is: when a user issues a delete request, how
does it eventually get deleted? Due to Ozone's distributed nature, it can be
tricky to understand all the moving parts, so this page gives a holistic
overview of delete.
Trace every part of a delete request from beginning to end. This includes:
- Client getting encryption keys
- Client calling OM to delete the key
- OM validating client's Kerberos principal
- OM checking permissions (Ranger or Native ACLs) - OMKeyDeleteRequest
- OM marking the key for deletion and removing it from the namespace
- OM acking to the client
- OM sending blocks to delete to SCM
- SCM sending blocks to delete to the Datanode
- Datanode removes data
Note that delete works the same regardless of replication type. Also document
timing of background services and their batch size to estimate the rate of
deletion.
---
Inside OM:
✦ The deletion process in Apache Ozone is a multi-phase, asynchronous workflow
designed to handle massive scale and File System Optimized (FSO) structures
efficiently.
Based on the provided source files, here is the lifecycle of a deleted file
or directory as it moves through the Ozone Manager (OM) tables:
1. Initial Deletion (Synchronous Phase)
When a client issues a delete request, OMKeyDeleteRequest handles the
transaction:
* For Files: The entry is removed from the fileTable (or keyTable in non-FSO
buckets) and a new entry is added to the deletedTable.
* For Directories (FSO): The directory is removed from the directoryTable
and moved to the deletedDirectoryTable.
* Quota Update: The bucket's used bytes and namespace quota are updated at
this stage to reflect the "logical" deletion.
2. Directory Expansion (DirectoryDeletingService)
For FSO buckets, deleting a directory does not immediately move all its
children. Instead, the DirectoryDeletingService processes them asynchronously:
* Scanning: It scans the deletedDirectoryTable for pending deletions.
* Recursive Movement:
* Sub-files: It finds all files within the deleted directory and moves
them from the fileTable to the deletedTable.
* Sub-directories: It finds sub-directories and moves them from the
directoryTable to the deletedDirectoryTable so they can be processed in the next
iteration.
* Purging the Directory: Once a directory is empty (all children have been
moved to the deleted tables), the service sends a PurgePathRequest to remove
the directory itself from the deletedDirectoryTable.
3. Block Reclamation (KeyDeletingService)
The KeyDeletingService is responsible for the physical reclamation of storage
space occupied by files in the deletedTable:
* SCM Coordination: It scans the deletedTable, extracts the block list
(BlockGroups) for each file, and sends a request to the Storage Container
Manager
(SCM) to delete these blocks.
* Wait for ACK: It waits for SCM to acknowledge that the block deletion
intent has been persisted.
* Trigger Purge: Once SCM acknowledges, the service initiates the final
cleanup by submitting an OMKeyPurgeRequest.
4. Final Metadata Cleanup (OMKeyPurgeRequest)
This is the terminal state for a file's metadata:
* Purging: The OMKeyPurgeRequest removes the entries from the deletedTable.
* Snapshot Handling: If snapshots are involved, it may move keys to a
"previous" snapshot's deleted table instead of purging them, ensuring data
consistency across the snapshot chain.
Summary Table Flow
┌──────┬────────────┬───────────────────────┬───────────────────────┬──────────────────────────────────────────────┐
│ Step │ Entity │ Source Table │ Destination Table │ Handled
By │
├──────┼────────────┼───────────────────────┼───────────────────────┼──────────────────────────────────────────────┤
│ 1 │ File │ fileTable │ deletedTable │
OMKeyDeleteRequest │
│ 1 │ Directory │ directoryTable │ deletedDirectoryTable │
OMKeyDeleteRequest (FSO) │
│ 2 │ Child File │ fileTable │ deletedTable │
DirectoryDeletingService │
│ 2 │ Child Dir │ directoryTable │ deletedDirectoryTable │
DirectoryDeletingService │
│ 3 │ Final File │ deletedTable │ Removed │
KeyDeletingService -> OMKeyPurgeRequest │
│ 4 │ Final Dir │ deletedDirectoryTable │ Removed │
DirectoryDeletingService -> PurgePathRequest │
└──────┴────────────┴───────────────────────┴───────────────────────┴──────────────────────────────────────────────┘
was:
A very common question users ask is: when a user issues a delete request, how
does it eventually get deleted? Due to Ozone's distributed nature, it can be
tricky to understand all the moving parts, so this page gives a holistic
overview of delete.
Trace every part of a delete request from beginning to end. This includes:
- Client getting encryption keys
- Client calling OM to delete the key
- OM validating client's Kerberos principal
- OM checking permissions (Ranger or Native ACLs) - OMKeyDeleteRequest
- OM marking the key for deletion and removing it from the namespace
- OM acking to the client
- OM sending blocks to delete to SCM
- SCM sending blocks to delete to the Datanode
- Datanode removes data
Note that delete works the same regardless of replication type. Also document
timing of background services and their batch size to estimate the rate of
deletion.
> [Docs] [System Internals] Implementation of Delete Operations
> -------------------------------------------------------------
>
> Key: HDDS-15027
> URL: https://issues.apache.org/jira/browse/HDDS-15027
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: documentation
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> A very common question users ask is: when a user issues a delete request, how
> does it eventually get deleted? Due to Ozone's distributed nature, it can be
> tricky to understand all the moving parts, so this page gives a holistic
> overview of delete.
> Trace every part of a delete request from beginning to end. This includes:
> - Client getting encryption keys
> - Client calling OM to delete the key
> - OM validating client's Kerberos principal
> - OM checking permissions (Ranger or Native ACLs) - OMKeyDeleteRequest
> - OM marking the key for deletion and removing it from the namespace
> - OM acking to the client
> - OM sending blocks to delete to SCM
> - SCM sending blocks to delete to the Datanode
> - Datanode removes data
> Note that delete works the same regardless of replication type. Also document
> timing of background services and their batch size to estimate the rate of
> deletion.
> ---
> Inside OM:
> ✦ The deletion process in Apache Ozone is a multi-phase, asynchronous
> workflow designed to handle massive scale and File System Optimized (FSO)
> structures
> efficiently.
> Based on the provided source files, here is the lifecycle of a deleted file
> or directory as it moves through the Ozone Manager (OM) tables:
> 1. Initial Deletion (Synchronous Phase)
> When a client issues a delete request, OMKeyDeleteRequest handles the
> transaction:
> * For Files: The entry is removed from the fileTable (or keyTable in
> non-FSO buckets) and a new entry is added to the deletedTable.
> * For Directories (FSO): The directory is removed from the directoryTable
> and moved to the deletedDirectoryTable.
> * Quota Update: The bucket's used bytes and namespace quota are updated at
> this stage to reflect the "logical" deletion.
> 2. Directory Expansion (DirectoryDeletingService)
> For FSO buckets, deleting a directory does not immediately move all its
> children. Instead, the DirectoryDeletingService processes them asynchronously:
> * Scanning: It scans the deletedDirectoryTable for pending deletions.
> * Recursive Movement:
> * Sub-files: It finds all files within the deleted directory and moves
> them from the fileTable to the deletedTable.
> * Sub-directories: It finds sub-directories and moves them from the
> directoryTable to the deletedDirectoryTable so they can be processed in the
> next
> iteration.
> * Purging the Directory: Once a directory is empty (all children have been
> moved to the deleted tables), the service sends a PurgePathRequest to remove
> the directory itself from the deletedDirectoryTable.
> 3. Block Reclamation (KeyDeletingService)
> The KeyDeletingService is responsible for the physical reclamation of
> storage space occupied by files in the deletedTable:
> * SCM Coordination: It scans the deletedTable, extracts the block list
> (BlockGroups) for each file, and sends a request to the Storage Container
> Manager
> (SCM) to delete these blocks.
> * Wait for ACK: It waits for SCM to acknowledge that the block deletion
> intent has been persisted.
> * Trigger Purge: Once SCM acknowledges, the service initiates the final
> cleanup by submitting an OMKeyPurgeRequest.
> 4. Final Metadata Cleanup (OMKeyPurgeRequest)
> This is the terminal state for a file's metadata:
> * Purging: The OMKeyPurgeRequest removes the entries from the deletedTable.
> * Snapshot Handling: If snapshots are involved, it may move keys to a
> "previous" snapshot's deleted table instead of purging them, ensuring data
> consistency across the snapshot chain.
> Summary Table Flow
>
> ┌──────┬────────────┬───────────────────────┬───────────────────────┬──────────────────────────────────────────────┐
> │ Step │ Entity │ Source Table │ Destination Table │
> Handled By │
>
> ├──────┼────────────┼───────────────────────┼───────────────────────┼──────────────────────────────────────────────┤
> │ 1 │ File │ fileTable │ deletedTable │
> OMKeyDeleteRequest │
> │ 1 │ Directory │ directoryTable │ deletedDirectoryTable │
> OMKeyDeleteRequest (FSO) │
> │ 2 │ Child File │ fileTable │ deletedTable │
> DirectoryDeletingService │
> │ 2 │ Child Dir │ directoryTable │ deletedDirectoryTable │
> DirectoryDeletingService │
> │ 3 │ Final File │ deletedTable │ Removed │
> KeyDeletingService -> OMKeyPurgeRequest │
> │ 4 │ Final Dir │ deletedDirectoryTable │ Removed │
> DirectoryDeletingService -> PurgePathRequest │
>
> └──────┴────────────┴───────────────────────┴───────────────────────┴──────────────────────────────────────────────┘
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
