[ 
https://issues.apache.org/jira/browse/HDDS-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDDS-15027:
-----------------------------------
    Description: 
A very common question users ask is: when a user issues a delete request, how 
does it eventually get deleted? Due to Ozone's distributed nature, it can be 
tricky to understand all the moving parts, so this page gives a holistic 
overview of delete.

Trace every part of a delete request from beginning to end. This includes:

- Client getting encryption keys
- Client calling OM to delete the key
- OM validating client's Kerberos principal
- OM checking permissions (Ranger or Native ACLs) - OMKeyDeleteRequest
- OM marking the key for deletion and removing it from the namespace 
- OM acking to the client
- OM sending blocks to delete to SCM
- SCM sending blocks to delete to the Datanode
- Datanode removes data

Note that delete works the same regardless of replication type. Also document 
timing of background services and their batch size to estimate the rate of 
deletion.

---
Inside OM:

✦ The deletion process in Apache Ozone is a multi-phase, asynchronous workflow 
designed to handle massive scale and File System Optimized (FSO) structures
  efficiently.

  Based on the provided source files, here is the lifecycle of a deleted file 
or directory as it moves through the Ozone Manager (OM) tables:

  1. Initial Deletion (Synchronous Phase)
  When a client issues a delete request, OMKeyDeleteRequest handles the 
transaction:
   * For Files: The entry is removed from the fileTable (or keyTable in non-FSO 
buckets) and a new entry is added to the deletedTable.
   * For Directories (FSO): The directory is removed from the directoryTable 
and moved to the deletedDirectoryTable.
   * Quota Update: The bucket's used bytes and namespace quota are updated at 
this stage to reflect the "logical" deletion.

  2. Directory Expansion (DirectoryDeletingService)
  For FSO buckets, deleting a directory does not immediately move all its 
children. Instead, the DirectoryDeletingService processes them asynchronously:
   * Scanning: It scans the deletedDirectoryTable for pending deletions.
   * Recursive Movement:
       * Sub-files: It finds all files within the deleted directory and moves 
them from the fileTable to the deletedTable.
       * Sub-directories: It finds sub-directories and moves them from the 
directoryTable to the deletedDirectoryTable so they can be processed in the next
         iteration.
   * Purging the Directory: Once a directory is empty (all children have been 
moved to the deleted tables), the service sends a PurgePathRequest to remove
     the directory itself from the deletedDirectoryTable.

  3. Block Reclamation (KeyDeletingService)
  The KeyDeletingService is responsible for the physical reclamation of storage 
space occupied by files in the deletedTable:
   * SCM Coordination: It scans the deletedTable, extracts the block list 
(BlockGroups) for each file, and sends a request to the Storage Container 
Manager
     (SCM) to delete these blocks.
   * Wait for ACK: It waits for SCM to acknowledge that the block deletion 
intent has been persisted.
   * Trigger Purge: Once SCM acknowledges, the service initiates the final 
cleanup by submitting an OMKeyPurgeRequest.

  4. Final Metadata Cleanup (OMKeyPurgeRequest)
  This is the terminal state for a file's metadata:
   * Purging: The OMKeyPurgeRequest removes the entries from the deletedTable.
   * Snapshot Handling: If snapshots are involved, it may move keys to a 
"previous" snapshot's deleted table instead of purging them, ensuring data
     consistency across the snapshot chain.

  Summary Table Flow

  
┌──────┬────────────┬───────────────────────┬───────────────────────┬──────────────────────────────────────────────┐
  │ Step │ Entity     │ Source Table          │ Destination Table     │ Handled 
By                                   │
  
├──────┼────────────┼───────────────────────┼───────────────────────┼──────────────────────────────────────────────┤
  │ 1    │ File       │ fileTable             │ deletedTable          │ 
OMKeyDeleteRequest                           │
  │ 1    │ Directory  │ directoryTable        │ deletedDirectoryTable │ 
OMKeyDeleteRequest (FSO)                     │
  │ 2    │ Child File │ fileTable             │ deletedTable          │ 
DirectoryDeletingService                     │
  │ 2    │ Child Dir  │ directoryTable        │ deletedDirectoryTable │ 
DirectoryDeletingService                     │
  │ 3    │ Final File │ deletedTable          │ Removed               │ 
KeyDeletingService -> OMKeyPurgeRequest      │
  │ 4    │ Final Dir  │ deletedDirectoryTable │ Removed               │ 
DirectoryDeletingService -> PurgePathRequest │
  
└──────┴────────────┴───────────────────────┴───────────────────────┴──────────────────────────────────────────────┘

---

SCM:

✦ The block reclamation process in SCM (Storage Container Manager) is an 
asynchronous, transaction-based workflow that ensures data is physically deleted
  from Datanodes while maintaining consistency across replicas.

  Here is the step-by-step implementation of how SCM handles block reclamation:

  1. Transaction Persistence (DeletedBlockLog)
  When SCM receives a deleteKeyBlocks request from the Ozone Manager (OM), it 
doesn't delete the blocks immediately. Instead, it creates a durable record:
   * Grouping: SCM groups the blocks by their Container ID.
   * Logging: It creates a DeletedBlocksTransaction for each container. Each 
transaction is assigned a unique Transaction ID (TXID).
   * Storage: These transactions are persisted in the SCM metadata database in 
the DeletedBlocksTXTable. This ensures that even if SCM restarts, it won't
     "forget" to delete these blocks.

  2. Throttled Command Dispatch (SCMBlockDeletingService)
  A background service called SCMBlockDeletingService periodically scans the 
DeletedBlocksTXTable to move the process forward:
   * Node Selection: It identifies healthy Datanodes that have room in their 
command queues (to avoid overwhelming them).
   * Transaction Matching: For each pending transaction, it identifies which 
Datanodes hold a replica of that container.
   * Command Creation: It bundles multiple transactions into a 
DeleteBlocksCommand for a specific Datanode.
   * Dispatch: The command is sent to the Datanode, typically as a payload in a 
heartbeat response.

  3. Physical Deletion (Datanode Side)
  Once a Datanode receives the DeleteBlocksCommand:
   * It iterates through the transactions and physically deletes the chunks 
associated with the listed Block IDs from its local disks.
   * It records the success or failure of each transaction.

  4. Acknowledgement and Final Cleanup
  SCM needs to ensure that all replicas of a block are deleted before it 
considers the task complete:
   * ACK Reporting: The Datanode sends back a ContainerBlocksDeletionACKProto 
(via DeleteBlockStatus report) to SCM.
   * Tracking: SCM’s SCMDeletedBlockTransactionStatusManager tracks which 
Datanodes have acknowledged which TXIDs.
   * Removal: Only after every replica (e.g., all 3 nodes in a Ratis pipeline) 
has successfully acknowledged the deletion does SCM finally remove the
     transaction from the DeletedBlocksTXTable.

  Why this architecture?
   * Fault Tolerance: By using a persistent log, SCM ensures that deletions 
happen eventually even if nodes are temporarily down.
   * Consistency: SCM won't stop retrying until every single replica is 
confirmed deleted, preventing "orphan" blocks from wasting space.
   * Performance: Batching multiple block deletions into a single transaction 
and command reduces RPC overhead between OM, SCM, and Datanodes.


  was:
A very common question users ask is: when a user issues a delete request, how 
does it eventually get deleted? Due to Ozone's distributed nature, it can be 
tricky to understand all the moving parts, so this page gives a holistic 
overview of delete.

Trace every part of a delete request from beginning to end. This includes:

- Client getting encryption keys
- Client calling OM to delete the key
- OM validating client's Kerberos principal
- OM checking permissions (Ranger or Native ACLs) - OMKeyDeleteRequest
- OM marking the key for deletion and removing it from the namespace 
- OM acking to the client
- OM sending blocks to delete to SCM
- SCM sending blocks to delete to the Datanode
- Datanode removes data

Note that delete works the same regardless of replication type. Also document 
timing of background services and their batch size to estimate the rate of 
deletion.

---
Inside OM:

✦ The deletion process in Apache Ozone is a multi-phase, asynchronous workflow 
designed to handle massive scale and File System Optimized (FSO) structures
  efficiently.

  Based on the provided source files, here is the lifecycle of a deleted file 
or directory as it moves through the Ozone Manager (OM) tables:

  1. Initial Deletion (Synchronous Phase)
  When a client issues a delete request, OMKeyDeleteRequest handles the 
transaction:
   * For Files: The entry is removed from the fileTable (or keyTable in non-FSO 
buckets) and a new entry is added to the deletedTable.
   * For Directories (FSO): The directory is removed from the directoryTable 
and moved to the deletedDirectoryTable.
   * Quota Update: The bucket's used bytes and namespace quota are updated at 
this stage to reflect the "logical" deletion.

  2. Directory Expansion (DirectoryDeletingService)
  For FSO buckets, deleting a directory does not immediately move all its 
children. Instead, the DirectoryDeletingService processes them asynchronously:
   * Scanning: It scans the deletedDirectoryTable for pending deletions.
   * Recursive Movement:
       * Sub-files: It finds all files within the deleted directory and moves 
them from the fileTable to the deletedTable.
       * Sub-directories: It finds sub-directories and moves them from the 
directoryTable to the deletedDirectoryTable so they can be processed in the next
         iteration.
   * Purging the Directory: Once a directory is empty (all children have been 
moved to the deleted tables), the service sends a PurgePathRequest to remove
     the directory itself from the deletedDirectoryTable.

  3. Block Reclamation (KeyDeletingService)
  The KeyDeletingService is responsible for the physical reclamation of storage 
space occupied by files in the deletedTable:
   * SCM Coordination: It scans the deletedTable, extracts the block list 
(BlockGroups) for each file, and sends a request to the Storage Container 
Manager
     (SCM) to delete these blocks.
   * Wait for ACK: It waits for SCM to acknowledge that the block deletion 
intent has been persisted.
   * Trigger Purge: Once SCM acknowledges, the service initiates the final 
cleanup by submitting an OMKeyPurgeRequest.

  4. Final Metadata Cleanup (OMKeyPurgeRequest)
  This is the terminal state for a file's metadata:
   * Purging: The OMKeyPurgeRequest removes the entries from the deletedTable.
   * Snapshot Handling: If snapshots are involved, it may move keys to a 
"previous" snapshot's deleted table instead of purging them, ensuring data
     consistency across the snapshot chain.

  Summary Table Flow

  
┌──────┬────────────┬───────────────────────┬───────────────────────┬──────────────────────────────────────────────┐
  │ Step │ Entity     │ Source Table          │ Destination Table     │ Handled 
By                                   │
  
├──────┼────────────┼───────────────────────┼───────────────────────┼──────────────────────────────────────────────┤
  │ 1    │ File       │ fileTable             │ deletedTable          │ 
OMKeyDeleteRequest                           │
  │ 1    │ Directory  │ directoryTable        │ deletedDirectoryTable │ 
OMKeyDeleteRequest (FSO)                     │
  │ 2    │ Child File │ fileTable             │ deletedTable          │ 
DirectoryDeletingService                     │
  │ 2    │ Child Dir  │ directoryTable        │ deletedDirectoryTable │ 
DirectoryDeletingService                     │
  │ 3    │ Final File │ deletedTable          │ Removed               │ 
KeyDeletingService -> OMKeyPurgeRequest      │
  │ 4    │ Final Dir  │ deletedDirectoryTable │ Removed               │ 
DirectoryDeletingService -> PurgePathRequest │
  
└──────┴────────────┴───────────────────────┴───────────────────────┴──────────────────────────────────────────────┘



> [Docs] [System Internals] Implementation of Delete Operations
> -------------------------------------------------------------
>
>                 Key: HDDS-15027
>                 URL: https://issues.apache.org/jira/browse/HDDS-15027
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Wei-Chiu Chuang
>            Priority: Major
>
> A very common question users ask is: when a user issues a delete request, how 
> does it eventually get deleted? Due to Ozone's distributed nature, it can be 
> tricky to understand all the moving parts, so this page gives a holistic 
> overview of delete.
> Trace every part of a delete request from beginning to end. This includes:
> - Client getting encryption keys
> - Client calling OM to delete the key
> - OM validating client's Kerberos principal
> - OM checking permissions (Ranger or Native ACLs) - OMKeyDeleteRequest
> - OM marking the key for deletion and removing it from the namespace 
> - OM acking to the client
> - OM sending blocks to delete to SCM
> - SCM sending blocks to delete to the Datanode
> - Datanode removes data
> Note that delete works the same regardless of replication type. Also document 
> timing of background services and their batch size to estimate the rate of 
> deletion.
> ---
> Inside OM:
> ✦ The deletion process in Apache Ozone is a multi-phase, asynchronous 
> workflow designed to handle massive scale and File System Optimized (FSO) 
> structures
>   efficiently.
>   Based on the provided source files, here is the lifecycle of a deleted file 
> or directory as it moves through the Ozone Manager (OM) tables:
>   1. Initial Deletion (Synchronous Phase)
>   When a client issues a delete request, OMKeyDeleteRequest handles the 
> transaction:
>    * For Files: The entry is removed from the fileTable (or keyTable in 
> non-FSO buckets) and a new entry is added to the deletedTable.
>    * For Directories (FSO): The directory is removed from the directoryTable 
> and moved to the deletedDirectoryTable.
>    * Quota Update: The bucket's used bytes and namespace quota are updated at 
> this stage to reflect the "logical" deletion.
>   2. Directory Expansion (DirectoryDeletingService)
>   For FSO buckets, deleting a directory does not immediately move all its 
> children. Instead, the DirectoryDeletingService processes them asynchronously:
>    * Scanning: It scans the deletedDirectoryTable for pending deletions.
>    * Recursive Movement:
>        * Sub-files: It finds all files within the deleted directory and moves 
> them from the fileTable to the deletedTable.
>        * Sub-directories: It finds sub-directories and moves them from the 
> directoryTable to the deletedDirectoryTable so they can be processed in the 
> next
>          iteration.
>    * Purging the Directory: Once a directory is empty (all children have been 
> moved to the deleted tables), the service sends a PurgePathRequest to remove
>      the directory itself from the deletedDirectoryTable.
>   3. Block Reclamation (KeyDeletingService)
>   The KeyDeletingService is responsible for the physical reclamation of 
> storage space occupied by files in the deletedTable:
>    * SCM Coordination: It scans the deletedTable, extracts the block list 
> (BlockGroups) for each file, and sends a request to the Storage Container 
> Manager
>      (SCM) to delete these blocks.
>    * Wait for ACK: It waits for SCM to acknowledge that the block deletion 
> intent has been persisted.
>    * Trigger Purge: Once SCM acknowledges, the service initiates the final 
> cleanup by submitting an OMKeyPurgeRequest.
>   4. Final Metadata Cleanup (OMKeyPurgeRequest)
>   This is the terminal state for a file's metadata:
>    * Purging: The OMKeyPurgeRequest removes the entries from the deletedTable.
>    * Snapshot Handling: If snapshots are involved, it may move keys to a 
> "previous" snapshot's deleted table instead of purging them, ensuring data
>      consistency across the snapshot chain.
>   Summary Table Flow
>   
> ┌──────┬────────────┬───────────────────────┬───────────────────────┬──────────────────────────────────────────────┐
>   │ Step │ Entity     │ Source Table          │ Destination Table     │ 
> Handled By                                   │
>   
> ├──────┼────────────┼───────────────────────┼───────────────────────┼──────────────────────────────────────────────┤
>   │ 1    │ File       │ fileTable             │ deletedTable          │ 
> OMKeyDeleteRequest                           │
>   │ 1    │ Directory  │ directoryTable        │ deletedDirectoryTable │ 
> OMKeyDeleteRequest (FSO)                     │
>   │ 2    │ Child File │ fileTable             │ deletedTable          │ 
> DirectoryDeletingService                     │
>   │ 2    │ Child Dir  │ directoryTable        │ deletedDirectoryTable │ 
> DirectoryDeletingService                     │
>   │ 3    │ Final File │ deletedTable          │ Removed               │ 
> KeyDeletingService -> OMKeyPurgeRequest      │
>   │ 4    │ Final Dir  │ deletedDirectoryTable │ Removed               │ 
> DirectoryDeletingService -> PurgePathRequest │
>   
> └──────┴────────────┴───────────────────────┴───────────────────────┴──────────────────────────────────────────────┘
> ---
> SCM:
> ✦ The block reclamation process in SCM (Storage Container Manager) is an 
> asynchronous, transaction-based workflow that ensures data is physically 
> deleted
>   from Datanodes while maintaining consistency across replicas.
>   Here is the step-by-step implementation of how SCM handles block 
> reclamation:
>   1. Transaction Persistence (DeletedBlockLog)
>   When SCM receives a deleteKeyBlocks request from the Ozone Manager (OM), it 
> doesn't delete the blocks immediately. Instead, it creates a durable record:
>    * Grouping: SCM groups the blocks by their Container ID.
>    * Logging: It creates a DeletedBlocksTransaction for each container. Each 
> transaction is assigned a unique Transaction ID (TXID).
>    * Storage: These transactions are persisted in the SCM metadata database 
> in the DeletedBlocksTXTable. This ensures that even if SCM restarts, it won't
>      "forget" to delete these blocks.
>   2. Throttled Command Dispatch (SCMBlockDeletingService)
>   A background service called SCMBlockDeletingService periodically scans the 
> DeletedBlocksTXTable to move the process forward:
>    * Node Selection: It identifies healthy Datanodes that have room in their 
> command queues (to avoid overwhelming them).
>    * Transaction Matching: For each pending transaction, it identifies which 
> Datanodes hold a replica of that container.
>    * Command Creation: It bundles multiple transactions into a 
> DeleteBlocksCommand for a specific Datanode.
>    * Dispatch: The command is sent to the Datanode, typically as a payload in 
> a heartbeat response.
>   3. Physical Deletion (Datanode Side)
>   Once a Datanode receives the DeleteBlocksCommand:
>    * It iterates through the transactions and physically deletes the chunks 
> associated with the listed Block IDs from its local disks.
>    * It records the success or failure of each transaction.
>   4. Acknowledgement and Final Cleanup
>   SCM needs to ensure that all replicas of a block are deleted before it 
> considers the task complete:
>    * ACK Reporting: The Datanode sends back a ContainerBlocksDeletionACKProto 
> (via DeleteBlockStatus report) to SCM.
>    * Tracking: SCM’s SCMDeletedBlockTransactionStatusManager tracks which 
> Datanodes have acknowledged which TXIDs.
>    * Removal: Only after every replica (e.g., all 3 nodes in a Ratis 
> pipeline) has successfully acknowledged the deletion does SCM finally remove 
> the
>      transaction from the DeletedBlocksTXTable.
>   Why this architecture?
>    * Fault Tolerance: By using a persistent log, SCM ensures that deletions 
> happen eventually even if nodes are temporarily down.
>    * Consistency: SCM won't stop retrying until every single replica is 
> confirmed deleted, preventing "orphan" blocks from wasting space.
>    * Performance: Batching multiple block deletions into a single transaction 
> and command reduces RPC overhead between OM, SCM, and Datanodes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to