jojochuang opened a new pull request, #8570:
URL: https://github.com/apache/ozone/pull/8570

   ## What changes were proposed in this pull request?
   With implementation of ReclaimableKeyFilter & ReclaimableDirectoryFilter 
both snapshot directory deep cleaning and AOS DirectoryDeletingService can be 
unified which will simplify the entire implementation rather than having 
multiple implementations for DirectoryDeletingService. 
   Algorithm in play:
   
   1. The directory deleting service would be iterate through all the entries 
in the deletedDirectoryTable for a store(AOS and snapshot). One thread 
processing deleted directory entry for a particular store.
   2. This thread would further initialize a DeletedDirectorySupplier which is 
an implementation for thread safe table iterator(We can eventually use a 
TableSpliterator after #8203 gets merged).
   3. Each thread here would be iterating through the deletedDirectoryTable 
independently and would submit purgeDirectoriesRequests where the logic for 
directoryPurge is as follows:
                  -  Check if the deletedDirectory object from 
deletedDirectoryTable can be reclaimed as in check if the reference       for 
directory object exists in the previous directory.
                  - If the directory exists in the previous snapshot then we 
need not expand the sub files inside the files[Number of files would be a lot 
generally (number of files >>> number of directories)] and we would just expand 
all the subdirectories under the directory(this would help in the recursively 
iterating on the sub files in the next iteration instead of traversing again 
from root all over). Now we cannot purge the directories as a reference exists 
we would just move all sub directories present in the AOS directory table and 
files which can be reclaimed i.e. files which are not present in the previous 
snapshot's fileTable by iterating through the sub file list based on deleted 
directory prefix in the AOS file table and move it to the deleted space of the 
store[Check #8509].
                  - If the directory doesn't exist in the previous snapshot we 
have to blindly remove all entries[sub files & sub directories] inside this 
directory irrespective whether it can be reclaimed or not and purge the 
directory if all the entries are going to be moved from AOS to the specific 
store's deleted space of the store.
   
   4. Now once all threads have processed if none of the threads have been able 
to move any entry from the key space to deleted space(All keys already have 
been reclaimed in the previous run) we can go ahead update the exclusive size 
of the previous snapshot and the deep clean flag of the snapshot.               
                 
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-13034
   
   ## How was this patch tested?
   Existing unit tests and additional assertions added on existing tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to