jojochuang opened a new pull request, #8570:
URL: https://github.com/apache/ozone/pull/8570
## What changes were proposed in this pull request?
With implementation of ReclaimableKeyFilter & ReclaimableDirectoryFilter
both snapshot directory deep cleaning and AOS DirectoryDeletingService can be
unified which will simplify the entire implementation rather than having
multiple implementations for DirectoryDeletingService.
Algorithm in play:
1. The directory deleting service would be iterate through all the entries
in the deletedDirectoryTable for a store(AOS and snapshot). One thread
processing deleted directory entry for a particular store.
2. This thread would further initialize a DeletedDirectorySupplier which is
an implementation for thread safe table iterator(We can eventually use a
TableSpliterator after #8203 gets merged).
3. Each thread here would be iterating through the deletedDirectoryTable
independently and would submit purgeDirectoriesRequests where the logic for
directoryPurge is as follows:
- Check if the deletedDirectory object from
deletedDirectoryTable can be reclaimed as in check if the reference for
directory object exists in the previous directory.
- If the directory exists in the previous snapshot then we
need not expand the sub files inside the files[Number of files would be a lot
generally (number of files >>> number of directories)] and we would just expand
all the subdirectories under the directory(this would help in the recursively
iterating on the sub files in the next iteration instead of traversing again
from root all over). Now we cannot purge the directories as a reference exists
we would just move all sub directories present in the AOS directory table and
files which can be reclaimed i.e. files which are not present in the previous
snapshot's fileTable by iterating through the sub file list based on deleted
directory prefix in the AOS file table and move it to the deleted space of the
store[Check #8509].
- If the directory doesn't exist in the previous snapshot we
have to blindly remove all entries[sub files & sub directories] inside this
directory irrespective whether it can be reclaimed or not and purge the
directory if all the entries are going to be moved from AOS to the specific
store's deleted space of the store.
4. Now once all threads have processed if none of the threads have been able
to move any entry from the key space to deleted space(All keys already have
been reclaimed in the previous run) we can go ahead update the exclusive size
of the previous snapshot and the deep clean flag of the snapshot.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13034
## How was this patch tested?
Existing unit tests and additional assertions added on existing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]