swamirishi commented on PR #10876: URL: https://github.com/apache/ozone/pull/10876#issuecomment-5101357018
> > > Is there some bulk delete in the buckets? > > > > > > @swamirishi Yes, we have the S3 lifecycle configuration service ([HDDS-8342](https://issues.apache.org/jira/browse/HDDS-8342)) that generates a few millions deletions in a very short time. In LEGACY bucket `getFileStatus` can trigger a RocksDB seek that is done under a bucket lock, if there are a lot of tombstones, then the lock will be held for a long time which can block Ratis applier and cause the OM write to be stuck. We have resolved it in #9932 by moving the createFakeDirIfShould outside the lock, but new patches can easily fall to the same trap since some people might thing that RocksDB seek behaves like a point query rather than a range query. > > > We can do something similar for the delete as well and add delete range tombstone as well. > > > > > > The lifecycle configuration service cannot use deleteRange (unlike DirectoryDeletingService since AFAIK DirectoryDeletingService only handles orphan directories and no keys will be created under the directory anymore) since another key can be created in the deleteRange which can cause valid keys to be inadvertently deleted, causing data loss. > > No that should not happen as long as you decide to do the iteration and delete range inside validateAndUpdateCache method of > > https://github.com/apache/ozone/blob/daef2f19c2946ad8e909323f5956b1884bba64fe/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysDeleteRequest.java#L117 > > > We can sort the keys to be deleted and initialize a rocksdb iterator for key table and do delete range for contiguous keys to be deleted in the iterator. Basically have a 2 pointer iteration b/w the keys to be deleted and the rocksdb iterator this would figure out the contiguous keys to be deleted. If the million of keys are contiguous it would optimize the number of tombstones we create by a huge magnitude. Basically I would propose to change this unoptimal loop which does individual seeks and get for each delete key which can be very unoptimal in every case having a rocksdb iterator would be way more optimal. > https://github.com/apache/ozone/blob/daef2f19c2946ad8e909323f5956b1884bba64fe/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysDeleteRequest.java#L187-L192 I am assuming you guys are using the bulk delete api to delete since I don't think it would be possible to do millions of deletes when deleting individual keys anyhow because of our om ratis constraints. > > > > Do you guys have ACL bloat problem? > > > > > > We previously had one, but we already resolved it so that now a single key usually only have 1 ACL only. Our custom authorizer allows us to have a stable ACL per key. > > But each OmKeyInfo can be large due to a lot of blocks (for example, if a key is uploaded using MPU with very small parts), so the risk is still there unless we make OmKeyInfo to not contain any unbounded list (i.e. ACLs and Blocks). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
