swamirishi commented on PR #10876: URL: https://github.com/apache/ozone/pull/10876#issuecomment-5100287357
> > Probably a good idea to revisit the built-in mechanisms rather than reinvent the wheels. > > +1, need to revisit RocksDB as a whole. > > > this is great writeup. Has this been tested in your prod environment? If so let's merge it asap. > > Not yet since we need to reproduce it in our test cluster. We have previously run some stress test in our test cluster to check the risk of the compaction service, but the issue is not triggered during the test so not sure how hard it is to reproduce it in our test cluster. > > Currently, we decided to disable compaction service for the foreseeable future until we understand the RocksDB enough to be confident to enable it, the risk is too high. > > > We won't have the tombstone and slow rocksdb iterator problem to begin with. > > @swamirishi Seems your patches are related to deleteRange and snapshot. Our cluster does encounter tombstone and slow rocksdb iterator caused by holding the RocksDB iterator during RocksDB seek in the LEGACY bucket , see #9932 . Not sure how the attached patches will help prevent our case (we don't use Ozone snapshot in production). I am sorry @ivandika3 I redundantly added links to the same patches twice and forgot to add this. The following [PR](https://github.com/apache/ozone/pull/9423) solves slow iterator problem for FSO. For OBS I would love to understand how the keys are being deleted in production. Is there some bulk delete in the buckets? We can do something similar for the delete as well and add delete range tombstone as well. BTW can you check what is the size of each OMKeyInfo in your keyTable/FileTable/DirectoryTable. Do you guys have ACL bloat problem? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
