ashishkumar50 commented on PR #11564: URL: https://github.com/apache/ozone/pull/11564#issuecomment-6261265399
> What is the driver for this change? Ie, why does OM send duplicates in the first place? If we want to skip them on SCM then the approach here makes sense, but a 1M entry cache will add about 60MB to the SCM heap. This isn't that much in the scope of a 30+GB heap, but its still a fair chunk just for avoiding duplicate deletes. @sodonnel Thanks for looking into this. There are multiple instance we have seen OM has sent duplicates to SCM. Once such instance is [here](https://github.com/apache/ozone/pull/6099/changes#diff-8e436163a13fd1abf438443ad0386a9c46f4a30ec0e666ff1b2703fcbe9a2657:~:text=blocksForKeyDelete.add(keyBlocks)%3B), where duplicate keys are sent thousands of time and caused very high SCM DB size which lead to unnesesarry overhead in SCM and caused very slow deletion. Even though these bugs are fixed in latest version but it is worth adding duplicate check in SCM to avoid any future or unknown problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
