sodonnel commented on PR #10082: URL: https://github.com/apache/ozone/pull/10082#issuecomment-4269541184
There are a few things to think about in this change. First for a 3-2 container, if there are 3 node being decommissions, reconstruction isn't possible to recover all the decommissioning copies. Ideally, there would be a task scheduled to reconstruct all 3 at a time, but with 3 out that won't work and it will have to pick just 2 to recover and leave the other one for now. Or it could schedule two tasks at the same time for the container. I know that within Cloudera we had a case where a cluster was showing slow decommission progress on EC containers, but I am not sure if we investigated if the datanode was being kept sufficiently busy. Ie, did it have enough replications scheduled so that its queue did not run empty between each heartbeat. Did it have enough worker threads to keep all the disks busy etc. I think that is important to understand before adding this complexity, but also if the DNs are not being kept busy enough, then it will be the same problem for reconstruction commands too. At the moment, the logic in the ECUnderReplicationHandler deals with missing indexes, then decommissioning indexes. If it gets a "node over utilised" exception dealing with the decommissioning indexes, the simplest change it to probably mask off the decommissioning indexes up to the parity number, add them to an excluded list and then call `processingMissingIndexes` again up to some limit. There should already be a global concurrency limit in RM, so a new one should not be needed. I also agree with @adoroszlai that this change should be at least 2 or 3 separate changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
