sodonnel commented on PR #10082:
URL: https://github.com/apache/ozone/pull/10082#issuecomment-4269541184

   There are a few things to think about in this change.
   
   First for a 3-2 container, if there are 3 node being decommissions, 
reconstruction isn't possible to recover all the decommissioning copies. 
Ideally, there would be a task scheduled to reconstruct all 3 at a time, but 
with 3 out that won't work and it will have to pick just 2 to recover and leave 
the other one for now. Or it could schedule two tasks at the same time for the 
container.
   
   I know that within Cloudera we had a case where a cluster was showing slow 
decommission progress on EC containers, but I am not sure if we investigated if 
the datanode was being kept sufficiently busy. Ie, did it have enough 
replications scheduled so that its queue did not run empty between each 
heartbeat. Did it have enough worker threads to keep all the disks busy etc. I 
think that is important to understand before adding this complexity, but also 
if the DNs are not being kept busy enough, then it will be the same problem for 
reconstruction commands too.
   
   At the moment, the logic in the ECUnderReplicationHandler deals with missing 
indexes, then decommissioning indexes. If it gets a "node over utilised" 
exception dealing with the decommissioning indexes, the simplest change it to 
probably mask off the decommissioning indexes up to the parity number, add them 
to an excluded list and then call `processingMissingIndexes` again up to some 
limit.
   
   There should already be a global concurrency limit in RM, so a new one 
should not be needed.
   
   I also agree with @adoroszlai that this change should be at least 2 or 3 
separate changes.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to