sodonnel commented on PR #3610: URL: https://github.com/apache/ozone/pull/3610#issuecomment-1190394902
> 2 in ReplicationManager#processAll, we get two new instances of queue for both overReplicated and underReplicated tasks every time, and they are protected by a lock. there may be a case that the earlier generated task queue is replaced by a later generated task queue without completing all the tasks in the earlier one. for example, the interval of OverReplicatedProcessor is 10m, and the interval of RM is 2m. so in 10m, RM will produce 5 task queues for overreplicated containers, but OverReplicatedProcessor can only consume the latest one. Not sure if an earlier commit had a different time, but the intention is that the under / over replication threads will run at a faster interval than the replication manager. Eg 30 seconds, as that is the DN heartbeart interval. The queues will be regenerated each time the RM thread runs, probably every 5 or 10 minutes. As we are not using an event driven model where, for example, the dead node event causes the containers to be marked under replicated or a node registration causes them to be over replicated (this is how HDFS works), we decided to keep things simple and just re-create the queues on each iteration rather than trying to maintain a single queue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
