Liron Aravot has uploaded a new change for review. Change subject: sp: race in domains upgrade prevents further pool upgrades ......................................................................
sp: race in domains upgrade prevents further pool upgrades When executing _upgradePool() a callback a domain upgrade callback is registered to the domain monitor and an initial update thread is started for each of the domain in self._domainsToUpgrade (which contains the active pool domains). Whenever upgrade for domain is ended, the upgrade thread deletes the domain from self._domainsToUpgrade, which is racey because that might happen while we still iterate over the list. If that does happen, the results are unexpected and on some inspected cases we didn't start the upgrade threads for all the domains which causes to a bug - an update will be considered running and no further pool upgrades could be initiated. This patch copies the list to a temporary list before attempting to register the callback/start the upgrade threads and by that avoiding that race. this solution is simple as its targeted to the 3.6 branch, a more complete solution which will also allow concurrent updates is introduced in I8e14a3aa33bfab4751ab5d1e3becbeda892da4c3 Change-Id: Ie384c63315214786dc62f2b4998002320e314d30 Bug-Url: http://bugzilla.redhat.com/1319523 Signed-off-by: Liron Aravot <[email protected]> --- M vdsm/storage/sp.py 1 file changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/50/57450/1 diff --git a/vdsm/storage/sp.py b/vdsm/storage/sp.py index f5e69e8..fc825e6 100644 --- a/vdsm/storage/sp.py +++ b/vdsm/storage/sp.py @@ -459,7 +459,11 @@ self.domainMonitor.onDomainStateChange.register( self._upgradeCallback) self.log.debug("Running initial domain upgrade threads") - for sdUUID in self._domainsToUpgrade: + # We need to copy the list as the domain monitor registered + # callback and the initiated update threads may modify the list + # while we iterate over it. + # http://bugzilla.redhat.com/1319523 + for sdUUID in self._domainsToUpgrade[:]: threading.Thread(target=self._upgradeCallback, args=(sdUUID, True), kwargs={"__securityOverride": True}).start() -- To view, visit https://gerrit.ovirt.org/57450 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ie384c63315214786dc62f2b4998002320e314d30 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: ovirt-3.6 Gerrit-Owner: Liron Aravot <[email protected]> _______________________________________________ vdsm-patches mailing list [email protected] https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
