Liron Aravot has uploaded a new change for review.
Change subject: sp: race in domains upgrade prevents further pool upgrades
......................................................................
sp: race in domains upgrade prevents further pool upgrades
When executing _upgradePool() a callback a domain upgrade callback
is registered to the domain monitor and an initial update thread is
started for each of the domain in self._domainsToUpgrade (which contains
the active pool domains).
Whenever upgrade for domain is ended, the upgrade thread deletes the domain
from self._domainsToUpgrade, which is racey because that might happen while
we still iterate over the list. If that does happen, the results are unexpected
and on some inspected cases we didn't start the upgrade threads for all the
domains which causes to a bug - an update will be considered running and
no further pool upgrades could be initiated.
This patch copies the list to a temporary list before attempting to register
the callback/start the upgrade threads and by that avoiding that race.
this solution is simple as its targeted to the 3.6 branch, a more complete
solution which will also allow concurrent updates is introduced in
I8e14a3aa33bfab4751ab5d1e3becbeda892da4c3
Change-Id: Ie384c63315214786dc62f2b4998002320e314d30
Signed-off-by: Liron Aravot <lara...@redhat.com>
---
M vdsm/storage/sp.py
1 file changed, 5 insertions(+), 1 deletion(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/15/57315/1
diff --git a/vdsm/storage/sp.py b/vdsm/storage/sp.py
index af838de..e6ed8a4 100644
--- a/vdsm/storage/sp.py
+++ b/vdsm/storage/sp.py
@@ -457,11 +457,15 @@
except ValueError:
pass
+ # This is needed as the domain monitor registered callback and
+ # the initiated update threads may modify the list while we
+ # iterate over it
+ domainsToUpgrade = self._domainsToUpgrade[:]
self.log.debug("Registering with state change event")
self.domainMonitor.onDomainStateChange.register(
self._upgradeCallback)
self.log.debug("Running initial domain upgrade threads")
- for sdUUID in self._domainsToUpgrade:
+ for sdUUID in domainsToUpgrade:
concurrent.thread(self._upgradeCallback,
args=(sdUUID, True),
kwargs={"__securityOverride": True},
--
To view, visit https://gerrit.ovirt.org/57315
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie384c63315214786dc62f2b4998002320e314d30
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Liron Aravot <lara...@redhat.com>
_______________________________________________
vdsm-patches mailing list
vdsm-patches@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
