Change in vdsm[master]: vm: improve safety between startup and shutdown
gerrit-hooks has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 8: * #912390::Update tracker: OK -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 8 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Francesco Romani has abandoned this change. Change subject: vm: improve safety between startup and shutdown .. Abandoned replaced by 55150 and 55151 -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 8 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Milan Zamazal has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 8: (2 comments) https://gerrit.ovirt.org/#/c/44989/8/vdsm/virt/vm.py File vdsm/virt/vm.py: Line 251: vmdevices.graphics.initLegacyConf(self.conf) Line 252: self.cif = cif Line 253: self.log = SimpleLogAdapter(self.log, {"vmId": self.conf['vmId']}) Line 254: self._destroyed = threading.Event() Line 255: self._destroyRequested = threading.Event() > though call. On one hand, we should move (as fast as possible) towards pep8 OK. Line 256: self._recovery_file = recovery.File(self.conf['vmId']) Line 257: self._monitorResponse = 0 Line 258: self.memCommitted = 0 Line 259: self._consoleDisconnectAction = ConsoleDisconnectAction.LOCK_SCREEN Line 1765: self._created.set() Line 1766: Line 1767: def _domDependentInit(self): Line 1768: if self._destroyRequested.is_set(): Line 1769: raise DestroyedOnStartupError(vmexitreason.DESTROYED_ON_STARTUP) > The key change here is the introduction of the "created" event. I see, I got confused by something, perhaps thinking about too many things at once. Thanks for explanation, it's fine for me. Line 1770: Line 1771: if not self._dom.connected: Line 1772: raise MissingLibvirtDomainError(vmexitreason.LIBVIRT_START_FAILED) Line 1773: -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 8 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: Yes ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Francesco Romani has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 8: (1 comment) I don't want to push obfuscated changes. let me clean up this more. https://gerrit.ovirt.org/#/c/44989/8/vdsm/virt/vm.py File vdsm/virt/vm.py: Line 1765: self._created.set() Line 1766: Line 1767: def _domDependentInit(self): Line 1768: if self._destroyRequested.is_set(): Line 1769: raise DestroyedOnStartupError(vmexitreason.DESTROYED_ON_STARTUP) > What happens if destroy gets called after this check? Is there anything pre The key change here is the introduction of the "created" event. destroy() should go ahead only if the "created" event is set, so the checkpoint is reached -and the creation thread is done. This _is_ a bit extreme, but I can't see a better (= more efficient) way to make the flows safe with respect to each other. On the bright side, I don't expect this code to be triggered too often, destroy on startup is relatively infrequent AFAIK. So, the creation thread must be completed before the destroy can go ahead in a fully safe way, but this may take a while. Hence we need a timeout (implemented), and a shortcut. The shortcut is the 'destroyRequested' event. It has to be set as soon as possible, but _after_ the Domain object is created. Perhaps it is better to split this 'optimization' from the main change, it could make things clearer. Line 1770: Line 1771: if not self._dom.connected: Line 1772: raise MissingLibvirtDomainError(vmexitreason.LIBVIRT_START_FAILED) Line 1773: -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 8 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: Yes ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Francesco Romani has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 8: (2 comments) https://gerrit.ovirt.org/#/c/44989/8/vdsm/virt/vm.py File vdsm/virt/vm.py: Line 251: vmdevices.graphics.initLegacyConf(self.conf) Line 252: self.cif = cif Line 253: self.log = SimpleLogAdapter(self.log, {"vmId": self.conf['vmId']}) Line 254: self._destroyed = threading.Event() Line 255: self._destroyRequested = threading.Event() > We shouldn't use camel case for attribute names, right? though call. On one hand, we should move (as fast as possible) towards pep8 compliancy. On the other hand, this module and this class are massively camelCased, so having mixed case is, IMO, the worst possible outcome. Line 256: self._recovery_file = recovery.File(self.conf['vmId']) Line 257: self._monitorResponse = 0 Line 258: self.memCommitted = 0 Line 259: self._consoleDisconnectAction = ConsoleDisconnectAction.LOCK_SCREEN Line 1306: try: Line 1307: self.guestAgent.stop() Line 1308: except Exception: Line 1309: pass Line 1310: self._created.set() > Setting this at two quite different places worries me a bit. E.g. didn't we I see your point. Let me think harder about a better solution. Line 1311: self.saveState() Line 1312: if event_data: Line 1313: self.send_status_event(**event_data) Line 1314: -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 8 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: Yes ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Milan Zamazal has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 8: (3 comments) I can't say there is something clearly wrong with the change, but it looks a bit suspicious to me. While it probably shouldn't cause regressions, I'm not sure it handles the problem completely. Maybe just some source comment is missing, maybe something should be done a bit differently. I don't know now. https://gerrit.ovirt.org/#/c/44989/8/vdsm/virt/vm.py File vdsm/virt/vm.py: Line 251: vmdevices.graphics.initLegacyConf(self.conf) Line 252: self.cif = cif Line 253: self.log = SimpleLogAdapter(self.log, {"vmId": self.conf['vmId']}) Line 254: self._destroyed = threading.Event() Line 255: self._destroyRequested = threading.Event() We shouldn't use camel case for attribute names, right? Line 256: self._recovery_file = recovery.File(self.conf['vmId']) Line 257: self._monitorResponse = 0 Line 258: self.memCommitted = 0 Line 259: self._consoleDisconnectAction = ConsoleDisconnectAction.LOCK_SCREEN Line 1306: try: Line 1307: self.guestAgent.stop() Line 1308: except Exception: Line 1309: pass Line 1310: self._created.set() Setting this at two quite different places worries me a bit. E.g. didn't we forget about another place? Line 1311: self.saveState() Line 1312: if event_data: Line 1313: self.send_status_event(**event_data) Line 1314: Line 1765: self._created.set() Line 1766: Line 1767: def _domDependentInit(self): Line 1768: if self._destroyRequested.is_set(): Line 1769: raise DestroyedOnStartupError(vmexitreason.DESTROYED_ON_STARTUP) What happens if destroy gets called after this check? Is there anything preventing it? Or is it harmless? Line 1770: Line 1771: if not self._dom.connected: Line 1772: raise MissingLibvirtDomainError(vmexitreason.LIBVIRT_START_FAILED) Line 1773: -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 8 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: Yes ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Francesco Romani has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 8: Code-Review-1 (1 comment) the commit message is horrible https://gerrit.ovirt.org/#/c/44989/8/vdsm/virt/vm.py File vdsm/virt/vm.py: PS8, Line 1761: def _post_initialize(self): : try: : self._domDependentInit() : finally: : self._created.set() not sure this is worth anymore: 1. we *need* to have a safety set in setDownStatus(), so 2. if _domDependentInit() raises, we are covered anyway so I could just move line 1765 at the end of domDependentInit and that's it. -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 8 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: Yes ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Martin Polednik has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 7: (9 comments) Not-so-well worded message, and my gerrit doesn't show +1/-1 buttons. https://gerrit.ovirt.org/#/c/44989/6//COMMIT_MSG Commit Message: PS6, Line 9: the You are already refering to 'the flows' with the first the. PS6, Line 12: on in PS6, Line 13: _means_ PS6, Line 14: 's PS6, Line 15: learn learning PS6, Line 15: to drop PS6, Line 19: start starts https://gerrit.ovirt.org/#/c/44989/7/lib/vdsm/virt/vmexitreason.py File lib/vdsm/virt/vmexitreason.py: PS7, Line 34: Unrelated, do we really want 2 newlines here? https://gerrit.ovirt.org/#/c/44989/6/vdsm/virt/vm.py File vdsm/virt/vm.py: PS6, Line 2305: self._waitForDeviceRemoval(nic) : except HotunplugTimeout as e: Have you rebased properly? Why is it showing up here? -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 7 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: Yes ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
gerrit-hooks has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 8: * #912390::Update tracker: OK * Check Bug-Url::OK * Check Public Bug::#912390::OK, public bug * Check Product::#912390::OK, Correct classification oVirt * Check TM::SKIP, not in a monitored branch (ovirt-3.6 ovirt-3.5 ovirt-3.4 ovirt-3.3 ovirt-3.2) * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.6', 'ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 8 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Francesco Romani has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 7: Verified+1 verification: - non regression: created and destroyed (poweroff) VM using Engine's webadmin, no issue - used patch 55132 to semi-reliably check the new path In each and every test verified that the output of virsh list and vdsClient list was consistent, and that no errors were reported in Vdsm logs. Everything looks OK. -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 7 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
gerrit-hooks has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 7: * #912390::Update tracker: OK * Check Bug-Url::OK * Check Public Bug::#912390::OK, public bug * Check Product::#912390::OK, Correct classification oVirt * Check TM::SKIP, not in a monitored branch (ovirt-3.6 ovirt-3.5 ovirt-3.4 ovirt-3.3 ovirt-3.2) * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.6', 'ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 7 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Francesco Romani has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 5: (2 comments) https://gerrit.ovirt.org/#/c/44989/5/vdsm/virt/vm.py File vdsm/virt/vm.py: PS5, Line 90: _DESTROY_ON_STARTUP_TIMEOUT = 30 # seconds > Arbitrary or backed up? Completely arbitrary. PS5, Line 192: class DestroyedOnStartupError(Exception): : """ : Domain destroyed during startup process. : """ > (not related to *this* patch) do we have plans to move the exceptions to th yep, probably like lib/vdsm/virt/exceptions.py - to be used internally *in Vdsm*, not part of any external API (e.g. with Engine). This is queued up behind currently started cleanups. -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 5 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: Yes ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Martin Polednik has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 6: Code-Review+1 (2 comments) Commented PS#5, would still like answers. Nothing block worthy imo. https://gerrit.ovirt.org/#/c/44989/5/vdsm/virt/vm.py File vdsm/virt/vm.py: PS5, Line 90: _DESTROY_ON_STARTUP_TIMEOUT = 30 # seconds Arbitrary or backed up? PS5, Line 192: class DestroyedOnStartupError(Exception): : """ : Domain destroyed during startup process. : """ (not related to *this* patch) do we have plans to move the exceptions to their module? -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 6 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: Yes ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
gerrit-hooks has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 6: * #912390::Update tracker: OK * Check Bug-Url::OK * Check Public Bug::#912390::OK, public bug * Check Product::#912390::OK, Correct classification oVirt * Check TM::SKIP, not in a monitored branch (ovirt-3.6 ovirt-3.5 ovirt-3.4 ovirt-3.3 ovirt-3.2) * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.6', 'ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 6 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Francesco Romani has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 5: (2 comments) https://gerrit.ovirt.org/#/c/44989/5/lib/api/vdsmapi-schema.json File lib/api/vdsmapi-schema.json: PS5, Line 7051: > We do not need this space here. Thanks, will fix both. https://gerrit.ovirt.org/#/c/44989/5/lib/vdsm/virt/vmexitreason.py File lib/vdsm/virt/vmexitreason.py: PS5, Line 46: , > In old version we had no comma at the end. Do we need it now? could make things easier for future updates, but no big deal -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 5 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: Yes ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Piotr Kliczewski has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 5: (3 comments) https://gerrit.ovirt.org/#/c/44989/5/lib/api/vdsmapi-schema.json File lib/api/vdsmapi-schema.json: PS5, Line 7051: . This dot is not needed. PS5, Line 7051: We do not need this space here. https://gerrit.ovirt.org/#/c/44989/5/lib/vdsm/virt/vmexitreason.py File lib/vdsm/virt/vmexitreason.py: PS5, Line 46: , In old version we had no comma at the end. Do we need it now? -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 5 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: Yes ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
gerrit-hooks has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 5: * #912390::Update tracker: OK * Check Bug-Url::OK * Check Public Bug::#912390::OK, public bug * Check Product::#912390::OK, Correct classification oVirt * Check TM::SKIP, not in a monitored branch (ovirt-3.6 ovirt-3.5 ovirt-3.4 ovirt-3.3 ovirt-3.2) * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.6', 'ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 5 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Arik Hadas Gerrit-Reviewer: Dan Kenigsberg Gerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Michal Skrivanek Gerrit-Reviewer: Milan Zamazal Gerrit-Reviewer: Piotr Kliczewski Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
gerrit-hooks has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 4: * #912390::Update tracker: OK * Check Bug-Url::OK * Check Public Bug::#912390::OK, public bug * Check Product::#912390::OK, Correct classification oVirt * Check TM::SKIP, not in a monitored branch (ovirt-3.6 ovirt-3.5 ovirt-3.4 ovirt-3.3 ovirt-3.2) * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.6', 'ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 4 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
gerrit-hooks has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 3: * #912390::Update tracker: OK * Check Bug-Url::OK * Check Public Bug::#912390::OK, public bug * Check Product::#912390::OK, Correct classification oVirt * Check TM::SKIP, not in a monitored branch (ovirt-3.6 ovirt-3.5 ovirt-3.4 ovirt-3.3 ovirt-3.2) * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.6', 'ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Francesco Romani has restored this change. Change subject: vm: improve safety between startup and shutdown .. Restored let's try again for 4.0 -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: restore Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
gerrit-hooks has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 2: * #912390::Update tracker: OK -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Francesco Romani has abandoned this change. Change subject: vm: improve safety between startup and shutdown .. Abandoned -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Francesco Romani has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 2: Code-Review-1 should be split in two patches -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Francesco Romani Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: gerrit-hooks Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Martin Polednik has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 2: Would be easier to read if you noted (explicitly :) ) that there is no a behavior change - or so do I think :/ -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: automat...@ovirt.org Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
automat...@ovirt.org has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 2: * Update tracker::#912390::OK * Check Bug-Url::OK * Check Public Bug::#912390::OK, public bug * Check Product::#912390::OK, Correct product Red Hat Enterprise Virtualization Manager * Check TR::SKIP, not in a monitored branch (ovirt-3.5 ovirt-3.4 ovirt-3.3 ovirt-3.2) * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco RomaniGerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Martin Polednik Gerrit-Reviewer: Vinzenz Feenstra Gerrit-Reviewer: automat...@ovirt.org Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: vm: improve safety between startup and shutdown
Francesco Romani has uploaded a new change for review. Change subject: vm: improve safety between startup and shutdown .. vm: improve safety between startup and shutdown The VM startup and the VM shutdown flows are asynchronous to each other. Under normal conditions of load, there is a small window on which it is unsafe to call destroy while startup is running; here unsafe that the flows can stomp on each other toes. The result is confusion, so Engine is forced to wait for the dust to settle before to properly learn what happened, and one flow will fail with weird errors. Besides this confusion, there is no catastrophic effect, but to avoid further bitrot this patch start cleaning up, while we plan the much needed and deeper overhaul of the code. - replace booleans with threading.Events - better handling of the corner case on which a VM is destroyed while starting up, with dedicated vm exit reason. Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Bug-Url: https://bugzilla.redhat.com/912390 Signed-off-by: Francesco Romani from...@redhat.com --- M vdsm/virt/vm.py M vdsm/virt/vmexitreason.py 2 files changed, 26 insertions(+), 14 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/89/44989/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 0fe3ab1..10e6e1c 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -256,6 +256,12 @@ self.reason = reason +class DestroyedOnStartupError(Exception): + +Domain destroyed during startup process. + + + class Vm(object): Used for abstracting communication between various parts of the @@ -312,7 +318,7 @@ vmdevices.graphics.initLegacyConf(self.conf) self.cif = cif self.log = SimpleLogAdapter(self.log, {vmId: self.conf['vmId']}) -self._destroyed = False +self._destroyed = threading.Event() self._recoveryFile = constants.P_VDSM_RUN + \ str(self.conf['vmId']) + '.recovery' self._monitorResponse = 0 @@ -357,7 +363,7 @@ self._guestSocketFile, self.cif.channelListener, self.log, self._onGuestStatusChange) self._domain = DomainDescriptor.from_id(self.id) -self._released = False +self._released = threading.Event() self._releaseLock = threading.Lock() self.saveState() self._watchdogEvent = {} @@ -793,6 +799,13 @@ except MigrationError: self.log.exception(Failed to start a migration destination vm) self.setDownStatus(ERROR, vmexitreason.MIGRATION_FAILED) +except DestroyedOnStartupError: +self.log.info(Domain destroyed during startup) +if self._released.is_set(): +self.setDownStatus(NORMAL, vmexitreason.DESTROYED_ON_STARTUP) +# else something in the destroy path may have failed. +# We don't know, and we don't have the means to know. +# Don't do anything more, to avoid to increas the mess. except Exception as e: if self.recovering: self.log.info(Skipping errors on recovery, exc_info=True) @@ -815,7 +828,7 @@ def preparePaths(self, drives): for drive in drives: with self._volPrepareLock: -if self._destroyed: +if self._destroyed.is_set(): # A destroy request has been issued, exit early break drive['path'] = self.cif.prepareVolumePath(drive, self.id) @@ -865,7 +878,7 @@ pass def _saveStateInternal(self): -if self._destroyed: +if self._destroyed.is_set(): return toSave = self.status() toSave['startTime'] = self._startTime @@ -923,7 +936,7 @@ # This is not a definite fix, we're aware that there is still the # possibility of a race condition, however this covers more cases # than before and a quick gain -if not self.conf.get('clientIp', '') and not self._destroyed: +if not self.conf.get('clientIp', '') and not self._destroyed.is_set(): delay = config.get('vars', 'user_shutdown_timeout') timeout = config.getint('vars', 'sys_shutdown_timeout') CDA = ConsoleDisconnectAction @@ -1768,14 +1781,10 @@ uuidPath, name) def _domDependentInit(self): -if self._destroyed: +if self._destroyed.is_set(): # reaching here means that Vm.destroy() was called before we could # handle it. We must handle it now -try: -self._dom.destroy() -except Exception: -pass -raise Exception('destroy() called before Vm started') +raise DestroyedOnStartupError() if not self._dom.connected: raise
Change in vdsm[master]: vm: improve safety between startup and shutdown
automat...@ovirt.org has posted comments on this change. Change subject: vm: improve safety between startup and shutdown .. Patch Set 1: * Update tracker::#912390::OK * Check Bug-Url::OK * Check Public Bug::#912390::OK, public bug * Check Product::#912390::OK, Correct product Red Hat Enterprise Virtualization Manager * Check TR::SKIP, not in a monitored branch (ovirt-3.5 ovirt-3.4 ovirt-3.3 ovirt-3.2) * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I8718f58f1d255d9e603db75aa1f256c03c300f3a Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani from...@redhat.com Gerrit-Reviewer: automat...@ovirt.org Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches