Change in vdsm[master]: hsm: wait until lvm bootstrap is done before connecting to pool
Nir Soffer has abandoned this change. Change subject: hsm: wait until lvm bootstrap is done before connecting to pool .. Abandoned Replaced by http://gerrit.ovirt.org/21530 -- To view, visit http://gerrit.ovirt.org/21388 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I22e851dca4c2063d19446f34897a7b208b9cace4 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer nsof...@redhat.com Gerrit-Reviewer: Allon Mureinik amure...@redhat.com Gerrit-Reviewer: Ayal Baron aba...@redhat.com Gerrit-Reviewer: Federico Simoncelli fsimo...@redhat.com Gerrit-Reviewer: Nir Soffer nsof...@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: hsm: wait until lvm bootstrap is done before connecting to pool
Allon Mureinik has posted comments on this change.
Change subject: hsm: wait until lvm bootstrap is done before connecting to pool
..
Patch Set 1:
(1 comment)
File lib/vdsm/config.py.in
Line 267:
Line 268: ('use_volume_leases', 'false',
Line 269: 'Whether to use the volume leases or not.'),
Line 270:
Line 271: ('lvm_bootstrap_timeout', '120',
How did we come up with this number? looks a bit excessive to me.
Line 272: 'Time in seconds to wait until lvm bootstrap is done when'
Line 273: ' connecting to storage pool.')
Line 274: ]),
Line 275:
--
To view, visit http://gerrit.ovirt.org/21388
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I22e851dca4c2063d19446f34897a7b208b9cace4
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer nsof...@redhat.com
Gerrit-Reviewer: Allon Mureinik amure...@redhat.com
Gerrit-Reviewer: Ayal Baron aba...@redhat.com
Gerrit-Reviewer: Federico Simoncelli fsimo...@redhat.com
Gerrit-Reviewer: oVirt Jenkins CI Server
Gerrit-HasComments: Yes
___
vdsm-patches mailing list
vdsm-patches@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: hsm: wait until lvm bootstrap is done before connecting to pool
Nir Soffer has posted comments on this change.
Change subject: hsm: wait until lvm bootstrap is done before connecting to pool
..
Patch Set 1:
(1 comment)
File lib/vdsm/config.py.in
Line 267:
Line 268: ('use_volume_leases', 'false',
Line 269: 'Whether to use the volume leases or not.'),
Line 270:
Line 271: ('lvm_bootstrap_timeout', '120',
I invented it so we have something concrete to start with :-)
Line 272: 'Time in seconds to wait until lvm bootstrap is done when'
Line 273: ' connecting to storage pool.')
Line 274: ]),
Line 275:
--
To view, visit http://gerrit.ovirt.org/21388
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I22e851dca4c2063d19446f34897a7b208b9cace4
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer nsof...@redhat.com
Gerrit-Reviewer: Allon Mureinik amure...@redhat.com
Gerrit-Reviewer: Ayal Baron aba...@redhat.com
Gerrit-Reviewer: Federico Simoncelli fsimo...@redhat.com
Gerrit-Reviewer: Nir Soffer nsof...@redhat.com
Gerrit-Reviewer: oVirt Jenkins CI Server
Gerrit-HasComments: Yes
___
vdsm-patches mailing list
vdsm-patches@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: hsm: wait until lvm bootstrap is done before connecting to pool
Allon Mureinik has posted comments on this change.
Change subject: hsm: wait until lvm bootstrap is done before connecting to pool
..
Patch Set 1:
(1 comment)
File lib/vdsm/config.py.in
Line 267:
Line 268: ('use_volume_leases', 'false',
Line 269: 'Whether to use the volume leases or not.'),
Line 270:
Line 271: ('lvm_bootstrap_timeout', '120',
`-)
Offhand, I'd say that 120 secs is way to long, and that 60 sounds more
appropriate, but this requires some testing.
Line 272: 'Time in seconds to wait until lvm bootstrap is done when'
Line 273: ' connecting to storage pool.')
Line 274: ]),
Line 275:
--
To view, visit http://gerrit.ovirt.org/21388
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I22e851dca4c2063d19446f34897a7b208b9cace4
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer nsof...@redhat.com
Gerrit-Reviewer: Allon Mureinik amure...@redhat.com
Gerrit-Reviewer: Ayal Baron aba...@redhat.com
Gerrit-Reviewer: Federico Simoncelli fsimo...@redhat.com
Gerrit-Reviewer: Nir Soffer nsof...@redhat.com
Gerrit-Reviewer: oVirt Jenkins CI Server
Gerrit-HasComments: Yes
___
vdsm-patches mailing list
vdsm-patches@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: hsm: wait until lvm bootstrap is done before connecting to pool
Ayal Baron has posted comments on this change. Change subject: hsm: wait until lvm bootstrap is done before connecting to pool .. Patch Set 1: what scenarios have you tested? engine has a 3 minute timeout for connectStoragePool. If the combined time of connectStoragePool and waiting for bootstrap to finish is bigger than that then connect would fail. how long does it take when you have a VG with 1000 LVs on it? -- To view, visit http://gerrit.ovirt.org/21388 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I22e851dca4c2063d19446f34897a7b208b9cace4 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer nsof...@redhat.com Gerrit-Reviewer: Allon Mureinik amure...@redhat.com Gerrit-Reviewer: Ayal Baron aba...@redhat.com Gerrit-Reviewer: Federico Simoncelli fsimo...@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: hsm: wait until lvm bootstrap is done before connecting to pool
Nir Soffer has uploaded a new change for review.
Change subject: hsm: wait until lvm bootstrap is done before connecting to pool
..
hsm: wait until lvm bootstrap is done before connecting to pool
When hsm is created, a bootsrap thread is started, initializing lvm
cache and other parts of the system. This leads to races between
different threads when accessing lvm module when connecting to storage
pool. The races may lead to two threads trying to rebuild the cache in
the same time and possibly corrupt the cache.
This patch adds a synchronization point after lvm bootstrap is done. If
engine attempt to connect to storage pool while lvm cache is not ready
yet, it will wait until it is ready.
This patch will add a delay until a host can connect to the storage
pool. This delay is unavoidable if we want to have correct code.
Change-Id: I22e851dca4c2063d19446f34897a7b208b9cace4
Signed-off-by: Nir Soffer nsof...@redhat.com
---
M lib/vdsm/config.py.in
M vdsm/storage/hsm.py
2 files changed, 19 insertions(+), 1 deletion(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/88/21388/1
diff --git a/lib/vdsm/config.py.in b/lib/vdsm/config.py.in
index 9668772..8e075fd 100644
--- a/lib/vdsm/config.py.in
+++ b/lib/vdsm/config.py.in
@@ -267,6 +267,10 @@
('use_volume_leases', 'false',
'Whether to use the volume leases or not.'),
+
+('lvm_bootstrap_timeout', '120',
+'Time in seconds to wait until lvm bootstrap is done when'
+' connecting to storage pool.')
]),
# Section: [addresses]
diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py
index 5e53f3a..725279f 100644
--- a/vdsm/storage/hsm.py
+++ b/vdsm/storage/hsm.py
@@ -373,8 +373,16 @@
except Exception:
self.log.warn(Failed to clean Storage Repository., exc_info=True)
+self._bootstrapDone = threading.Event()
+
def storageRefresh():
-lvm._lvminfo.bootstrap()
+# This may take more then a minute when having lot of lvs. Until it
+# finish, we should not use the lvm module.
+try:
+lvm._lvminfo.bootstrap()
+finally:
+self._bootstrapDone.set()
+
sdCache.refreshStorage()
fileUtils.createdir(self.tasksDir)
@@ -1008,6 +1016,12 @@
spUUID=%s, msdUUID=%s, masterVersion=%s, hostID=%s,
scsiKey=%s % (spUUID, msdUUID, masterVersion,
hostID, scsiKey)))
+
+timeout = config.getint('vars', 'lvm_bootstrap_timeout')
+self._bootstrapDone.wait(timeout)
+if not self._bootstrapDone.is_set():
+raise se.StoragePoolConnectionError(Bootstrap is not done yet)
+
with rmanager.acquireResource(STORAGE, HSM_DOM_MON_LOCK,
rm.LockType.exclusive):
return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID,
--
To view, visit http://gerrit.ovirt.org/21388
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I22e851dca4c2063d19446f34897a7b208b9cace4
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer nsof...@redhat.com
___
vdsm-patches mailing list
vdsm-patches@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
Change in vdsm[master]: hsm: wait until lvm bootstrap is done before connecting to pool
oVirt Jenkins CI Server has posted comments on this change. Change subject: hsm: wait until lvm bootstrap is done before connecting to pool .. Patch Set 1: Build Successful http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/4693/ : SUCCESS http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/5493/ : SUCCESS http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/5572/ : SUCCESS -- To view, visit http://gerrit.ovirt.org/21388 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I22e851dca4c2063d19446f34897a7b208b9cace4 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer nsof...@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server Gerrit-HasComments: No ___ vdsm-patches mailing list vdsm-patches@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-patches
