----- Original Message ----- > From: "Daniel Helgenberger" <daniel.helgenber...@m-box.de> > To: users@ovirt.org > Sent: Tuesday, May 12, 2015 7:48:42 PM > Subject: [ovirt-users] HE setup failing: Cannot set temporary password for > console connection > > Hello, > > a week ago I set up a HE (3.5.2) with no problems; now this fails. I > suspect storage trouble but cannot pinpoint it. Did someth. change there > with the recent async release? > > The storage domain is NFS3 via a 1G link to a ZFS storage appliance. The > export is working quite fast; tested with iozone running for over 24h > without problems. > > It starts with the "Verifying sanlock lockspace initialization" phase; > which lasts about 7min (!). > > In the end it seems sanlock is locking the domain down. I can only get > it back by stopping sanlock and vdsm. > > Attached is the full vdsm lock. > > > Thanks! > > vdsm: > > Thread-27::ERROR::2015-05-12 > 19:17:51,470::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) > Error while collecting domain 2f83ea82-e110-465d-a2a6-ac30534a9f6a > monitoring information > Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in > _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) > StorageDomainDoesNotExist: Storage domain does not exist: > ('2f83ea82-e110-465d-a2a6-ac30534a9f6a',) > > > Detector thread::DEBUG::2015-05-12 > 19:24:33,163::BindingXMLRPC::1173::XmlDetector::(handleSocket) xml over http > detected from ('127.0.0.1', 47076) > Thread-93::DEBUG::2015-05-12 > 19:24:33,166::BindingXMLRPC::1133::vds::(wrapper) client [127.0.0.1]::call > vmSetTicket with ('e3d350c5-6254-4908-99e9-d60ca2e046f7', '5435uOVP', > '10800', 'disconnect', {}) {} > Thread-93::ERROR::2015-05-12 > 19:24:33,166::BindingXMLRPC::1152::vds::(wrapper) unexpected error > Traceback (most recent call last): > File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1136, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 383, in vmSetTicket > return vm.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/API.py", line 642, in setTicket > return v.setTicket(password, ttl, existingConnAction, params) > File "/usr/share/vdsm/virt/vm.py", line 4834, in setTicket > graphics = _domParseStr(self._dom.XMLDesc(0)).childNodes[0]. \ > AttributeError: 'NoneType' object has no attribute 'XMLDesc' > > and: > > Thread-91::DEBUG::2015-05-12 > 19:25:15,152::libvirtconnection::143::root::(wrapper) Unknown libvirterror: > ecode: 38 edom: 42 level: 2 message: Failed to acquire lock: No space left > on device > Thread-91::DEBUG::2015-05-12 > 19:25:15,154::vm::2294::vm.Vm::(_startUnderlyingVm) > vmId=`e3d350c5-6254-4908-99e9-d60ca2e046f7`::_ongoingCreations released > Thread-91::ERROR::2015-05-12 > 19:25:15,154::vm::2331::vm.Vm::(_startUnderlyingVm) > vmId=`e3d350c5-6254-4908-99e9-d60ca2e046f7`::The vm start process failed > Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 2271, in _startUnderlyingVm > self._run() > File "/usr/share/vdsm/virt/vm.py", line 3335, in _run > self._connection.createXML(domxml, flags), > File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line > 111, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3427, in > createXML > if ret is None:raise libvirtError('virDomainCreateXML() failed', > conn=self) > libvirtError: Failed to acquire lock: No space left on device
Could you please check also SELinux audit logs with ausearch -m avc > sanlock: > 2015-05-12 19:11:27+0200 4338 [708]: s2 lockspace > 2f83ea82-e110-465d-a2a6-ac30534a9f6a:1:/rhev/data-center/mnt/10.11.0.30:_volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/ids:0 > 2015-05-12 19:11:48+0200 4359 [702]: s2 host 1 1 4338 > 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i > 2015-05-12 19:11:48+0200 4359 [702]: s2 host 250 1 0 > 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i > 2015-05-12 19:11:48+0200 4359 [707]: s2:r2 resource > 2f83ea82-e110-465d-a2a6-ac30534a9f6a:SDM:/rhev/data-center/mnt/10.11.0.30:_volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/leases:1048576 > for 3,11,4167 > 2015-05-12 19:23:26+0200 5057 [708]: s3 lockspace > 2f83ea82-e110-465d-a2a6-ac30534a9f6a:1:/rhev/data-center/mnt/10.11.0.30:_volumes_ovirt_engine/2f83ea82-e110-465d-a2a6-ac30534a9f6a/dom_md/ids:0 > 2015-05-12 19:23:48+0200 5079 [702]: s3 host 1 2 5058 > 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i > 2015-05-12 19:23:48+0200 5079 [702]: s3 host 250 1 0 > 9f9fb274-9ae8-490e-bfba-72a743b0ebb2.hv01.sec.i > 2015-05-12 19:24:53+0200 5144 [702]: cmd 9 target pid 4893 not found > 2015-05-12 19:25:14+0200 5165 [708]: r3 cmd_acquire 2,9,4893 invalid > lockspace found -1 failed 0 name 2f83ea82-e110-465d-a2a6-ac30534a9f6a > > -- > Daniel Helgenberger > m box bewegtbild GmbH > > P: +49/30/2408781-22 > F: +49/30/2408781-10 > > ACKERSTR. 19 > D-10115 BERLIN > > > www.m-box.de www.monkeymen.tv > > Geschäftsführer: Martin Retschitzegger / Michaela Göllner > Handeslregister: Amtsgericht Charlottenburg / HRB 112767 > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users