Re: [Users] SD Again
Il 10/06/2013 11:54, Dafna Ron ha scritto: I did see issues with your storage in the logs previously to the detach issue but they seem to have been resolved so I do not think that they are related. what I can see in the logs is that the detach fails because the connectStorageServer command, which suppose to mount the storage, fails: Mount of `172.16.0.5:/home/external/migration` at `/rhev/data-center/mnt/172.16.0.5:_home_external_migration` does not exist so unless you had a momentary problem in your storage it actually looks like a vdsm bug to me. adding Eduardo, perhaps he will see something I missed, but I think that its possibly related to this issue: https://bugzilla.redhat.com/show_bug.cgi?id=971291 Thank you very much I confirm that to solve the problem I had to: -manually mkdir /rhev/data-center/mnt/172.16.0.5:_home_external_migration - give it properly permissions - manually mount the storage of Export Domain in the right place - detach the SD using oVirt GUI I looks similar to the ug you point Thank you for your assistance Ovirt looks terribly good but it sometimes makes my road longer without any apparent reason Best regards Alessandro Bianchi -- SkyNet SRL Via Maggiate 67/a - 28021 Borgomanero (NO) - tel. +39 0322-836487/834765 - fax +39 0322-836608 http://www.skynet.it Autorizzazione Ministeriale n.197 Le informazioni contenute in questo messaggio sono riservate e confidenziali ed è vietata la diffusione in qualunque modo eseguita. Qualora Lei non fosse la persona a cui il presente messaggio è destinato, La invitiamo ad eliminarlo ed a distruggerlo non divulgandolo, dandocene gentilmente comunicazione. Per qualsiasi informazione si prega di contattare i...@skynet.it (e-mail dell'azienda). Rif. D.L. 196/2003 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] SD Again
Il 07/06/2013 05:05, Shu Ming ha scritto: From engine log: 2013-06-06 12:36:58,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-3-thread-47) [7345dde6] START, ConnectStorageServerVDSCommand(HostName = nodo1, HostId = 3156bdac-ebfb-44cf-bea6-53d668b74a10, storagePoolId = ----, storageType = NFS, connectionList = [{ id: 45085cbf-da10-4852-9d85-754707d20a92, connection: 172.16.0.5:/home/external/migration, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 54464e6b2013-06-06 12:36:59,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-3-thread-47) [7345dde6] FINISH, ConnectStorageServerVDSCommand, return: {45085cbf-da10-4852-9d85-754707d20a92=100}, log id: 54464e6b 2013-06-06 12:36:59,150 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (pool-3-thread-47) [7345dde6] The connection with details 172.16.0.5:/home/external/migration failed because of error code 100 and error message is: generalexception 2013-06-06 12:36:59,175 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-3-thread-47) [7345dde6] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. *It seems that 45085cbf-da10-4852-9d85-754707d20a92 was disconnected at 12:36:49 from vdsm.log, just before this ConnectStorageServerVDSCommand() in engine.log at 12:36:58. What does that mean? Did it connect the Storage server after the VDSM disconnected the StorageServer in a very short time interval. ** *** From vdsm log: "Thread-281877::INFO::2013-06-06 12:36:49,933::logUtils::41::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='----', conList=[{'connection': '172.16.0.5:/home/external/migration', 'iqn': '', 'portal': '', 'user': '', 'password': '**', 'id': '45085cbf-da10-4852-9d85-754707d20a92', 'port': ''}], options=None) *Thread-281877::DEBUG::2013-06-06 12:36:49,933::misc::84::Storage.Misc.excCmd::() '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/172.16.0.5:_home_external_migration' (cwd None)*" "Thread-281877::DEBUG::2013-06-06 12:36:52,190::task::1151::TaskManager.Task::(prepare) Task=`c0d1c115-cb3d-4f39-8c9b-448401097921`::finished: {'statuslist': [{'status': 0, 'id': '45085cbf-da10-4852-9d85-754707d20a92'}]}" I suppose something went wrong in the flow of deactivating the SD and then detach it As soon as I saw the red mark showing the down status of the Export domain I placed in Maint, I clicked the "detach" button and kaboom! IMO it took the "right time" before the mark went red and SD was marked as "Maintenance" so I didn't raise any suspect, but immediately after it failed to detach and was NOT mounted on the node. Did the GUI say SD was down while it wasn't yet? After I remounted the Export SD on the node by hand I was able to detach the Export Domain using the GUI. It looks like some sort of "asynchronous" execution of commands on the node isn't it? Thanks and bye Alessandro -- SkyNet SRL Via Maggiate 67/a - 28021 Borgomanero (NO) - tel. +39 0322-836487/834765 - fax +39 0322-836608 http://www.skynet.it Autorizzazione Ministeriale n.197 Le informazioni contenute in questo messaggio sono riservate e confidenziali ed è vietata la diffusione in qualunque modo eseguita. Qualora Lei non fosse la persona a cui il presente messaggio è destinato, La invitiamo ad eliminarlo ed a distruggerlo non divulgandolo, dandocene gentilmente comunicazione. Per qualsiasi informazione si prega di contattare i...@skynet.it (e-mail dell'azienda). Rif. D.L. 196/2003 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] SD Again
Il 06/06/2013 15:55, users-requ...@ovirt.org ha scritto: Message: 5 Date: Thu, 06 Jun 2013 15:38:48 +0300 From: Dafna Ron To: users@ovirt.org Subject: Re: [Users] SD Again Message-ID: <51b082d8.5040...@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed You can attach full engine and vdsm logs and I'll try to debug it (I need to see the first umount). On 06/06/2013 02:25 PM, Alessandro Bianchi wrote: > Hi guys > > I solved manually remounting the Expord SD after restarting the NFS > server and then detaching the export domain > > I wonder if someone knows why this rhings happen ... > > Best regards > > Hope I'm providing the right stuff I may send you all the file but these are the relevat infos I suppose vdsm.log Thread-281877::DEBUG::2013-06-06 12:36:49,932::BindingXMLRPC::161::vds::(wrapper) [172.16.0.5] Thread-281877::DEBUG::2013-06-06 12:36:49,932::task::568::TaskManager.Task::(_updateState) Task=`c0d1c115-cb3d-4f39-8c9b-448401097921`::moving from state init -> state preparing Thread-281877::INFO::2013-06-06 12:36:49,933::logUtils::41::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='----', conList=[{'connection': '172.16.0.5:/home/external/migration', 'iqn': '', 'portal': '', 'user': '', 'password': '**', 'id': '45085cbf-da10-4852-9d85-754707d20a92', 'port': ''}], options=None) Thread-281877::DEBUG::2013-06-06 12:36:49,933::misc::84::Storage.Misc.excCmd::() '/usr/bin/sudo -n /usr/bin/umount -f -l /rhev/data-center/mnt/172.16.0.5:_home_external_migration' (cwd None) Thread-281877::DEBUG::2013-06-06 12:36:50,012::misc::1054::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) Thread-281877::DEBUG::2013-06-06 12:36:50,012::misc::1056::SamplingMethod::(__call__) Got in to sampling method Thread-281877::DEBUG::2013-06-06 12:36:50,012::misc::1054::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) Thread-281877::DEBUG::2013-06-06 12:36:50,013::misc::1056::SamplingMethod::(__call__) Got in to sampling method Thread-281877::DEBUG::2013-06-06 12:36:50,013::misc::84::Storage.Misc.excCmd::() '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) Thread-281877::DEBUG::2013-06-06 12:36:50,021::misc::84::Storage.Misc.excCmd::() FAILED: = 'iscsiadm: No session found.\n'; = 21 Thread-281877::DEBUG::2013-06-06 12:36:50,021::misc::1064::SamplingMethod::(__call__) Returning last result Thread-281877::DEBUG::2013-06-06 12:36:52,029::misc::84::Storage.Misc.excCmd::() '/usr/bin/sudo -n /sbin/multipath' (cwd None) Thread-281877::DEBUG::2013-06-06 12:36:52,188::misc::84::Storage.Misc.excCmd::() SUCCESS: = ''; = 0 Thread-281877::DEBUG::2013-06-06 12:36:52,188::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-281877::DEBUG::2013-06-06 12:36:52,189::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-281877::DEBUG::2013-06-06 12:36:52,189::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-281877::DEBUG::2013-06-06 12:36:52,189::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-281877::DEBUG::2013-06-06 12:36:52,189::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-281877::DEBUG::2013-06-06 12:36:52,189::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-281877::DEBUG::2013-06-06 12:36:52,189::misc::1064::SamplingMethod::(__call__) Returning last result Thread-281877::INFO::2013-06-06 12:36:52,189::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '45085cbf-da10-4852-9d85-754707d20a92'}]} Thread-281877::DEBUG::2013-06-06 12:36:52,190::task::1
Re: [Users] SD Again
Il 06/06/2013 13:56, Maor Lipchuk ha scritto: Hi Alessandro, Is there a possibility some one else is using the export domain in another DC or setup? Regards, Maor On 06/06/2013 02:25 PM, Alessandro Bianchi wrote: Hi guys I solved manually remounting the Expord SD after restarting the NFS server and then detaching the export domain I wonder if someone knows why this rhings happen ... Best regards ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users Hi and Thank you for your answer No: it's not possible since I'm the only admin of the ovirt nodes (3 nodes holding about 14 VMs) and I attached the Export SD, then exported the VM and got the lock on export domain. It's really funny that sometimes I keep in trouble with SD: they don't come up or don't go down (or they simply don't like me :-) All my nodes and the engine host are F 18 with all latest updates applied (engine is also checked daily for updates) except for kernel due to this nasty bug https://bugzilla.redhat.com/show_bug.cgi?id=902012 Am I the only one fighting with SD's? If yes it may be some sort of network/nfs/"x-file" problem, otherwise I don't know what to investigate ... Thanks again Best regards Alessandro Bianchi -- SkyNet SRL Via Maggiate 67/a - 28021 Borgomanero (NO) - tel. +39 0322-836487/834765 - fax +39 0322-836608 http://www.skynet.it Autorizzazione Ministeriale n.197 Le informazioni contenute in questo messaggio sono riservate e confidenziali ed è vietata la diffusione in qualunque modo eseguita. Qualora Lei non fosse la persona a cui il presente messaggio è destinato, La invitiamo ad eliminarlo ed a distruggerlo non divulgandolo, dandocene gentilmente comunicazione. Per qualsiasi informazione si prega di contattare i...@skynet.it (e-mail dell'azienda). Rif. D.L. 196/2003 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [Users] SD Again
Hi guys I solved manually remounting the Expord SD after restarting the NFS server and then detaching the export domain I wonder if someone knows why this rhings happen ... Best regards ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[Users] SD again ...
Hi all Another problem with SD (those SD drive me crazy !) I've attached the Export Domain to a Local domain and succesfully exported a VM Then I've placed the Export SD in Maintenance Then I asked oVirt to Detach the Domain and that seems impossible I see the mount point has been unomunted on the node but oVirt doesn't mark the SD as unattached ... I tried to reacttivate it with non luck (it's not mounted at the moment) Andy advice about how to get Export domain available again? I'm on Fedora 18 Here is the log 2013-06-06 13:03:12,676 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-3-thread-50) [59909ba8] Running command: DetachStorageDomainFromPoolCommand internal: false. Entities affected : ID: e79cd423-ae17-4f8b-9f53-28d851cc9822 Type: Storage 2013-06-06 13:03:12,676 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-3-thread-50) [59909ba8] Start detach storage domain 2013-06-06 13:03:12,684 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-3-thread-50) [59909ba8] Detach storage domain: before connect 2013-06-06 13:03:12,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-3-thread-47) [69a827a7] START, ValidateStorageServerConnectionVDSCommand(HostName = nodo1, HostId = 3156bdac-ebfb-44cf-bea6-53d668b74a10, storagePoolId = ----, storageType = NFS, connectionList = [{ id: 45085cbf-da10-4852-9d85-754707d20a92, connection: 172.16.0.5:/home/external/migration, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1b41b932 2013-06-06 13:03:12,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-3-thread-47) [69a827a7] FINISH, ValidateStorageServerConnectionVDSCommand, return: {45085cbf-da10-4852-9d85-754707d20a92=0}, log id: 1b41b932 2013-06-06 13:03:12,695 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-3-thread-47) [69a827a7] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa0----123456789aaa Type: System 2013-06-06 13:03:12,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-3-thread-47) [69a827a7] START, ConnectStorageServerVDSCommand(HostName = nodo1, HostId = 3156bdac-ebfb-44cf-bea6-53d668b74a10, storagePoolId = ----, storageType = NFS, connectionList = [{ id: 45085cbf-da10-4852-9d85-754707d20a92, connection: 172.16.0.5:/home/external/migration, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3f7041da 2013-06-06 13:03:12,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-3-thread-47) [69a827a7] FINISH, ConnectStorageServerVDSCommand, return: {45085cbf-da10-4852-9d85-754707d20a92=100}, log id: 3f7041da 2013-06-06 13:03:12,739 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (pool-3-thread-47) [69a827a7] The connection with details 172.16.0.5:/home/external/migration failed because of error code 100 and error message is: generalexception 2013-06-06 13:03:12,741 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-3-thread-47) [69a827a7] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. 2013-06-06 13:03:12,741 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-3-thread-50) [59909ba8] Detach storage domain: after connect 2013-06-06 13:03:12,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (pool-3-thread-50) [59909ba8] START, DetachStorageDomainVDSCommand( storagePoolId = d76c9edf-34cb-48eb-b53b-32d27bedc26a, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = e79cd423-ae17-4f8b-9f53-28d851cc9822, masterDomainId = ----, masterVersion = 1, force = false), log id: 6e692e6d 2013-06-06 13:03:14,327 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) No string for UNASSIGNED type. Use default Log 2013-06-06 13:03:14,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) [59909ba8] Fai