We’ve finally completed this migration. I’m not exactly sure, but I believe the missing step was to manually run "rescan-scsi-bus.sh —forcerescan” on each ovirt server to make sure they all saw the new LUN. This allowed the new hosted_engine storage domain to activate and the setup to complete?
Thanks, Devin > On Sep 11, 2024, at 9:57 AM, Devin A. Bougie <[email protected]> wrote: > > Hello, > > We are attempting to move a hosted engine from an NFS to an iSCSI storage > domain, following the normal backup / restore procedure. > > With a little intervention in the new hosted engine VM, everything seems to > be working with the new engine and the process gets to the point of trying to > create the new iSCSI hosted_storage domain. At this point, it fails with > "Storage domain cannot be reached. Please ensure it is accessible from the > host(s).” > > The host itself does see the storage, and everything looks fine using > iscsiadm and multipath commands. However, if I look at the new > hosted_storage domain in the new engine, it’s stuck “unattached” and the > “hosted-engine --deploy --restore-from-file …” command loops at the "Please > specify the storage you would like to use” step. > > Please see below for an excerpt of the output and logs, and let me know what > additional information I can provide. > > Many thanks, > Devin > > Here’s the output from the “hosted-engine --deploy --restore-from-file …” > command. > ——— > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : iSCSI login] > [ INFO ] changed: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get iSCSI LUNs] > [ INFO ] ok: [localhost] > The following luns have been found on the requested target: > [1] mpathm 1024.0GiB IFT DS 3000 Series > status: free, paths: 8 active > Please select the destination LUN (1) [1]: > [ INFO ] iSCSI discard after delete is disabled > [ INFO ] Creating Storage Domain > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Execute just a specific set > of steps] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Force facts gathering] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Wait for the storage > interface to be up] > [ INFO ] skipping: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Check local VM dir stat] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce local VM dir > existence] > [ INFO ] skipping: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Obtain SSO token using > username/password credentials] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch host facts] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster ID] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster facts] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter facts] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter ID] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter name] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster name] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster version] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce cluster major > version] > [ INFO ] skipping: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce cluster minor > version] > [ INFO ] skipping: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set storage_format] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add NFS storage domain] > [ INFO ] skipping: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add glusterfs storage > domain] > [ INFO ] skipping: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add iSCSI storage domain] > [ INFO ] changed: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add Fibre Channel storage > domain] > [ INFO ] skipping: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get storage domain details] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Find the appliance OVF] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get ovf data] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get disk size from ovf data] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get required size] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Remove unsuitable storage > domain] > [ INFO ] skipping: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Check storage domain free > space] > [ INFO ] skipping: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Activate storage domain] > [ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail > is "[Storage domain cannot be reached. Please ensure it is accessible from > the host(s).]". HTTP response code is 400. > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault > reason is \"Operation Failed\". Fault detail is \"[Storage domain cannot be > reached. Please ensure it is accessible from the host(s).]\". HTTP response > code is 400.”} > ——— > > Here’s an excerpt from the > "ovirt-hosted-engine-setup-ansible-create_storage_domain-20240911081524-c7yrkf.log”: > ——— > 2024-09-11 08:16:12,354-0400 INFO ansible task start {'status': 'OK', > 'ansible_type': 'task', 'ansible_playbook': > '/usr/share/ovirt-hosted-engine-setup/he_ansible/trigger_role.yml', > 'ansible_task': 'ovirt.ovirt.hosted_engine_setup : Activate storage domain'} > 2024-09-11 08:16:12,354-0400 DEBUG ansible on_any args TASK: > ovirt.ovirt.hosted_engine_setup : Activate storage domain kwargs > is_conditional:False 2024-09-11 08:16:12,355-0400 DEBUG ansible on_any args > localhost TASK: ovirt.ovirt.hosted_engine_setup : Activate storage domain > kwargs 2024-09-11 08:16:25,550-0400 DEBUG var changed: host "localhost" var > "ansible_play_hosts" type "<class 'list'>" value: "[]" > 2024-09-11 08:16:25,551-0400 DEBUG var changed: host "localhost" var > "ansible_play_batch" type "<class 'list'>" value: "[]" > 2024-09-11 08:16:25,551-0400 DEBUG var changed: host "localhost" var > "play_hosts" type "<class 'list'>" value: "[]" > 2024-09-11 08:16:25,551-0400 ERROR ansible failed { > "ansible_host": "localhost", > "ansible_playbook": > "/usr/share/ovirt-hosted-engine-setup/he_ansible/trigger_role.yml", > "ansible_result": { > "_ansible_no_log": false, > "changed": false, > "exception": "Traceback (most recent call last):\n File > \"/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\", > line 811, in main\n File > \"/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\", > line 660, in post_create_check\n File > \"/usr/lib64/python3.9/site-packages/ovirtsdk4/services.py\", line 3647, in > add\n return self._internal_add(storage_domain, headers, query, wait)\n > File \"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 232, > in _internal_add\n return future.wait() if wait else future\n File > \"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 55, in > wait\n return self._code(response)\n File > \"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 229, in > callback\n self._check_fault(response)\n File > \"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 132, in > _check_fault\n self._raise_error(response, body)\n File > \"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 118, in > _raise_error\n raise error\novirtsdk4.Error: Fault reason is \"Operation > Failed\". Fault detail is \"[Storage domain cannot be reached. Please ensure > it is accessible from the host(s).]\". HTTP response code is 400.\n", > "invocation": { > "module_args": { > "backup": null, > "comment": null, > "critical_space_action_blocker": null, > "data_center": "Default", > "description": null, > "destroy": null, > "discard_after_delete": null, > "domain_function": "data", > "fcp": null, > "fetch_nested": false, > "format": null, > "glusterfs": null, > "host": "lnxvirt01-p55.classe.cornell.edu", > "id": null, > "iscsi": null, > "localfs": null, > "managed_block_storage": null, > "name": "hosted_storage", > "nested_attributes": [], > "nfs": null, > "poll_interval": 3, > "posixfs": null, > "state": "present", > "storage_format": null, > "timeout": 180, > "wait": true, > "warning_low_space": null, > "wipe_after_delete": null > } > }, > "msg": "Fault reason is \"Operation Failed\". Fault detail is > \"[Storage domain cannot be reached. Please ensure it is accessible from the > host(s).]\". HTTP response code is 400." > }, > "ansible_task": "Activate storage domain", > "ansible_type": "task", > "status": "FAILED", > "task_duration": 13 > } > 2024-09-11 08:16:25,551-0400 DEBUG ansible on_any args > <ansible.executor.task_result.TaskResult object at 0x7fac823de1f0> kwargs > ignore_errors:None 2024-09-11 08:16:25,552-0400 INFO ansible stats { > "ansible_playbook": > "/usr/share/ovirt-hosted-engine-setup/he_ansible/trigger_role.yml", > "ansible_playbook_duration": "00:59 Minutes", > "ansible_result": "type: <class 'dict'>\nstr: {'localhost': {'ok': 20, > 'failures': 1, 'unreachable': 0, 'changed': 1, 'skipped': 9, 'rescued': 0, > 'ignored': 0}}", > "ansible_type": "finish", > "status": "FAILED" > } > 2024-09-11 08:16:25,552-0400 INFO SUMMARY: > Duration Task Name > -------- -------- > [ < 1 sec ] Execute just a specific set of steps > [ 00:03 ] Force facts gathering > [ 00:01 ] Check local VM dir stat > [ 00:01 ] Obtain SSO token using username/password credentials > [ 00:01 ] Fetch host facts > [ < 1 sec ] Fetch cluster ID > [ 00:01 ] Fetch cluster facts > [ 00:01 ] Fetch Datacenter facts > [ < 1 sec ] Fetch Datacenter ID > [ < 1 sec ] Fetch Datacenter name > [ < 1 sec ] Fetch cluster name > [ < 1 sec ] Fetch cluster version > [ < 1 sec ] Set storage_format > [ 00:15 ] Add iSCSI storage domain > [ 00:01 ] Get storage domain details > [ 00:01 ] Find the appliance OVF > [ 00:01 ] Get ovf data > [ < 1 sec ] Get disk size from ovf data > [ < 1 sec ] Get required size > [ FAILED ] Activate storage domain > 2024-09-11 08:16:25,553-0400 DEBUG ansible on_any args > <ansible.executor.stats.AggregateStats object at 0x7fac83c8a700> kwargs > ——— > > And here’s from the "ovirt-hosted-engine-setup-20240910222757-5ru4mo.log”: > ——— > 2024-09-11 08:16:12,144-0400 INFO > otopi.ovirt_hosted_engine_setup.ansible_utils > ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : > Activate storage domain] > 2024-09-11 08:16:25,264-0400 DEBUG > otopi.ovirt_hosted_engine_setup.ansible_utils > ansible_utils._process_output:109 {'exception': 'Traceback (most recent call > last):\n File > "/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py", > line 811, in main\n File > "/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py", > line 660, in post_create_check\n File > "/usr/lib64/python3.9/site-packages/ovirtsdk4/services.py", line 3647, in > add\n return self._internal_add(storage_domain, headers, query, wait)\n > File "/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 232, in > _internal_add\n return future.wait() if wait else future\n File > "/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 55, in wait\n > return self._code(response)\n File > "/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 229, in > callback\n self._check_fault(response)\n File > "/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 132, in > _check_fault\n self._raise_error(response, body)\n File > "/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 118, in > _raise_error\n raise error\novirtsdk4.Error: Fault reason is "Operation > Failed". Fault detail is "[Storage domain cannot be reached. Please ensure it > is accessible from the host(s).]". HTTP response code is 400.\n', 'msg': > 'Fault reason is "Operation Failed". Fault detail is "[Storage domain cannot > be reached. Please ensure it is accessible from the host(s).]". HTTP response > code is 400.', 'invocation': {'module_args': {'host': > 'lnxvirt01-p55.classe.cornell.edu', 'data_center': 'Default', 'name': > 'hosted_storage', 'wait': True, 'state': 'present', 'timeout': 180, > 'poll_interval': 3, 'fetch_nested': False, 'nested_attributes': [], > 'domain_function': 'data', 'id': None, 'description': None, 'comment': None, > 'localfs': None, 'nfs': None, 'iscsi': None, 'managed_block_storage': None, > 'posixfs': None, 'glusterfs': None, 'fcp': None, 'wipe_after_delete': None, > 'backup': None, 'critical_space_action_blocker': None, 'warning_low_space': > None, 'destroy': None, 'format': None, 'discard_after_delete': None, > 'storage_format': None}}, '_ansible_no_log': False, 'changed': False} > 2024-09-11 08:16:25,364-0400 ERROR > otopi.ovirt_hosted_engine_setup.ansible_utils > ansible_utils._process_output:113 ovirtsdk4.Error: Fault reason is "Operation > Failed". Fault detail is "[Storage domain cannot be reached. Please ensure it > is accessible from the host(s).]". HTTP response code is 400. > 2024-09-11 08:16:25,464-0400 ERROR > otopi.ovirt_hosted_engine_setup.ansible_utils > ansible_utils._process_output:113 fatal: [localhost]: FAILED! => {"changed": > false, "msg": "Fault reason is \"Operation Failed\". Fault detail is > \"[Storage domain cannot be reached. Please ensure it is accessible from the > host(s).]\". HTTP response code is 400."} > 2024-09-11 08:16:25,765-0400 DEBUG > otopi.ovirt_hosted_engine_setup.ansible_utils > ansible_utils._process_output:109 PLAY RECAP [localhost] : ok: 20 changed: 1 > unreachable: 0 skipped: 9 failed: 1 > 2024-09-11 08:16:25,866-0400 DEBUG > otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:226 > ansible-playbook rc: 2 > 2024-09-11 08:16:25,866-0400 DEBUG > otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:233 > ansible-playbook stdout: > 2024-09-11 08:16:25,867-0400 DEBUG > otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:236 > ansible-playbook stderr: > 2024-09-11 08:16:25,867-0400 DEBUG otopi.plugins.otopi.dialog.human > human.queryString:174 query OVEHOSTED_STORAGE_DOMAIN_TYPE > 2024-09-11 08:16:25,868-0400 DEBUG otopi.plugins.otopi.dialog.human > dialog.__logString:204 DIALOG:SEND Please specify the storage > you would like to use (glusterfs, iscsi, fc, nfs)[nfs]: > ——— > > _______________________________________________ > Users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/[email protected]/message/J2OHANJ2NNM7T5U4W62DKS6NTSCGOTJ4/ _______________________________________________ Users mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/4S62OBX3GVKYJYLDWFYMDWKCEXNUCMFL/
