Hello,
We are attempting to move a hosted engine from an NFS to an iSCSI storage
domain, following the normal backup / restore procedure.
With a little intervention in the new hosted engine VM, everything seems to be
working with the new engine and the process gets to the point of trying to
create the new iSCSI hosted_storage domain. At this point, it fails with
"Storage domain cannot be reached. Please ensure it is accessible from the
host(s).”
The host itself does see the storage, and everything looks fine using iscsiadm
and multipath commands. However, if I look at the new hosted_storage domain in
the new engine, it’s stuck “unattached” and the “hosted-engine --deploy
--restore-from-file …” command loops at the "Please specify the storage you
would like to use” step.
Please see below for an excerpt of the output and logs, and let me know what
additional information I can provide.
Many thanks,
Devin
Here’s the output from the “hosted-engine --deploy --restore-from-file …”
command.
———
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : iSCSI login]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get iSCSI LUNs]
[ INFO ] ok: [localhost]
The following luns have been found on the requested target:
[1] mpathm 1024.0GiB IFT DS 3000 Series
status: free, paths: 8 active
Please select the destination LUN (1) [1]:
[ INFO ] iSCSI discard after delete is disabled
[ INFO ] Creating Storage Domain
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Execute just a specific set
of steps]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Force facts gathering]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Wait for the storage
interface to be up]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Check local VM dir stat]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce local VM dir
existence]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Obtain SSO token using
username/password credentials]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch host facts]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster ID]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster facts]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter facts]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter ID]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch Datacenter name]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster name]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch cluster version]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce cluster major version]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Enforce cluster minor version]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set storage_format]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add NFS storage domain]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add glusterfs storage domain]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add iSCSI storage domain]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add Fibre Channel storage
domain]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get storage domain details]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Find the appliance OVF]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get ovf data]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get disk size from ovf data]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get required size]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Remove unsuitable storage
domain]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Check storage domain free
space]
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Activate storage domain]
[ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is
"[Storage domain cannot be reached. Please ensure it is accessible from the
host(s).]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault
reason is \"Operation Failed\". Fault detail is \"[Storage domain cannot be
reached. Please ensure it is accessible from the host(s).]\". HTTP response
code is 400.”}
———
Here’s an excerpt from the
"ovirt-hosted-engine-setup-ansible-create_storage_domain-20240911081524-c7yrkf.log”:
———
2024-09-11 08:16:12,354-0400 INFO ansible task start {'status': 'OK',
'ansible_type': 'task', 'ansible_playbook':
'/usr/share/ovirt-hosted-engine-setup/he_ansible/trigger_role.yml',
'ansible_task': 'ovirt.ovirt.hosted_engine_setup : Activate storage domain'}
2024-09-11 08:16:12,354-0400 DEBUG ansible on_any args TASK:
ovirt.ovirt.hosted_engine_setup : Activate storage domain kwargs
is_conditional:False 2024-09-11 08:16:12,355-0400 DEBUG ansible on_any args
localhost TASK: ovirt.ovirt.hosted_engine_setup : Activate storage domain
kwargs 2024-09-11 08:16:25,550-0400 DEBUG var changed: host "localhost" var
"ansible_play_hosts" type "<class 'list'>" value: "[]"
2024-09-11 08:16:25,551-0400 DEBUG var changed: host "localhost" var
"ansible_play_batch" type "<class 'list'>" value: "[]"
2024-09-11 08:16:25,551-0400 DEBUG var changed: host "localhost" var
"play_hosts" type "<class 'list'>" value: "[]"
2024-09-11 08:16:25,551-0400 ERROR ansible failed {
"ansible_host": "localhost",
"ansible_playbook":
"/usr/share/ovirt-hosted-engine-setup/he_ansible/trigger_role.yml",
"ansible_result": {
"_ansible_no_log": false,
"changed": false,
"exception": "Traceback (most recent call last):\n File
\"/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\",
line 811, in main\n File
\"/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\",
line 660, in post_create_check\n File
\"/usr/lib64/python3.9/site-packages/ovirtsdk4/services.py\", line 3647, in
add\n return self._internal_add(storage_domain, headers, query, wait)\n
File \"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 232, in
_internal_add\n return future.wait() if wait else future\n File
\"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 55, in wait\n
return self._code(response)\n File
\"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 229, in
callback\n self._check_fault(response)\n File
\"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 132, in
_check_fault\n self._raise_error(response, body)\n File
\"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py\", line 118, in
_raise_error\n raise error\novirtsdk4.Error: Fault reason is \"Operation
Failed\". Fault detail is \"[Storage domain cannot be reached. Please ensure it
is accessible from the host(s).]\". HTTP response code is 400.\n",
"invocation": {
"module_args": {
"backup": null,
"comment": null,
"critical_space_action_blocker": null,
"data_center": "Default",
"description": null,
"destroy": null,
"discard_after_delete": null,
"domain_function": "data",
"fcp": null,
"fetch_nested": false,
"format": null,
"glusterfs": null,
"host": "lnxvirt01-p55.classe.cornell.edu",
"id": null,
"iscsi": null,
"localfs": null,
"managed_block_storage": null,
"name": "hosted_storage",
"nested_attributes": [],
"nfs": null,
"poll_interval": 3,
"posixfs": null,
"state": "present",
"storage_format": null,
"timeout": 180,
"wait": true,
"warning_low_space": null,
"wipe_after_delete": null
}
},
"msg": "Fault reason is \"Operation Failed\". Fault detail is
\"[Storage domain cannot be reached. Please ensure it is accessible from the
host(s).]\". HTTP response code is 400."
},
"ansible_task": "Activate storage domain",
"ansible_type": "task",
"status": "FAILED",
"task_duration": 13
}
2024-09-11 08:16:25,551-0400 DEBUG ansible on_any args
<ansible.executor.task_result.TaskResult object at 0x7fac823de1f0> kwargs
ignore_errors:None 2024-09-11 08:16:25,552-0400 INFO ansible stats {
"ansible_playbook":
"/usr/share/ovirt-hosted-engine-setup/he_ansible/trigger_role.yml",
"ansible_playbook_duration": "00:59 Minutes",
"ansible_result": "type: <class 'dict'>\nstr: {'localhost': {'ok': 20,
'failures': 1, 'unreachable': 0, 'changed': 1, 'skipped': 9, 'rescued': 0,
'ignored': 0}}",
"ansible_type": "finish",
"status": "FAILED"
}
2024-09-11 08:16:25,552-0400 INFO SUMMARY:
Duration Task Name
-------- --------
[ < 1 sec ] Execute just a specific set of steps
[ 00:03 ] Force facts gathering
[ 00:01 ] Check local VM dir stat
[ 00:01 ] Obtain SSO token using username/password credentials
[ 00:01 ] Fetch host facts
[ < 1 sec ] Fetch cluster ID
[ 00:01 ] Fetch cluster facts
[ 00:01 ] Fetch Datacenter facts
[ < 1 sec ] Fetch Datacenter ID
[ < 1 sec ] Fetch Datacenter name
[ < 1 sec ] Fetch cluster name
[ < 1 sec ] Fetch cluster version
[ < 1 sec ] Set storage_format
[ 00:15 ] Add iSCSI storage domain
[ 00:01 ] Get storage domain details
[ 00:01 ] Find the appliance OVF
[ 00:01 ] Get ovf data
[ < 1 sec ] Get disk size from ovf data
[ < 1 sec ] Get required size
[ FAILED ] Activate storage domain
2024-09-11 08:16:25,553-0400 DEBUG ansible on_any args
<ansible.executor.stats.AggregateStats object at 0x7fac83c8a700> kwargs
———
And here’s from the "ovirt-hosted-engine-setup-20240910222757-5ru4mo.log”:
———
2024-09-11 08:16:12,144-0400 INFO otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup :
Activate storage domain]
2024-09-11 08:16:25,264-0400 DEBUG
otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109
{'exception': 'Traceback (most recent call last):\n File
"/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py",
line 811, in main\n File
"/tmp/ansible_ovirt_storage_domain_payload_cne75az3/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py",
line 660, in post_create_check\n File
"/usr/lib64/python3.9/site-packages/ovirtsdk4/services.py", line 3647, in add\n
return self._internal_add(storage_domain, headers, query, wait)\n File
"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 232, in
_internal_add\n return future.wait() if wait else future\n File
"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 55, in wait\n
return self._code(response)\n File
"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 229, in
callback\n self._check_fault(response)\n File
"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 132, in
_check_fault\n self._raise_error(response, body)\n File
"/usr/lib64/python3.9/site-packages/ovirtsdk4/service.py", line 118, in
_raise_error\n raise error\novirtsdk4.Error: Fault reason is "Operation
Failed". Fault detail is "[Storage domain cannot be reached. Please ensure it
is accessible from the host(s).]". HTTP response code is 400.\n', 'msg': 'Fault
reason is "Operation Failed". Fault detail is "[Storage domain cannot be
reached. Please ensure it is accessible from the host(s).]". HTTP response code
is 400.', 'invocation': {'module_args': {'host':
'lnxvirt01-p55.classe.cornell.edu', 'data_center': 'Default', 'name':
'hosted_storage', 'wait': True, 'state': 'present', 'timeout': 180,
'poll_interval': 3, 'fetch_nested': False, 'nested_attributes': [],
'domain_function': 'data', 'id': None, 'description': None, 'comment': None,
'localfs': None, 'nfs': None, 'iscsi': None, 'managed_block_storage': None,
'posixfs': None, 'glusterfs': None, 'fcp': None, 'wipe_after_delete': None,
'backup': None, 'critical_space_action_blocker': None, 'warning_low_space':
None, 'destroy': None, 'format': None, 'discard_after_delete': None,
'storage_format': None}}, '_ansible_no_log': False, 'changed': False}
2024-09-11 08:16:25,364-0400 ERROR
otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113
ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Storage
domain cannot be reached. Please ensure it is accessible from the host(s).]".
HTTP response code is 400.
2024-09-11 08:16:25,464-0400 ERROR
otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is
\"Operation Failed\". Fault detail is \"[Storage domain cannot be reached.
Please ensure it is accessible from the host(s).]\". HTTP response code is
400."}
2024-09-11 08:16:25,765-0400 DEBUG
otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109
PLAY RECAP [localhost] : ok: 20 changed: 1 unreachable: 0 skipped: 9 failed: 1
2024-09-11 08:16:25,866-0400 DEBUG
otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:226
ansible-playbook rc: 2
2024-09-11 08:16:25,866-0400 DEBUG
otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:233
ansible-playbook stdout:
2024-09-11 08:16:25,867-0400 DEBUG
otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:236
ansible-playbook stderr:
2024-09-11 08:16:25,867-0400 DEBUG otopi.plugins.otopi.dialog.human
human.queryString:174 query OVEHOSTED_STORAGE_DOMAIN_TYPE
2024-09-11 08:16:25,868-0400 DEBUG otopi.plugins.otopi.dialog.human
dialog.__logString:204 DIALOG:SEND Please specify the storage
you would like to use (glusterfs, iscsi, fc, nfs)[nfs]:
———
_______________________________________________
Users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/[email protected]/message/J2OHANJ2NNM7T5U4W62DKS6NTSCGOTJ4/
