Hi team,
We have installed ovirt 4.4.
We have a self hosted Engine setup in the environment which has 1 hosted engine
on top of 1 deployment-host.
Goal:
We want to create a storage domain of POSIX compliant type for mounting a ceph
based infrastructure.
We have done SRV based resolution in our DNS server but we are unable to create
a storage domain.
Issue:
We are passing the following information:
path: :/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9
vfs-type: ceph
mounting option:rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA=
We get the following errors:
====================vdsm.log==================================
2023-04-20 11:26:30,318+0530 INFO (jsonrpc/7) [storage.Mount] mounting
:/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8 at
/rhev/data-center/mnt/ :_volumes_xyz_conf_2ee9c2d0-873b-4d04-8c46-4c0da02787b8
(mount:207)
2023-04-20 11:26:30,384+0530 ERROR (jsonrpc/7) [storage.HSM] Could not connect
to storageServer (hsm:2374)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2371, in
connectStorageServer
conObj.connect()
File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line
180, in connect
six.reraise(t, v, tb)
File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line
171, in connect
self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)
File "/usr/lib/python3.6/site-packages/vdsm/storage/mount.py", line 210, in
mount
cgroup=cgroup)
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 56, in
__call__
return callMethod()
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 54, in
<lambda>
**kwargs)
File "<string>", line 2, in mount
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in
_callmethod
raise convert_to_error(kind, result)
vdsm.storage.mount.MountError: Command ['/usr/bin/mount', '-t', 'ceph', '-o',
'rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==', '
:/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8',
'/rhev/data-center/mnt/
:_volumes_xyz_conf_2ee9c2d0-873b-4d04-8c46-4c0da02787b8'] failed with rc=32
out=b'' err=b'mount error 3 = No such process\n'
2023-04-20 11:31:05,715+0530 ERROR (jsonrpc/3) [storage.Dispatcher] FINISH
connectStorageServer
error=:/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8 is not a valid
hosttail address: (dispatcher:87)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/common/network/address.py", line
42, in hosttail_split
raise ValueError
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 74,
in wrapper
result = ctask.prepare(func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 110, in
wrapper
return m(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1190, in
prepare
raise self.error
File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 884, in
_run
return fn(*args, **kargs)
File "<decorator-gen-117>", line 2, in connectStorageServer
File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method
ret = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2368, in
connectStorageServer
conObj = storageServer.ConnectionFactory.createConnection(conInfo)
File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line
741, in createConnection
return ctor(**params)
File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line
154, in __init__
self._remotePath = fileUtils.normalize_path(spec)
File "/usr/lib/python3.6/site-packages/vdsm/storage/fileUtils.py", line 98,
in normalize_path
host, tail = address.hosttail_split(path)
File "/usr/lib/python3.6/site-packages/vdsm/common/network/address.py", line
45, in hosttail_split
raise HosttailError('%s is not a valid hosttail address:' % hosttail)
vdsm.common.network.address.HosttailError:
:/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8 is not a valid hosttail
address:
2023-04-20 11:31:05,715+0530 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call
StoragePool.connectStorageServer failed (error 451) in 0.00 seconds
(__init__:312)
====================engine.log================================
2023-04-20 11:31:03,818+05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand]
(default task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] HostName =
deployment-host
2023-04-20 11:31:03,818+05 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand]
(default task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] Command
'ConnectStorageServerVDSCommand(HostName = deployment-host,
StorageServerConnectionManagementVDSParameters:{hostId='745b7584-0a43-47d9-985f-af0a0155e787',
storagePoolId='00000000-0000-0000-0000-000000000000', storageType='POSIXFS',
connectionList='[StorageServerConnections:{id='null',
connection=':/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8',
iqn='null', vfsType='ceph',
mountOptions='rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null',
netIfaceName='null'}]', sendNetworkEventOnFailure='true'})' execution failed:
VDSGenericException: VDSErrorException: Failed to ConnectStorageServerVDS,
error = Error storage server connection: ("domType=6,
spUUID=00000000-0000-0000-0000-000000000000, conList=[{'password': '********',
'vfs_type'
: 'ceph', 'port': '', 'mnt_options':
'rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==', 'iqn': '',
'connection': ':/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8',
'ipv6_enabled': 'false', 'id': '00000000-0000-0000-0000-000000000000', 'user':
'', 'tpgt': '1'}]",), code = 451
2023-04-20 11:31:03,818+05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand]
(default task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] FINISH,
ConnectStorageServerVDSCommand, return: , log id: 19c06f3f
2023-04-20 11:31:03,819+05 ERROR
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] Command
'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand'
failed: EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to ConnectStorageServerVDS,
error = Error storage server connection: ("domType=6,
spUUID=00000000-0000-0000-0000-000000000000, conList=[{'password': '********',
'vfs_type': 'ceph', 'port': '', 'mnt_options':
'rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==', 'iqn': '',
'connection': ':/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8',
'ipv6_enabled': 'false', 'id': '00000000-0000-0000-0000-000000000000', 'user':
'', 'tpgt': '1'}]",), code = 451 (Failed with error
StorageServerConnectionError and code 451)
2023-04-20 11:31:03,824+05 ERROR
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] Transaction
rolled-back for command
'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand'.
2023-04-20 11:31:03,847+05 INFO
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-29) [6d7913e2-83cf-450d-8746-40f1582d959d] Lock freed to object
'EngineLock:{exclusiveLocks='[:/volumes/xyz/conf/2ee9c2d0-873b-4d04-8c46-4c0da02787b8=STORAGE_CONNECTION]',
sharedLocks=''}'
2023-04-20 11:34:27,788+05 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28)
[63fbb9de] Lock Acquired to object
'EngineLock:{exclusiveLocks='[c6e0aa38-20ff-4be3-9b34-81a7b0fabb6a=PROVIDER]',
sharedLocks=''}'
2023-04-20 11:34:27,809+05 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28)
[63fbb9de] Running command: SyncNetworkProviderCommand internal: true.
2023-04-20 11:34:28,017+05 INFO
[org.ovirt.engine.core.sso.service.AuthenticationService] (default task-29) []
User admin@internal-authz with profile [internal] successfully logged in with
scopes: ovirt-app-api ovirt-ext=token-info:authz-search
ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate
ovirt-ext=token:password-access
===============================================================================
We tried creating a NFS type storage domain using the hostname of the nfs
server. This hostname is resolved using the dns server. We are able to create
this storage domain.
We also tried creating the POSIX compliant domain using the hostname of the
ceph-mon node but it is failing.
We are passing the following information:
path: ceph-node2:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9
vfs-type: ceph
mounting option:rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA=
In this scenario though the mounting is happening but the storage domain is not
created correctly.
we are getting the following errors:
====================vdsm.log==================================
2023-04-20 12:23:54,759+0530 INFO (jsonrpc/0) [api.virt] FINISH getStats
return={'status': {'code': 0, 'message': 'Done'}, 'statsList': [{'statusTime':
'3009297867', 'status': 'Up', 'vmId': '9ea50595-33cc-4fde-9cd2-ac35c4e97a2c',
'vmName': 'HostedEngine', 'vmType': 'kvm', 'kvmEnable': 'true', 'acpiEnable':
'true', 'elapsedTime': '71325', 'monitorResponse': '0', 'clientIp': '',
'timeOffset': '0', 'pauseCode': 'NOERR', 'cpuUser': '7.12', 'cpuSys': '1.80',
'cpuUsage': '1264730000000', 'network': {'vnet1': {'macAddr':
'00:16:3e:54:7f:23', 'name': 'vnet1', 'speed': '1000', 'state': 'unknown',
'rxErrors': '0', 'rxDropped': '0', 'txErrors': '0', 'txDropped': '0', 'rx':
'94068359', 'tx': '107873946', 'sampleTime': 861814.22111595}}, 'disks':
{'sdc': {'truesize': '0', 'apparentsize': '0', 'readLatency': '0',
'writeLatency': '0', 'flushLatency': '0', 'writtenBytes': '0', 'writeOps': '0',
'readOps': '22', 'readBytes': '406', 'readRate': '0.0', 'writeRate': '0.0'},
'vda': {'truesize': '76436357
12', 'apparentsize': '53689188352', 'readLatency': '0', 'writeLatency':
'18963692.40625', 'flushLatency': '132326.26666666666', 'writtenBytes':
'1946506752', 'writeOps': '165176', 'readOps': '31194', 'readBytes':
'859866624', 'readRate': '0.0', 'writeRate': '14743.1258832571', 'imageID':
'bcd042b1-0978-43bd-bf24-eb1f554cd520'}}, 'balloonInfo': {'balloon_max':
'13489152', 'balloon_min': '13489152', 'balloon_cur': '13489152',
'balloon_target': '13489152', 'ballooning_enabled': True}, 'vcpuCount': '4',
'memoryStats': {'mem_total': '12887304', 'mem_unused': '7100564', 'mem_free':
'7994896', 'swap_in': 0, 'swap_out': 0, 'majflt': 0, 'minflt': 196, 'pageflt':
196}, 'displayInfo': [{'type': 'vnc', 'port': '5900', 'tlsPort': '',
'ipAddress': '10.0.1.47'}, {'type': 'spice', 'port': '5901', 'tlsPort': '5902',
'ipAddress': '10.0.1.47'}], 'hash': '-7465763413854713032', 'vmJobs': {},
'vcpuQuota': '-1', 'vcpuPeriod': 100000, 'username': 'root', 'session':
'Unknown', 'memUsage': '40', 'guestCPUCo
unt': -1, 'appsList': ('kernel-4.18.0-486.el8.x86_64',
'qemu-guest-agent-6.2.0'), 'guestIPs': '', 'guestFQDN':
'manager-hosted-engine.com', 'netIfaces': [{'hw': '00:00:00:00:00:00', 'inet':
['127.0.0.1'], 'inet6': ['::1'], 'name': 'lo'}, {'hw': '00:16:3e:54:7f:23',
'inet': ['10.0.1.48'], 'inet6': ['fe80::216:3eff:fe54:7f23'], 'name': 'eth0'}],
'disksUsage': [{'path': '/', 'total': '7505707008', 'used': '4872785920', 'fs':
'xfs'}, {'path': '/var', 'total': '21464350720', 'used': '1688195072', 'fs':
'xfs'}, {'path': '/home', 'total': '1063256064', 'used': '41271296', 'fs':
'xfs'}, {'path': '/tmp', 'total': '2136997888', 'used': '48943104', 'fs':
'xfs'}, {'path': '/var/log', 'total': '10726932480', 'used': '132661248', 'fs':
'xfs'}, {'path': '/var/log/audit', 'total': '1063256064', 'used': '42233856',
'fs': 'xfs'}, {'path': '/boot', 'total': '1063256064', 'used': '352448512',
'fs': 'xfs'}], 'guestName': 'manager-hosted-engine.com', 'guestOs':
'4.18.0-486.el8.x86_64', 'guestOsInfo': {'t
ype': 'linux', 'arch': 'x86_64', 'kernel': '4.18.0-486.el8.x86_64',
'distribution': 'CentOS Stream', 'version': '8', 'codename': ''},
'guestTimezone': {'offset': 330, 'zone': 'IST'}}]} from=::1,57200,
vmId=9ea50595-33cc-4fde-9cd2-ac35c4e97a2c (api:54)
2023-04-20 12:23:55,230+0530 INFO (jsonrpc/3) [vdsm.api] START
repoStats(domains=['963b7fd2-c32f-400d-9abf-ade3d702cb4b']) from=::1,57200,
task_id=0279b915-ac22-46d2-afbc-8466144d22d8 (api:48)
2023-04-20 12:23:55,231+0530 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats
return={'963b7fd2-c32f-400d-9abf-ade3d702cb4b': {'code': 0, 'lastCheck': '5.1',
'delay': '0.000864457', 'valid': True, 'version': 5, 'acquired': True,
'actual': True}} from=::1,57200, task_id=0279b915-ac22-46d2-afbc-8466144d22d8
(api:54)
2023-04-20 12:23:55,735+0530 ERROR (monitor/6ac48b4) [storage.Monitor] Error
checking domain 6ac48b40-a9dc-4475-b17e-247b6018abc1 (monitor:453)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 434, in
_checkDomainStatus
self.domain.selftest()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 48, in
__getattr__
return getattr(self.getRealDomain(), attrName)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 51, in
getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in
_realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 156, in
_findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 146, in
findDomain
return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
File "/usr/lib/python3.6/site-packages/vdsm/storage/nfsSD.py", line 136, in
findDomainPath
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not
exist: ('6ac48b40-a9dc-4475-b17e-247b6018abc1',)
2023-04-20 12:23:56,307+0530 ERROR (check/loop) [storage.Monitor] Error
checking path
/rhev/data-center/mnt/[abcd:abcd:abcd::51]:6789:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9/6ac48b40-a9dc-4475-b17e-247b6018abc1/dom_md/metadata
(monitor:511)
====================engine.log================================
2023-04-20 12:23:55,064+05 INFO
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Lock Acquired to
object
'EngineLock:{exclusiveLocks='[ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9=STORAGE_CONNECTION]',
sharedLocks=''}'
2023-04-20 12:23:55,118+05 INFO
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Running command:
AddStorageServerConnectionCommand internal: false. Entities affected : ID:
aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group
CREATE_STORAGE_DOMAIN with role type ADMIN
2023-04-20 12:23:55,119+05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand]
(default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] START,
ConnectStorageServerVDSCommand(HostName = deployment-host,
StorageServerConnectionManagementVDSParameters:{hostId='745b7584-0a43-47d9-985f-af0a0155e787',
storagePoolId='00000000-0000-0000-0000-000000000000', storageType='POSIXFS',
connectionList='[StorageServerConnections:{id='null',
connection='ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9',
iqn='null', vfsType='ceph',
mountOptions='rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==',
nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null',
netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 7e34045e
2023-04-20 12:23:55,223+05 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand]
(default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] FINISH,
ConnectStorageServerVDSCommand, return:
{00000000-0000-0000-0000-000000000000=100}, log id: 7e34045e
2023-04-20 12:23:55,248+05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default
task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] EVENT_ID:
STORAGE_DOMAIN_ERROR(996), The error message for connection
ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9
returned by VDSM was: General Exception
2023-04-20 12:23:55,248+05 ERROR
[org.ovirt.engine.core.bll.storage.connection.FileStorageHelper] (default
task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] The connection with details
'ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9'
failed because of error code '100' and error message is: general exception
2023-04-20 12:23:55,249+05 ERROR
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Command
'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand'
failed: EngineException: GeneralException (Failed with error GeneralException
and code 100)
2023-04-20 12:23:55,255+05 ERROR
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Transaction
rolled-back for command
'org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand'.
2023-04-20 12:23:55,278+05 INFO
[org.ovirt.engine.core.bll.storage.connection.AddStorageServerConnectionCommand]
(default task-37) [31e859ef-97d0-435c-81ba-9d29bf637527] Lock freed to object
'EngineLock:{exclusiveLocks='[ceph-node2.myhsc.com:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9=STORAGE_CONNECTION]',
sharedLocks=''}'
This also doesn't work.
We tried mounting the ceph-mon nodes cluster manually on the deployment host
CLI , we were able to mount the same.
sudo mount -t ceph :/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9
/rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9
-o rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA==
[root@deployment-host mnt]# df -kh
df: /run/user/0/gvfs: Transport endpoint is not connected
Filesystem
Size Used Avail Use%
Mounted on
[abcd:abcd:abcd::51]:6789,[abcd:abcd:abcd::52]:6789,[abcd:abcd:abcd::53]:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9
19G 0 19G 0%
/rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9
Queries:
1. Could you help us in creating a storage domain for the same from UI.
2. Do we have any process of doing the same using the CLI.
Thanks and Regards
Kushagra Gupta
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FSX5LHWQHUR6BYB5X62HSCQGBBBY4P4D/
