On Fri, Jan 25, 2019 at 3:55 PM Nir Soffer <nsof...@redhat.com> wrote:
> On Fri, Jan 25, 2019 at 3:18 PM Jorick Astrego <jor...@netbulae.eu> wrote:
>
>> Hi,
>>
>> We're having problems mounting the preexisting 3.12 glusterfs storage
>> domains in ovirt node ng 4.3.0 rc2.
>>
>> Getting
>>
>> There are no iptables blocks on the storage network, the ip's are
>> pingable bothe ways. I can telnet to the glusterfs ports and I see no
>> messages in the logs of the glusterfs servers.
>>
>> When I try the mount command manually it hangs for ever:
>>
>> /usr/bin/mount -t glusterfs -o backup-volfile-servers=*.*.*.*:*.*.*.*
>> *.*.*.*:/sdd8 /mnt/temp
>>
>> I haven't submitted a bug yet....
>>
>> from supervdsm.log
>>
>> MainProcess|jsonrpc/2::DEBUG::2019-01-25
>> 13:42:45,282::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper)
>> call volumeInfo with (u'sdd8', u'*.*.*.*') {}
>> MainProcess|jsonrpc/2::DEBUG::2019-01-25
>> 13:42:45,282::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list
>> 0-63 /usr/sbin/gluster --mode=script volume info --remote-host=*.*.*.* sdd8
>> --xml (cwd None)
>> MainProcess|jsonrpc/2::DEBUG::2019-01-25
>> 13:44:45,399::commands::219::root::(execCmd) FAILED: <err> = ''; <rc> = 1
>> MainProcess|jsonrpc/2::DEBUG::2019-01-25
>> 13:44:45,399::logutils::319::root::(_report_stats) ThreadedHandler is ok
>> in the last 120 seconds (max pending: 2)
>>
>
> This looks like
> https://bugzilla.redhat.com/show_bug.cgi?id=1666123#c18
>
> We should see "ThreadedHandler is ok" every 60 seconds when using debug
> log level.
>
> Looks like your entire supervdsmd process was hang for 120 seconds.
>
>
>> MainProcess|jsonrpc/2::ERROR::2019-01-25
>> 13:44:45,399::supervdsm_server::104::SuperVdsm.ServerCallback::(wrapper)
>> Error in volumeInfo
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line
>> 102, in wrapper
>> res = func(*args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 529,
>> in volumeInfo
>> xmltree = _execGlusterXml(command)
>> File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 131,
>> in _execGlusterXml
>> return _getTree(rc, out, err)
>> File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 112,
>> in _getTree
>> raise ge.GlusterCmdExecFailedException(rc, out, err)
>> GlusterCmdExecFailedException: Command execution failed
>> error: E
>> r
>> r
>> o
>> r
>>
>> :
>>
>> R
>> e
>> q
>> u
>> e
>> s
>> t
>>
>> t
>> i
>> m
>> e
>> d
>>
>> o
>> u
>> t
>>
> Looks like side effect of
> https://gerrit.ovirt.org/c/94784/
>
> GlusterException assumes that it accept list of lines, but we started to
> raise
> strings. The class should be fixed to handle strings.
>
Fixed in https://gerrit.ovirt.org/c/97316/
I think we need this in 4.2.8.
Denis, please check.
>
>>
>> return code: 1
>> MainProcess|jsonrpc/2::DEBUG::2019-01-25
>> 13:44:45,400::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper)
>> call mount with (<vdsm.supervdsm_server._SuperVdsm object at
>> 0x7f6eb8d0a2d0>, u'*.*.*.*:/sdd8',
>> u'/rhev/data-center/mnt/glusterSD/*.*.*.*:_sdd8') {'vfstype': u'glusterfs',
>> 'mntOpts': u'backup-volfile-servers=*.*.*.*:*.*.*.*', 'cgroup':
>> 'vdsm-glusterfs'}
>> MainProcess|jsonrpc/2::DEBUG::2019-01-25
>> 13:44:45,400::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list
>> 0-63 /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount -t
>> glusterfs -o backup-volfile-servers=*.*.*.*:*.*.*.* *.*.*.*:/sdd8
>> /rhev/data-center/mnt/glusterSD/*.*.*.*:_sdd8 (cwd None)
>> MainProcess|jsonrpc/0::DEBUG::2019-01-25
>> 13:45:02,884::commands::219::root::(execCmd) FAILED: <err> = 'Running scope
>> as unit run-38676.scope.\nMount failed. Please check the log file for more
>> details.\n'; <rc> = 1
>> MainProcess|jsonrpc/0::ERROR::2019-01-25
>> 13:45:02,884::supervdsm_server::104::SuperVdsm.ServerCallback::(wrapper)
>> Error in mount
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line
>> 102, in wrapper
>> res = func(*args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line
>> 144, in mount
>> cgroup=cgroup)
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line
>> 277, in _mount
>> _runcmd(cmd)
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line
>> 305, in _runcmd
>> raise MountError(rc, b";".join((out, err)))
>> MountError: (1, ';Running scope as unit run-38676.scope.\nMount failed.
>> Please check the log file for more details.\n')
>>
>
> The mount failure is probably related to glusterfs. There are glusterfs
> logs on the host that
> can give more info on this error.
>
>> MainProcess|jsonrpc/0::DEBUG::2019-01-25
>> 13:45:02,894::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper)
>> call volumeInfo with (u'ssd9', u'*.*.*.*') {}
>> MainProcess|jsonrpc/0::DEBUG::2019-01-25
>> 13:45:02,894::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list
>> 0-63 /usr/sbin/gluster --mode=script volume info --remote-host=*.*.*.* ssd9
>> --xml (cwd None)
>>
>>
>> from vdsm.log
>>
>> 2019-01-25 13:46:03,519+0100 WARN (vdsm.Scheduler) [Executor] Worker
>> blocked: <Worker name=jsonrpc/2 running <Task <JsonRpcTask {'params':
>> {u'connectionParams': [{u'mnt_options':
>> u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'6b6b7899-c82b-4417-b453-0b3b0ac11deb', u'connection': u'*.*.*.*:ssd4',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''},
>> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'b036005a-d44d-4689-a8c3-13e1bbf55af7', u'connection': u'*.*.*.*:ssd5',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''},
>> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'40d191b0-b7f8-48f9-bf6f-327275f51fef', u'connection': u'*.*.*.*:ssd6',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''},
>> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'26fbd2d6-6f25-4520-ab7f-15e9001f07b9', u'connection': u'*.*.*.*:/hdd2',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''},
>> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'f11fed97-513a-4a10-b85c-2afe68f42608', u'connection': u'*.*.*.*:/ssd3',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''},
>> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'f20b8691-528e-4e38-89ad-1e27684dee8b', u'connection': u'*.*.*.*:/sdd8',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''},
>> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'188e71dc-3d81-43d3-b930-238a4c6711e6', u'connection': u'*.*.*.*:/ssd9',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}],
>> u'storagepoolID': u'00000001-0001-0001-0001-000000000043', u'domainType':
>> 7}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id':
>> u'581e2ad3-0682-4d44-95b4-bdc088b45f66'} at 0x7f9be815c850> timeout=60,
>> duration=1260.00 at 0x7f9be815ca10> task#=98 at 0x7f9be83bb750>, traceback:
>> File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
>> self.__bootstrap_inner()
>> File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
>> self.run()
>> File: "/usr/lib64/python2.7/threading.py", line 765, in run
>> self.__target(*self.__args, **self.__kwargs)
>> File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line
>> 195, in run
>> ret = func(*args, **kwargs)
>> File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in
>> _run
>> self._execute_task()
>> File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
>> _execute_task
>> task()
>> File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
>> __call__
>> self._callable()
>> File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262,
>> in __call__
>> self._handler(self._ctx, self._req)
>> File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305,
>> in _serveRequest
>> response = self._handle_request(req, ctx)
>> File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345,
>> in _handle_request
>> res = method(**params)
>> File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in
>> _dynamicMethod
>> result = fn(*methodArgs)
>> File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1103, in
>> connectStorageServer
>> connectionParams)
>> File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line
>> 72, in wrapper
>> result = ctask.prepare(func, *args, **kwargs)
>> File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108,
>> in wrapper
>> return m(self, *a, **kw)
>> File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179,
>> in prepare
>> result = self._run(func, *args, **kwargs)
>> File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882,
>> in _run
>> return fn(*args, **kargs)
>> File: "<string>", line 2, in connectStorageServer
>> File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
>> method
>> ret = func(*args, **kwargs)
>> File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2410,
>> in connectStorageServer
>> conObj.connect()
>>
>> from messages:
>>
>> Jan 25 13:49:07 node9 vdsm[31968]: WARN Worker blocked: <Worker
>> name=jsonrpc/3 running <Task <JsonRpcTask {'params': {u'connectionParams':
>> [{u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'6b6b7899-c82b-4417-b453-0b3b0ac11deb', u'connection':
>> u'192.168.99.15:ssd4',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''},
>> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'b036005a-d44d-4689-a8c3-13e1bbf55af7', u'connection': u'*.*.*.*:ssd5',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''},
>> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'40d191b0-b7f8-48f9-bf6f-327275f51fef', u'connection': u'*.*.*.*:ssd6',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''},
>> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'26fbd2d6-6f25-4520-ab7f-15e9001f07b9', u'connection': u'*.*.*.*:/hdd2',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''},
>> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'f11fed97-513a-4a10-b85c-2afe68f42608', u'connection': u'*.*.*.*:/ssd3',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''},
>> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'f20b8691-528e-4e38-89ad-1e27684dee8b', u'connection': u'*.*.*.*:/sdd8',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''},
>> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id':
>> u'188e71dc-3d81-43d3-b930-238a4c6711e6', u'connection': u'*.*.*.*:/ssd9',
>> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false',
>> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}],
>> u'storagepoolID': u'00000001-0001-0001-0001-000000000043', u'domainType':
>> 7}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id':
>> u'918f4d06-ca89-4ec0-a396-3407f1bdb8f9'} at 0x7f9be82ff250> timeout=60,
>> duration=840.00 at 0x7f9be82ffa50> task#=93 at 0x7f9be83bba10>,
>> traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in
>> __bootstrap#012 self.__bootstrap_inner()#012File:
>> "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012
>> self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in
>> run#012 self.__target(*self.__args, **self.__kwargs)#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in
>> run#012 ret = func(*args, **kwargs)#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012
>> self._execute_task()#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
>> _execute_task#012 task()#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
>> __call__#012 self._callable()#012File:
>> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
>> __call__#012 self._handler(self._ctx, self._req)#012File:
>> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
>> _serveRequest#012 response = self._handle_request(req, ctx)#012File:
>> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in
>> _handle_request#012 res = method(**params)#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in
>> _dynamicMethod#012 result = fn(*methodArgs)#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1103, in
>> connectStorageServer#012 connectionParams)#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 72, in
>> wrapper#012 result = ctask.prepare(func, *args, **kwargs)#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
>> wrapper#012 return m(self, *a, **kw)#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in
>> prepare#012 result = self._run(func, *args, **kwargs)#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
>> _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in
>> connectStorageServer#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
>> method#012 ret = func(*args, **kwargs)#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2410, in
>> connectStorageServer#012 conObj.connect()#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 172,
>> in connect#012 self._mount.mount(self.options, self._vfsType,
>> cgroup=self.CGROUP)#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in
>> mount#012 cgroup=cgroup)#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in
>> __call__#012 return callMethod()#012File:
>> "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in
>> <lambda>#012 **kwargs)#012File: "<string>", line 2, in mount#012File:
>> "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in
>> _callmethod#012 kind, result = conn.recv()
>>
> This matches the theory of blocked supervdsmd.
>
> Met vriendelijke groet, With kind regards,
>>
>> Jorick Astrego
>>
>> *Netbulae Virtualization Experts *
>> ------------------------------
>> Tel: 053 20 30 270 i...@netbulae.eu Staalsteden 4-3A KvK 08198180
>> Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01
>> ------------------------------
>>
>> _______________________________________________
>> Users mailing list -- users@ovirt.org
>> To unsubscribe send an email to users-le...@ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/UEFI2DXVP4D4B4J5WN2DL5SBSZSIOHFD/
>>
>
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/WLR3HKII4I3RIYIEQP6XG4RHCA47TXZM/
