On Fri, Jan 25, 2019 at 3:55 PM Nir Soffer <nsof...@redhat.com> wrote:
> On Fri, Jan 25, 2019 at 3:18 PM Jorick Astrego <jor...@netbulae.eu> wrote: > >> Hi, >> >> We're having problems mounting the preexisting 3.12 glusterfs storage >> domains in ovirt node ng 4.3.0 rc2. >> >> Getting >> >> There are no iptables blocks on the storage network, the ip's are >> pingable bothe ways. I can telnet to the glusterfs ports and I see no >> messages in the logs of the glusterfs servers. >> >> When I try the mount command manually it hangs for ever: >> >> /usr/bin/mount -t glusterfs -o backup-volfile-servers=*.*.*.*:*.*.*.* >> *.*.*.*:/sdd8 /mnt/temp >> >> I haven't submitted a bug yet.... >> >> from supervdsm.log >> >> MainProcess|jsonrpc/2::DEBUG::2019-01-25 >> 13:42:45,282::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) >> call volumeInfo with (u'sdd8', u'*.*.*.*') {} >> MainProcess|jsonrpc/2::DEBUG::2019-01-25 >> 13:42:45,282::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list >> 0-63 /usr/sbin/gluster --mode=script volume info --remote-host=*.*.*.* sdd8 >> --xml (cwd None) >> MainProcess|jsonrpc/2::DEBUG::2019-01-25 >> 13:44:45,399::commands::219::root::(execCmd) FAILED: <err> = ''; <rc> = 1 >> MainProcess|jsonrpc/2::DEBUG::2019-01-25 >> 13:44:45,399::logutils::319::root::(_report_stats) ThreadedHandler is ok >> in the last 120 seconds (max pending: 2) >> > > This looks like > https://bugzilla.redhat.com/show_bug.cgi?id=1666123#c18 > > We should see "ThreadedHandler is ok" every 60 seconds when using debug > log level. > > Looks like your entire supervdsmd process was hang for 120 seconds. > > >> MainProcess|jsonrpc/2::ERROR::2019-01-25 >> 13:44:45,399::supervdsm_server::104::SuperVdsm.ServerCallback::(wrapper) >> Error in volumeInfo >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line >> 102, in wrapper >> res = func(*args, **kwargs) >> File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 529, >> in volumeInfo >> xmltree = _execGlusterXml(command) >> File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 131, >> in _execGlusterXml >> return _getTree(rc, out, err) >> File "/usr/lib/python2.7/site-packages/vdsm/gluster/cli.py", line 112, >> in _getTree >> raise ge.GlusterCmdExecFailedException(rc, out, err) >> GlusterCmdExecFailedException: Command execution failed >> error: E >> r >> r >> o >> r >> >> : >> >> R >> e >> q >> u >> e >> s >> t >> >> t >> i >> m >> e >> d >> >> o >> u >> t >> > Looks like side effect of > https://gerrit.ovirt.org/c/94784/ > > GlusterException assumes that it accept list of lines, but we started to > raise > strings. The class should be fixed to handle strings. > Fixed in https://gerrit.ovirt.org/c/97316/ I think we need this in 4.2.8. Denis, please check. > >> >> return code: 1 >> MainProcess|jsonrpc/2::DEBUG::2019-01-25 >> 13:44:45,400::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) >> call mount with (<vdsm.supervdsm_server._SuperVdsm object at >> 0x7f6eb8d0a2d0>, u'*.*.*.*:/sdd8', >> u'/rhev/data-center/mnt/glusterSD/*.*.*.*:_sdd8') {'vfstype': u'glusterfs', >> 'mntOpts': u'backup-volfile-servers=*.*.*.*:*.*.*.*', 'cgroup': >> 'vdsm-glusterfs'} >> MainProcess|jsonrpc/2::DEBUG::2019-01-25 >> 13:44:45,400::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list >> 0-63 /usr/bin/systemd-run --scope --slice=vdsm-glusterfs /usr/bin/mount -t >> glusterfs -o backup-volfile-servers=*.*.*.*:*.*.*.* *.*.*.*:/sdd8 >> /rhev/data-center/mnt/glusterSD/*.*.*.*:_sdd8 (cwd None) >> MainProcess|jsonrpc/0::DEBUG::2019-01-25 >> 13:45:02,884::commands::219::root::(execCmd) FAILED: <err> = 'Running scope >> as unit run-38676.scope.\nMount failed. Please check the log file for more >> details.\n'; <rc> = 1 >> MainProcess|jsonrpc/0::ERROR::2019-01-25 >> 13:45:02,884::supervdsm_server::104::SuperVdsm.ServerCallback::(wrapper) >> Error in mount >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line >> 102, in wrapper >> res = func(*args, **kwargs) >> File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line >> 144, in mount >> cgroup=cgroup) >> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line >> 277, in _mount >> _runcmd(cmd) >> File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line >> 305, in _runcmd >> raise MountError(rc, b";".join((out, err))) >> MountError: (1, ';Running scope as unit run-38676.scope.\nMount failed. >> Please check the log file for more details.\n') >> > > The mount failure is probably related to glusterfs. There are glusterfs > logs on the host that > can give more info on this error. > >> MainProcess|jsonrpc/0::DEBUG::2019-01-25 >> 13:45:02,894::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) >> call volumeInfo with (u'ssd9', u'*.*.*.*') {} >> MainProcess|jsonrpc/0::DEBUG::2019-01-25 >> 13:45:02,894::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list >> 0-63 /usr/sbin/gluster --mode=script volume info --remote-host=*.*.*.* ssd9 >> --xml (cwd None) >> >> >> from vdsm.log >> >> 2019-01-25 13:46:03,519+0100 WARN (vdsm.Scheduler) [Executor] Worker >> blocked: <Worker name=jsonrpc/2 running <Task <JsonRpcTask {'params': >> {u'connectionParams': [{u'mnt_options': >> u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'6b6b7899-c82b-4417-b453-0b3b0ac11deb', u'connection': u'*.*.*.*:ssd4', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, >> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'b036005a-d44d-4689-a8c3-13e1bbf55af7', u'connection': u'*.*.*.*:ssd5', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, >> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'40d191b0-b7f8-48f9-bf6f-327275f51fef', u'connection': u'*.*.*.*:ssd6', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, >> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'26fbd2d6-6f25-4520-ab7f-15e9001f07b9', u'connection': u'*.*.*.*:/hdd2', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, >> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'f11fed97-513a-4a10-b85c-2afe68f42608', u'connection': u'*.*.*.*:/ssd3', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, >> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'f20b8691-528e-4e38-89ad-1e27684dee8b', u'connection': u'*.*.*.*:/sdd8', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, >> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'188e71dc-3d81-43d3-b930-238a4c6711e6', u'connection': u'*.*.*.*:/ssd9', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], >> u'storagepoolID': u'00000001-0001-0001-0001-000000000043', u'domainType': >> 7}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': >> u'581e2ad3-0682-4d44-95b4-bdc088b45f66'} at 0x7f9be815c850> timeout=60, >> duration=1260.00 at 0x7f9be815ca10> task#=98 at 0x7f9be83bb750>, traceback: >> File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap >> self.__bootstrap_inner() >> File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner >> self.run() >> File: "/usr/lib64/python2.7/threading.py", line 765, in run >> self.__target(*self.__args, **self.__kwargs) >> File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line >> 195, in run >> ret = func(*args, **kwargs) >> File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in >> _run >> self._execute_task() >> File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in >> _execute_task >> task() >> File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in >> __call__ >> self._callable() >> File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, >> in __call__ >> self._handler(self._ctx, self._req) >> File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, >> in _serveRequest >> response = self._handle_request(req, ctx) >> File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, >> in _handle_request >> res = method(**params) >> File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in >> _dynamicMethod >> result = fn(*methodArgs) >> File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1103, in >> connectStorageServer >> connectionParams) >> File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line >> 72, in wrapper >> result = ctask.prepare(func, *args, **kwargs) >> File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, >> in wrapper >> return m(self, *a, **kw) >> File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, >> in prepare >> result = self._run(func, *args, **kwargs) >> File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, >> in _run >> return fn(*args, **kargs) >> File: "<string>", line 2, in connectStorageServer >> File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in >> method >> ret = func(*args, **kwargs) >> File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2410, >> in connectStorageServer >> conObj.connect() >> >> from messages: >> >> Jan 25 13:49:07 node9 vdsm[31968]: WARN Worker blocked: <Worker >> name=jsonrpc/3 running <Task <JsonRpcTask {'params': {u'connectionParams': >> [{u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'6b6b7899-c82b-4417-b453-0b3b0ac11deb', u'connection': >> u'192.168.99.15:ssd4', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, >> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'b036005a-d44d-4689-a8c3-13e1bbf55af7', u'connection': u'*.*.*.*:ssd5', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, >> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'40d191b0-b7f8-48f9-bf6f-327275f51fef', u'connection': u'*.*.*.*:ssd6', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, >> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'26fbd2d6-6f25-4520-ab7f-15e9001f07b9', u'connection': u'*.*.*.*:/hdd2', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, >> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'f11fed97-513a-4a10-b85c-2afe68f42608', u'connection': u'*.*.*.*:/ssd3', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, >> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'f20b8691-528e-4e38-89ad-1e27684dee8b', u'connection': u'*.*.*.*:/sdd8', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}, >> {u'mnt_options': u'backup-volfile-servers=*.*.*.*:*.*.*.*', u'id': >> u'188e71dc-3d81-43d3-b930-238a4c6711e6', u'connection': u'*.*.*.*:/ssd9', >> u'iqn': u'', u'user': u'', u'tpgt': u'1', u'ipv6_enabled': u'false', >> u'vfs_type': u'glusterfs', u'password': '********', u'port': u''}], >> u'storagepoolID': u'00000001-0001-0001-0001-000000000043', u'domainType': >> 7}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': >> u'918f4d06-ca89-4ec0-a396-3407f1bdb8f9'} at 0x7f9be82ff250> timeout=60, >> duration=840.00 at 0x7f9be82ffa50> task#=93 at 0x7f9be83bba10>, >> traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in >> __bootstrap#012 self.__bootstrap_inner()#012File: >> "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 >> self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in >> run#012 self.__target(*self.__args, **self.__kwargs)#012File: >> "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in >> run#012 ret = func(*args, **kwargs)#012File: >> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 >> self._execute_task()#012File: >> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in >> _execute_task#012 task()#012File: >> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in >> __call__#012 self._callable()#012File: >> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in >> __call__#012 self._handler(self._ctx, self._req)#012File: >> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in >> _serveRequest#012 response = self._handle_request(req, ctx)#012File: >> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in >> _handle_request#012 res = method(**params)#012File: >> "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in >> _dynamicMethod#012 result = fn(*methodArgs)#012File: >> "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1103, in >> connectStorageServer#012 connectionParams)#012File: >> "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 72, in >> wrapper#012 result = ctask.prepare(func, *args, **kwargs)#012File: >> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in >> wrapper#012 return m(self, *a, **kw)#012File: >> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in >> prepare#012 result = self._run(func, *args, **kwargs)#012File: >> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in >> _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in >> connectStorageServer#012File: >> "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in >> method#012 ret = func(*args, **kwargs)#012File: >> "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2410, in >> connectStorageServer#012 conObj.connect()#012File: >> "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 172, >> in connect#012 self._mount.mount(self.options, self._vfsType, >> cgroup=self.CGROUP)#012File: >> "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in >> mount#012 cgroup=cgroup)#012File: >> "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in >> __call__#012 return callMethod()#012File: >> "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in >> <lambda>#012 **kwargs)#012File: "<string>", line 2, in mount#012File: >> "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in >> _callmethod#012 kind, result = conn.recv() >> > This matches the theory of blocked supervdsmd. > > Met vriendelijke groet, With kind regards, >> >> Jorick Astrego >> >> *Netbulae Virtualization Experts * >> ------------------------------ >> Tel: 053 20 30 270 i...@netbulae.eu Staalsteden 4-3A KvK 08198180 >> Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 >> ------------------------------ >> >> _______________________________________________ >> Users mailing list -- users@ovirt.org >> To unsubscribe send an email to users-le...@ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/UEFI2DXVP4D4B4J5WN2DL5SBSZSIOHFD/ >> >
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WLR3HKII4I3RIYIEQP6XG4RHCA47TXZM/