I restarted the host and it seems like it's running fine...However do it need to reboot other hosts or restart some services on other hosts as well clean this up. Because they still can see the Host ID=3 [root@iondelsvr10 vdsm]# hosted-engine --vm-status
--== Host iondelsvr10.iontrading.com (id: 1) status ==-- Host ID : 1 Host timestamp : 2672943 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : iondelsvr10.iontrading.com Local maintenance : False stopped : False crc32 : b737573e conf_on_shared_storage : True local_conf_timestamp : 2672943 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2672943 (Thu Feb 3 20:09:34 2022) host-id=1 score=3400 vm_conf_refresh_time=2672943 (Thu Feb 3 20:09:34 2022) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host iondelsvr11.iontrading.com (id: 2) status ==-- Host ID : 2 Host timestamp : 6255602 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : iondelsvr11.iontrading.com Local maintenance : False stopped : False crc32 : 5ff71c1a conf_on_shared_storage : True local_conf_timestamp : 6255602 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=6255602 (Thu Feb 3 20:09:24 2022) host-id=2 score=3400 vm_conf_refresh_time=6255602 (Thu Feb 3 20:09:24 2022) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host iondelsvr12.iontrading.com (id: 3) status ==-- Host ID : 3 Host timestamp : 734 Score : 1800 Engine status : unknown stale-data Hostname : iondelsvr12.iontrading.com Local maintenance : False stopped : False crc32 : 062e92e9 conf_on_shared_storage : True local_conf_timestamp : 734 Status up-to-date : False Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=734 (Fri Jan 28 17:35:18 2022) host-id=3 score=1800 vm_conf_refresh_time=734 (Fri Jan 28 17:35:18 2022) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host iondelsvr14.iontrading.com (id: 4) status ==-- Host ID : 4 Host timestamp : 2670 Score : 3400 Engine status : {"vm": "up", "health": "good", "detail": "Up"} Hostname : iondelsvr14.iontrading.com Local maintenance : False stopped : False crc32 : fdf46b17 conf_on_shared_storage : True local_conf_timestamp : 2670 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=2670 (Thu Feb 3 20:09:30 2022) host-id=4 score=3400 vm_conf_refresh_time=2670 (Thu Feb 3 20:09:30 2022) conf_on_shared_storage=True maintenance=False state=EngineUp stopped=False [root@iondelsvr10 vdsm]# On Thu, Feb 3, 2022 at 6:54 PM Ayansh Rocks <shashank123rast...@gmail.com> wrote: > VDSM Logs - > > ual': True}} from=internal, task_id=b017c5d2-a941-4ccd-af02-3fac0a767df5 > (api:54) > 2022-02-03 18:53:53,368+0530 ERROR (periodic/3) > [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host > 3: received 0 of 512 expected bytes (client:137) > 2022-02-03 18:53:54,529+0530 INFO (jsonrpc/5) [api.host] START getStats() > from=::1,38990 (api:48) > 2022-02-03 18:53:54,543+0530 INFO (jsonrpc/5) [vdsm.api] START > repoStats(domains=()) from=::1,38990, > task_id=3864f22e-98cc-426f-a4d5-2e2254f34aae (api:48) > 2022-02-03 18:53:54,543+0530 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats > return={'8ab8b572-39a3-4603-b0e7-2b6fe71c4d73': {'code': 0, 'lastCheck': > '6.3', 'delay': '0.000359743', 'valid': True, 'version': 5, 'acquired': > True, 'actual': True}, 'f3114b74-8b12-4493-9faf-0f514afa3a16': {'code': 0, > 'lastCheck': '2.1', 'delay': '0.000447464', 'valid': True, 'version': 5, > 'acquired': True, 'actual': True}, '26bfbe5d-e427-47a9-9111-cca8a628fea8': > {'code': 0, 'lastCheck': '2.0', 'delay': '0.00034851', 'valid': True, > 'version': 5, 'acquired': True, 'actual': True}, > '23540fa8-4de2-4715-97e1-4ce9fe4f66cc': {'code': 0, 'lastCheck': '2.1', > 'delay': '0.000352501', 'valid': True, 'version': 5, 'acquired': True, > 'actual': True}, '8fa0a28e-480f-444c-9cf2-bef536d3c52f': {'code': 0, > 'lastCheck': '2.0', 'delay': '0.000320087', 'valid': True, 'version': 5, > 'acquired': True, 'actual': True}, 'd1dba7f5-cb30-4571-8fe6-5fb5005bcdb2': > {'code': 0, 'lastCheck': '2.6', 'delay': '0.000452836', 'valid': True, > 'version': 0, 'acquired': True, 'actual': True}, > '88b94c9a-6855-4199-9d37-1f88d21f211c': {'code': 0, 'lastCheck': '2.0', > 'delay': '0.000299592', 'valid': True, 'version': 5, 'acquired': True, > 'actual': True}, '2e0ec671-5972-4276-aa49-aefc306f5da4': {'code': 0, > 'lastCheck': '2.0', 'delay': '0.000316513', 'valid': True, 'version': 5, > 'acquired': True, 'actual': True}} from=::1,38990, > task_id=3864f22e-98cc-426f-a4d5-2e2254f34aae (api:54) > 2022-02-03 18:53:54,543+0530 INFO (jsonrpc/5) [vdsm.api] START > multipath_health() from=::1,38990, > task_id=7ecaad0c-aa5d-45c0-96a5-dffc78572248 (api:48) > 2022-02-03 18:53:54,543+0530 INFO (jsonrpc/5) [vdsm.api] FINISH > multipath_health return={} from=::1,38990, > task_id=7ecaad0c-aa5d-45c0-96a5-dffc78572248 (api:54) > 2022-02-03 18:53:54,548+0530 ERROR (jsonrpc/5) > [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host > 3: received 0 of 512 expected bytes (client:137) > 2022-02-03 18:53:54,549+0530 INFO (jsonrpc/5) [api.host] FINISH getStats > return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} > from=::1,38990 (api:54) > 2022-02-03 18:53:56,038+0530 INFO (jsonrpc/4) [vdsm.api] START > repoStats(domains=['8ab8b572-39a3-4603-b0e7-2b6fe71c4d73']) from=::1,38990, > task_id=8855f314-9334-4b06-88e3-7a1a3f577049 (api:48) > 2022-02-03 18:53:56,038+0530 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats > return={'8ab8b572-39a3-4603-b0e7-2b6fe71c4d73': {'code': 0, 'lastCheck': > '1.3', 'delay': '0.000349788', 'valid': True, 'version': 5, 'acquired': > True, 'actual': True}} from=::1,38990, > task_id=8855f314-9334-4b06-88e3-7a1a3f577049 (api:54) > > > > On Thu, Feb 3, 2022 at 6:42 PM Martin Sivak <msi...@redhat.com> wrote: > >> >> >> On Thu, Feb 3, 2022 at 1:50 PM Ayansh Rocks <shashank123rast...@gmail.com> >> wrote: >> >>> Do I need to clean it from every host which has the hosted engine ? Sinc >>> the other host still can see it. >>> >> >> No, the metadata "whiteboard" is shared. Just one copy that you removed. >> Once node 3 start reporting again, it will simply put new content in there. >> >> Not sure about vdsm, maybe some component (vdsm, engine) is still trying >> to get status of node 3. I have been away for too long, so Didi will have >> to pitch in. >> >> Martin Sivak >> >> >> >>> >>> On Thu, Feb 3, 2022 at 6:02 PM Ayansh Rocks < >>> shashank123rast...@gmail.com> wrote: >>> >>>> Hi Yedidyah, >>>> >>>> Thanks, After cleaning it from one host i am getting below error >>>> messages on /var/log/messages >>>> >>>> Feb 3 18:01:02 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for >>>> host 3: received 0 of 512 expected bytes >>>> Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for >>>> host 3: received 0 of 512 expected bytes >>>> Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for >>>> host 3: received 0 of 512 expected bytes >>>> Feb 3 18:01:03 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for >>>> host 3: received 0 of 512 expected bytes >>>> Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for >>>> host 3: received 0 of 512 expected bytes >>>> Feb 3 18:01:13 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for >>>> host 3: received 0 of 512 expected bytes >>>> Feb 3 18:01:17 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for >>>> host 3: received 0 of 512 expected bytes >>>> Feb 3 18:01:18 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for >>>> host 3: received 0 of 512 expected bytes >>>> Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for >>>> host 3: received 0 of 512 expected bytes >>>> Feb 3 18:01:23 iondelsvr14 vdsm[11893]: ERROR Malformed metadata for >>>> host 3: received 0 of 512 expected bytes >>>> >>>> >>>> On Thu, Feb 3, 2022 at 1:04 PM Yedidyah Bar David <d...@redhat.com> >>>> wrote: >>>> >>>>> On Thu, Feb 3, 2022 at 9:11 AM Ayansh Rocks >>>>> <shashank123rast...@gmail.com> wrote: >>>>> > >>>>> > Hi Yedidyah, >>>>> > >>>>> > I am thinking to do below - >>>>> > >>>>> > # hosted-engine --set-maintenance --mode=global >>>>> > # hosted-engine --clean-metadata --host-id=<old_host_id> >>>>> --force-clean" >>>>> > >>>>> > I hope it is safe and tested ? >>>>> >>>>> It is tested, and it's "safe" as in "Does what it's designed to do". >>>>> >>>>> It's definitely not "safe" as in "riskless" - if you make a mistake, >>>>> you risk damage. I didn't use it myself recently, and do not remember >>>>> details anymore. It's definitely not something that you are supposed >>>>> to use/need casually. >>>>> >>>> >> It should be safe in global maintenance, the only risk is when you remove >> the report of a node that currently runs the engine. As other nodes might >> decide that the engine needs restarting. >> >> >> >>> If unsure, and if your environment is important to you (not just some >>>>> test/dev/learn thing), I warmly suggest to try this first on a test >>>>> env (can be done with nested-kvm VMs, no need for physical machines >>>>> for this). >>>>> >>>>> That said, others on this list (and elsewhere) did use it successfully, >>>>> and I recommend again to search for examples/reports/experiences. >>>>> >>>>> Good luck and best regards, >>>>> >>>>> > >>>>> > Thanks >>>>> > >>>>> > On Thu, Feb 3, 2022 at 12:17 PM Yedidyah Bar David <d...@redhat.com> >>>>> wrote: >>>>> >> >>>>> >> On Thu, Feb 3, 2022 at 5:53 AM Ayansh Rocks >>>>> >> <shashank123rast...@gmail.com> wrote: >>>>> >> > >>>>> >> > Hi All, >>>>> >> > >>>>> >> > How to clean stale hosted engine from the database. I want to >>>>> remove host id=3 >>>>> >> >>>>> >> Perhaps by using the options '--clean_metadata' and '--host-id' >>>>> >> of the command 'hosted-engine'. >>>>> >> You can check documentation, list archives, etc. for details. >>>>> >> >>>>> >> Good luck and best regards, >>>>> >> -- >>>>> >> Didi >>>>> >> >>>>> >>>>> >>>>> -- >>>>> Didi >>>>> >>>>> _______________________________________________ >>> Users mailing list -- users@ovirt.org >>> To unsubscribe send an email to users-le...@ovirt.org >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/IR3SMGNMRRGEKXUXTMCZBINAX6XZDRM5/ >>> >>
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/NWC2K7475A6HXAU2E6QET4FU4OR3WJX4/