Hi,
I solved this problem, added network “CloudLink-ISP2” to 2 problematic nodes and connected respective Ethernet interfaces to a separate switch bringing bringing up. > On 28 Apr 2022, at 07:45, Patrick Hibbs <hibbsncc1...@gmail.com> wrote: > > Your logs suggest that VDSM hasn't detected the removal of the network > yet. Did you try clicking on the "Sync Host Networks" button on the > non-operational hosts? > > If you've changed the network config, VDSM may have failed to set up > the host properly. What does the output of "ip addr show" on these > hosts? > > Manually removing the network from the hosts is only a temporary fix. > VDSM will attempt to recreate the network on restart if VDSM's config > isn't sync'd with the engine. > > -Patrick Hibbs > > On Wed, 2022-04-27 at 12:36 +0300, Andrei Verovski wrote: >> Hi, >> >> >> I run into nasty and unexpected problem (oVirt 4.4.7.6-1.el8 ) and 2 >> non-operational node hosts, which seem relate to ghost network >> glitch. >> Everything worked fine for a very long time until I did the >> following. >> >> 1) Made a backup of several VMs which have network “CloudLink-ISP2”. >> 2) Moved VM disks into another node hosts, which don’t have link >> “CloudLink-ISP2”, and forgot (preliminary to move) to re-configure >> network interface (remove link to “CloudLink-ISP2”) as I did before.. >> 3) Now I have 2 hosts in non-operational mode which is a really big >> problem - there are only 3 nodes total in a cluster. >> >> I tried to fix this by adding 3rd network “CloudLink-ISP2” to each >> non-operational hosts, and connected ethernet interfaces to a switch. >> Unfortunately, it didn’t helped either because for whatever reason >> link did not go up. >> >> How I can remove any reference to “CloudLink-ISP2” from node11 and >> node14? They don’t have any VMs needed that “CloudLink-ISP2”. I >> edited even inactive VMs and removed “CloudLink-ISP2”. >> May be there is an option to ignore missing or downed links whn >> activating node hosts? >> >> Thanks in advance for any help. >> >> >> —————————— >> >> Logs: >> >> 2022-04-27 11:16:59,769+03 ERROR >> [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE- >> ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) >> [5ab20328] Host 'node14' is set to Non-Operational, it is missing the >> following networks: 'CloudLink-ISP2' >> 2022-04-27 11:16:59,790+03 WARN >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector >> ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool- >> Thread-51) [5ab20328] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), >> Host node14 does not comply with the cluster ClusterRiga11 networks, >> the following networks are missing on host: 'CloudLink-ISP2' >> 2022-04-27 11:16:59,884+03 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector >> ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool- >> Thread-51) [55610e1e] EVENT_ID: VDS_DETECTED(13), Status of host >> node14 was set to NonOperational. >> 2022-04-27 11:16:59,907+03 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE- >> ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) >> [5d6ef791] Host 'node14'(aa871d44-94e2-4fdb-aeb3-ca0ae8dc568f) is >> already in NonOperational status for reason 'NETWORK_UNREACHABLE'. >> SetNonOperationalVds command is skipped. >> >> 2022-04-27 11:12:26,050+03 ERROR >> [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE- >> ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) >> [5400352e] Host 'node11' is set to Non-Operational, it is missing the >> following networks: 'CloudLink-ISP2' >> 2022-04-27 11:12:26,070+03 WARN >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector >> ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool- >> Thread-19) [5400352e] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), >> Host node11 does not comply with the cluster ClusterRiga11 networks, >> the following networks are missing on host: 'CloudLink-ISP2' >> 2022-04-27 11:12:26,192+03 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector >> ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool- >> Thread-19) [57d79e54] EVENT_ID: VDS_DETECTED(13), Status of host >> node11 was set to NonOperational. >> 2022-04-27 11:12:26,214+03 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE- >> ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) >> [151821e3] Host 'node11'(3c854f9c-2cdd-423e-bca0-37964ba76702) is >> already in NonOperational status for reason 'NETWORK_UNREACHABLE'. >> SetNonOperationalVds command is skipped. >> >> 2022-04-27 11:51:10,539+03 INFO >> [org.ovirt.engine.core.vdsbroker.VdsManager] (EE- >> ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62) >> [] Clearing domains data for host node14 >> 2022-04-27 11:51:10,539+03 INFO >> [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE- >> ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62) >> [] Host 'node14' moved to Non-Operational state because interface/s >> which are down are needed by required network/s in the current >> cluster: 'eno3 (CloudLink-ISP2)' >> 2022-04-27 11:51:10,569+03 WARN >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector >> ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool- >> Thread-62) [] EVENT_ID: VDS_SET_NONOPERATIONAL_IFACE_DOWN(603), Host >> node14 moved to Non-Operational state because interfaces which are >> down are needed by required networks in the current cluster: 'eno3 >> (CloudLink-ISP2)'. >> >> _______________________________________________ >> Users mailing list -- users@ovirt.org >> To unsubscribe send an email to users-le...@ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/EGENEHBHJYPLCM7HXTDITAADJQWU6PVX/ > > _______________________________________________ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z34DAZFEBAOX75JPIULGHE6IJB7GW6N3/ _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FEMZ7IBFV4UMHDVK7RFN35G7CORHF7E2/
