Public bug reported: We are creating a new Fedora 38 instance in Zed release, and we attach a new port to the instance when the VM status is active, but we got the following error in libvirt:
2024-01-24 17:41:13.999+0000: 451530: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:41:37.001+0000: 451529: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (52s, 0s, 0s) 2024-01-24 17:41:37.001+0000: 451529: error : virDomainObjBeginJobInternal:467 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainAttachDeviceFlags) 2024-01-24 17:41:39.645+0000: 451516: error : virNetSocketReadWire:1784 : End of file while reading data: Input/output error 2024-01-24 17:41:40.251+0000: 451533: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:41:44.639+0000: 2433335: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:42:06.802+0000: 451531: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (82s, 0s, 0s) 2024-01-24 17:42:06.802+0000: 451531: error : virDomainObjBeginJobInternal:467 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainAttachDeviceFlags) 2024-01-24 17:42:09.434+0000: 451516: error : virNetSocketReadWire:1784 : End of file while reading data: Input/output error 2024-01-24 17:42:10.859+0000: 451529: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:42:14.394+0000: 451535: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:42:36.585+0000: 2433335: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (111s, 0s, 0s) 2024-01-24 17:42:36.585+0000: 2433335: error : virDomainObjBeginJobInternal:467 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainAttachDeviceFlags) 2024-01-24 17:42:39.645+0000: 451516: error : virNetSocketReadWire:1784 : End of file while reading data: Input/output error 2024-01-24 17:42:40.446+0000: 451530: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:42:43.935+0000: 451531: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:43:06.803+0000: 451533: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (142s, 0s, 0s) It seems the instance is stuck in the attaching interface jobs, and we also checked the nova instance-action-list output, it reports the start time of the interface attaching event but no ends and may last for hours, when we hard reboot the instance, the interface attaching job gets failed immediately, seems all the blocked jobs are then released because of the hard reboot. I also attach the /var/log/kolla/libvirt.log.1 for more information. Thanks. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2051598 Title: virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats Status in OpenStack Compute (nova): New Bug description: We are creating a new Fedora 38 instance in Zed release, and we attach a new port to the instance when the VM status is active, but we got the following error in libvirt: 2024-01-24 17:41:13.999+0000: 451530: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:41:37.001+0000: 451529: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (52s, 0s, 0s) 2024-01-24 17:41:37.001+0000: 451529: error : virDomainObjBeginJobInternal:467 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainAttachDeviceFlags) 2024-01-24 17:41:39.645+0000: 451516: error : virNetSocketReadWire:1784 : End of file while reading data: Input/output error 2024-01-24 17:41:40.251+0000: 451533: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:41:44.639+0000: 2433335: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:42:06.802+0000: 451531: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (82s, 0s, 0s) 2024-01-24 17:42:06.802+0000: 451531: error : virDomainObjBeginJobInternal:467 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainAttachDeviceFlags) 2024-01-24 17:42:09.434+0000: 451516: error : virNetSocketReadWire:1784 : End of file while reading data: Input/output error 2024-01-24 17:42:10.859+0000: 451529: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:42:14.394+0000: 451535: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:42:36.585+0000: 2433335: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (111s, 0s, 0s) 2024-01-24 17:42:36.585+0000: 2433335: error : virDomainObjBeginJobInternal:467 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainAttachDeviceFlags) 2024-01-24 17:42:39.645+0000: 451516: error : virNetSocketReadWire:1784 : End of file while reading data: Input/output error 2024-01-24 17:42:40.446+0000: 451530: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:42:43.935+0000: 451531: error : virConnectListAllNetworks:117 : this function is not supported by the connection driver: virConnectListAllNetworks 2024-01-24 17:43:06.803+0000: 451533: warning : virDomainObjBeginJobInternal:433 : Cannot start job (query, none, none) in API remoteDispatchDomainMemoryStats for domain instance-00002dec; current job is (modify, none, none) owned by (451532 remoteDispatchDomainAttachDeviceFlags, 0 <null>, 0 <null> (flags=0x0)) for (142s, 0s, 0s) It seems the instance is stuck in the attaching interface jobs, and we also checked the nova instance-action-list output, it reports the start time of the interface attaching event but no ends and may last for hours, when we hard reboot the instance, the interface attaching job gets failed immediately, seems all the blocked jobs are then released because of the hard reboot. I also attach the /var/log/kolla/libvirt.log.1 for more information. Thanks. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2051598/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp