Public bug reported: This is happening on Nova 3:25.2.1 (Yoga)
We have some nodes running 300+ VMs, when restarting nova-compute we saw on journal following errors Jul 21 08:50:44 ra2-n5 dbus-daemon[37971]: [system] The maximum number of pending replies for ":1.131331" (uid=0 pid=2649667 comm="/usr/sbin/libvirtd --timeout 120 " label="libvirtd (enforce)") has been reached (max_replies_per_connection=128) Jul 21 08:50:44 ra2-n5 libvirtd[2649667]: error from service: GDBus.Error:org.freedesktop.DBus.Error.LimitsExceeded: The maximum number of pending replies per connection has been reached At the same time nova-compute which is configured with resume_guests_state_on_host_boot = True started rebooting half of VMs running on the node logging the following 2025-07-21 09:07:33.912 2665574 DEBUG nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Checking state _get_power_state /usr/lib/python3/dist-packages/nova/compute/manager.py:1674 2025-07-21 09:07:33.914 2665574 DEBUG nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Current state is 4, state in DB is 1. _init_instance /usr/lib/python3/dist-packages/nova/compute/manager.py:1302 2025-07-21 09:07:33.914 2665574 INFO nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Rebooting instance after nova-compute restart. 2025-07-21 09:07:33.946 2665574 INFO nova.virt.libvirt.driver [-] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Instance destroyed successfully. A possible workaround is to tune that dbus parameter controlling the max_replies_per_connection with # /etc/dbus-1/system-local.conf <!DOCTYPE busconfig PUBLIC "-//freedesktop//DTD D-Bus Bus Configuration 1.0//EN" "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd";> <busconfig> <limit name="max_replies_per_connection">1000</limit> </busconfig> # restart dbus sudo systemctl restart dbus ** Affects: nova Importance: Undecided Status: New ** Summary changed: - Nova restart Virtual machines on service restart if hitting max_replies_per_connection limit with livirtd + Nova restart Virtual machines on service restart if hitting max_replies_per_connection limit with libvirtd -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2118397 Title: Nova restart Virtual machines on service restart if hitting max_replies_per_connection limit with libvirtd Status in OpenStack Compute (nova): New Bug description: This is happening on Nova 3:25.2.1 (Yoga) We have some nodes running 300+ VMs, when restarting nova-compute we saw on journal following errors Jul 21 08:50:44 ra2-n5 dbus-daemon[37971]: [system] The maximum number of pending replies for ":1.131331" (uid=0 pid=2649667 comm="/usr/sbin/libvirtd --timeout 120 " label="libvirtd (enforce)") has been reached (max_replies_per_connection=128) Jul 21 08:50:44 ra2-n5 libvirtd[2649667]: error from service: GDBus.Error:org.freedesktop.DBus.Error.LimitsExceeded: The maximum number of pending replies per connection has been reached At the same time nova-compute which is configured with resume_guests_state_on_host_boot = True started rebooting half of VMs running on the node logging the following 2025-07-21 09:07:33.912 2665574 DEBUG nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Checking state _get_power_state /usr/lib/python3/dist-packages/nova/compute/manager.py:1674 2025-07-21 09:07:33.914 2665574 DEBUG nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Current state is 4, state in DB is 1. _init_instance /usr/lib/python3/dist-packages/nova/compute/manager.py:1302 2025-07-21 09:07:33.914 2665574 INFO nova.compute.manager [req-489ef5b2-fe71-49e0-ad11-02fc9a4f491f - - - - -] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Rebooting instance after nova-compute restart. 2025-07-21 09:07:33.946 2665574 INFO nova.virt.libvirt.driver [-] [instance: 923aa838-a49f-41ad-90ce-abfa79d2e2b5] Instance destroyed successfully. A possible workaround is to tune that dbus parameter controlling the max_replies_per_connection with # /etc/dbus-1/system-local.conf <!DOCTYPE busconfig PUBLIC "-//freedesktop//DTD D-Bus Bus Configuration 1.0//EN" "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd";> <busconfig> <limit name="max_replies_per_connection">1000</limit> </busconfig> # restart dbus sudo systemctl restart dbus To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2118397/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
