[Yahoo-eng-team] [Bug 2033683] Re: openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n']
We are facing this issue in Puppet OpenStack CI which uses RDO stable/yoga and c8s, so this looks like a legit bug in iptables. I don't think this is also related to TripleO so I'll close this as invalid. ** Changed in: tripleo Status: Confirmed => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2033683 Title: openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n'] Status in neutron: Invalid Status in tripleo: Invalid Bug description: Description === Wallaby deployment via undercloud/overcloud started to fail recently on overcloud node provision Neutron constantly reports inability to update iptables that in turn makes baremetal to fail to boot from PXE From the review it seems that /usr/bin/update-alternatives set to legacy fails since neutron user doesn't have sudo to run it In the info I can see that neutron user has the following subset of commands it's able to run: ... (root) NOPASSWD: /usr/bin/update-alternatives --set iptables /usr/sbin/iptables-legacy (root) NOPASSWD: /usr/bin/update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy (root) NOPASSWD: /usr/bin/update-alternatives --auto iptables (root) NOPASSWD: /usr/bin/update-alternatives --auto ip6tables But the issue is the fact that command isn't found as it was moved to /usr/sbin/update-alternatives Steps to reproduce == 1. Deploy undercloud 2. Deploy networks and VIP 3. Add and introspect a node 4. Execute overcloud node provision ... that will timeout Expected result === Successful overcloud node baremetal provisioning Logs & Configs == 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-18d52177-9c93-401c-b97d-0334e488a257 - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['iptables-restore', '-n']; Stdin: # Generated by iptables_manager 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent COMMIT 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent # Completed by iptables_manager 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ; Stdout: ; Stderr: iptables-restore: line 23 failed Environment === Centos 9 Stream and undercloud deployment tool To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2033683/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2033683] Re: openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n']
Hi Alex, << Can someone take a look why the above patch https://review.opendev.org/c/openstack/kolla/+/761182 mentioned here has been excluded from the neutron image? It would have been just missed, since train release Tripleo builds container images natively and not use kolla, You can propose a patch in tripleo-common to fix it. As said i was more interested to know why the issue seen now as /usr/sbin/update-alternatives used to be the path from long back. But considering you are using CentOS8-stream containers on CentOS9-stream host i think you are hitting a recent iptables issue in CentOS8-stream[1], you can check version in your running container, if it matches iptables-1.8.5-8 you can downgrade it to resolve the issue temporary, as the fix for it is not yet merged. If there is no real reason to use CentOS8 images can move to use CentOS 9-Stream based images[2] [1] https://bugzilla.redhat.com/show_bug.cgi?id=2236501 [2] https://quay.io/repository/tripleowallabycentos9/openstack-neutron-server?tab=tags Again marking it as invalid for neutron, feel free to reopen but share what's expected fix is required in neutron project. ** Bug watch added: Red Hat Bugzilla #2236501 https://bugzilla.redhat.com/show_bug.cgi?id=2236501 ** Changed in: neutron Status: New => Invalid ** Changed in: tripleo Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2033683 Title: openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n'] Status in neutron: Invalid Status in tripleo: Confirmed Bug description: Description === Wallaby deployment via undercloud/overcloud started to fail recently on overcloud node provision Neutron constantly reports inability to update iptables that in turn makes baremetal to fail to boot from PXE From the review it seems that /usr/bin/update-alternatives set to legacy fails since neutron user doesn't have sudo to run it In the info I can see that neutron user has the following subset of commands it's able to run: ... (root) NOPASSWD: /usr/bin/update-alternatives --set iptables /usr/sbin/iptables-legacy (root) NOPASSWD: /usr/bin/update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy (root) NOPASSWD: /usr/bin/update-alternatives --auto iptables (root) NOPASSWD: /usr/bin/update-alternatives --auto ip6tables But the issue is the fact that command isn't found as it was moved to /usr/sbin/update-alternatives Steps to reproduce == 1. Deploy undercloud 2. Deploy networks and VIP 3. Add and introspect a node 4. Execute overcloud node provision ... that will timeout Expected result === Successful overcloud node baremetal provisioning Logs & Configs == 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-18d52177-9c93-401c-b97d-0334e488a257 - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['iptables-restore', '-n']; Stdin: # Generated by iptables_manager 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent COMMIT 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent # Completed by iptables_manager 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ; Stdout: ; Stderr: iptables-restore: line 23 failed Environment === Centos 9 Stream and undercloud deployment tool To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2033683/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2033683] Re: openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n']
Hello, It's hard to say what project should be patched - but sudo rules on the tripleowallaby / openstack-neutron-server has to be patched Image itself is using neutron user and it doesn't seem to be able to run the necessary commands to apply correct iptable rules This in turn prevents deployment of the new nodes as provisioning ain't working and renders the whole cluster failed. Can someone take a look why the above patch https://review.opendev.org/c/openstack/kolla/+/761182 mentioned here has been excluded from the neutron image? ** Changed in: neutron Status: Invalid => New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2033683 Title: openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n'] Status in neutron: New Status in tripleo: New Bug description: Description === Wallaby deployment via undercloud/overcloud started to fail recently on overcloud node provision Neutron constantly reports inability to update iptables that in turn makes baremetal to fail to boot from PXE From the review it seems that /usr/bin/update-alternatives set to legacy fails since neutron user doesn't have sudo to run it In the info I can see that neutron user has the following subset of commands it's able to run: ... (root) NOPASSWD: /usr/bin/update-alternatives --set iptables /usr/sbin/iptables-legacy (root) NOPASSWD: /usr/bin/update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy (root) NOPASSWD: /usr/bin/update-alternatives --auto iptables (root) NOPASSWD: /usr/bin/update-alternatives --auto ip6tables But the issue is the fact that command isn't found as it was moved to /usr/sbin/update-alternatives Steps to reproduce == 1. Deploy undercloud 2. Deploy networks and VIP 3. Add and introspect a node 4. Execute overcloud node provision ... that will timeout Expected result === Successful overcloud node baremetal provisioning Logs & Configs == 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-18d52177-9c93-401c-b97d-0334e488a257 - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['iptables-restore', '-n']; Stdin: # Generated by iptables_manager 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent COMMIT 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent # Completed by iptables_manager 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ; Stdout: ; Stderr: iptables-restore: line 23 failed Environment === Centos 9 Stream and undercloud deployment tool To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2033683/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2033683] Re: openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n']
** Changed in: neutron Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2033683 Title: openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n'] Status in neutron: Invalid Status in tripleo: New Bug description: Description === Wallaby deployment via undercloud/overcloud started to fail recently on overcloud node provision Neutron constantly reports inability to update iptables that in turn makes baremetal to fail to boot from PXE From the review it seems that /usr/bin/update-alternatives set to legacy fails since neutron user doesn't have sudo to run it In the info I can see that neutron user has the following subset of commands it's able to run: ... (root) NOPASSWD: /usr/bin/update-alternatives --set iptables /usr/sbin/iptables-legacy (root) NOPASSWD: /usr/bin/update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy (root) NOPASSWD: /usr/bin/update-alternatives --auto iptables (root) NOPASSWD: /usr/bin/update-alternatives --auto ip6tables But the issue is the fact that command isn't found as it was moved to /usr/sbin/update-alternatives Steps to reproduce == 1. Deploy undercloud 2. Deploy networks and VIP 3. Add and introspect a node 4. Execute overcloud node provision ... that will timeout Expected result === Successful overcloud node baremetal provisioning Logs & Configs == 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-18d52177-9c93-401c-b97d-0334e488a257 - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['iptables-restore', '-n']; Stdin: # Generated by iptables_manager 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent COMMIT 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent # Completed by iptables_manager 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ; Stdout: ; Stderr: iptables-restore: line 23 failed Environment === Centos 9 Stream and undercloud deployment tool To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2033683/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 2033683] Re: openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n']
** Also affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2033683 Title: openvswitch.agent.ovs_neutron_agent fails to Cmd: ['iptables-restore', '-n'] Status in neutron: New Status in tripleo: New Bug description: Description === Wallaby deployment via undercloud/overcloud started to fail recently on overcloud node provision Neutron constantly reports inability to update iptables that in turn makes baremetal to fail to boot from PXE From the review it seems that /usr/bin/update-alternatives set to legacy fails since neutron user doesn't have sudo to run it In the info I can see that neutron user has the following subset of commands it's able to run: ... (root) NOPASSWD: /usr/bin/update-alternatives --set iptables /usr/sbin/iptables-legacy (root) NOPASSWD: /usr/bin/update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy (root) NOPASSWD: /usr/bin/update-alternatives --auto iptables (root) NOPASSWD: /usr/bin/update-alternatives --auto ip6tables But the issue is the fact that command isn't found as it was moved to /usr/sbin/update-alternatives Steps to reproduce == 1. Deploy undercloud 2. Deploy networks and VIP 3. Add and introspect a node 4. Execute overcloud node provision ... that will timeout Expected result === Successful overcloud node baremetal provisioning Logs & Configs == 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-18d52177-9c93-401c-b97d-0334e488a257 - - - - -] Error while processing VIF ports: neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['iptables-restore', '-n']; Stdin: # Generated by iptables_manager 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent COMMIT 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent # Completed by iptables_manager 2023-08-31 18:21:28.613 4413 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ; Stdout: ; Stderr: iptables-restore: line 23 failed Environment === Centos 9 Stream and undercloud deployment tool To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2033683/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
