Public bug reported: For eval purpose I used to run several openvswitch agents on the same machine. Kolla-ansible proposes such a thing (https://docs.openstack.org/developer/kolla-ansible/nova-fake-driver.html) but this feature is nowaday broken. In this context each openvswitch agent are running on its own dedicated docker container and are configured to use a different ovs bridge (br-int-1, br-int-2, …).
Some preliminary tests shows that using the default `of_interface=native` led to some troubles. Environment : ------------- $) docker exec -ti neutron_openvswitch_agent_1 pip freeze | grep neutron neutron==11.0.0.0b2.dev408 neutron-fwaas==11.0.0.0b2.dev25 neutron-lib==1.7.0 python-neutronclient==6.2.0 $) docker exec -ti neutron_openvswitch_agent_1 ovs-vsctl --version ovs-vsctl (Open vSwitch) 2.6.1 DB Schema 7.14.0 Observation with of_interface=native ------------------------------------- On the nodes where several openvswitch agents run only one runs correctly. The others fail (3 agents in this example) showing the following log : 2017-06-05 20:24:16.056 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:17.117 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:21.061 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:22.123 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:26.064 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:27.125 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:31.065 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:32.128 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:36.072 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:37.131 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:41.077 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:42.136 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:42.163 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout $) openstack network agent list +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+ | 143b3b54-0f5e-4a2a-a441-fed5e5cc496f | L3 agent | enos-126-neutron-l3-agent | nova | True | UP | neutron-l3-agent | | a6d5e1d9-3102-4395-bd52-977372b0cfeb | Open vSwitch agent | enos-125-neutron-openvswitch-agent | None | True | UP | neutron-openvswitch-agent | | cd8d49c6-aaa6-4de7-a07f-68ef3b117e36 | Open vSwitch agent | enos-126-neutron-openvswitch-agent | None | True | UP | neutron-openvswitch-agent | | d951874e-d449-459e-bf47-bd746a45b16a | Metadata agent | enos-126-neutron-metadata-agent | None | True | UP | neutron-metadata-agent | | fe2cf638-ff26-4662-969e-5f49e6b57d4d | DHCP agent | enos-126-neutron-dhcp-agent | nova | True | UP | neutron-dhcp-agent | +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+ Observation with of_interface=ovs-ofctl ---------------------------------------- When switching to `of_interface=ovs-ofctl` everything works as expected : all the agents are running correctly : $) openstack network agent list +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+ | 21a45a3a-4a2c-4c3d-9246-26adfd6cc890 | Open vSwitch agent | enos-125-3 | None | True | UP | neutron-openvswitch-agent | | 563cb6d3-7527-4d99-a80a-bf278fbde57e | Open vSwitch agent | enos-125-1 | None | True | UP | neutron-openvswitch-agent | | 596ec4f1-0fe9-42a0-be8d-602deb8a4a4e | Open vSwitch agent | enos-126-neutron-openvswitch-agent | None | True | UP | neutron-openvswitch-agent | | 5bbb6722-4df9-465a-a433-fb8153f595b5 | Open vSwitch agent | enos-125-2 | None | True | UP | neutron-openvswitch-agent | | 797fdec3-6967-46e5-9d81-6280c888066d | L3 agent | enos-126-neutron-l3-agent | nova | True | UP | neutron-l3-agent | | b56549f1-c755-45dd-b2c9-072fdd418e78 | Open vSwitch agent | enos-125-neutron-openvswitch-agent | None | True | UP | neutron-openvswitch-agent | | cef1c889-0851-44f4-b1d3-e6339320473a | DHCP agent | enos-126-neutron-dhcp-agent | nova | True | UP | neutron-dhcp-agent | | df718b50-305c-4fcc-bce3-3dd252942dd3 | Metadata agent | enos-126-neutron-metadata-agent | None | True | UP | neutron-metadata-agent | +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+ I think I can enforce `of_interface=ovs-ofctl` in the configuration file, but it doesn't seem a viable solution dur to its future deprecation. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1695995 Title: of_interface=native makes openvswitch agents fail when run on the same node Status in neutron: New Bug description: For eval purpose I used to run several openvswitch agents on the same machine. Kolla-ansible proposes such a thing (https://docs.openstack.org/developer/kolla-ansible/nova-fake-driver.html) but this feature is nowaday broken. In this context each openvswitch agent are running on its own dedicated docker container and are configured to use a different ovs bridge (br-int-1, br-int-2, …). Some preliminary tests shows that using the default `of_interface=native` led to some troubles. Environment : ------------- $) docker exec -ti neutron_openvswitch_agent_1 pip freeze | grep neutron neutron==11.0.0.0b2.dev408 neutron-fwaas==11.0.0.0b2.dev25 neutron-lib==1.7.0 python-neutronclient==6.2.0 $) docker exec -ti neutron_openvswitch_agent_1 ovs-vsctl --version ovs-vsctl (Open vSwitch) 2.6.1 DB Schema 7.14.0 Observation with of_interface=native ------------------------------------- On the nodes where several openvswitch agents run only one runs correctly. The others fail (3 agents in this example) showing the following log : 2017-06-05 20:24:16.056 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:17.117 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:21.061 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:22.123 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:26.064 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:27.125 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:31.065 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:32.128 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:36.072 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:37.131 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:41.077 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:42.136 7 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 8 __log_wakeup /var/lib/kolla/venv/lib/python2.7/site-packages/ovs/poller.py:246 2017-06-05 20:24:42.163 7 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout $) openstack network agent list +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+ | 143b3b54-0f5e-4a2a-a441-fed5e5cc496f | L3 agent | enos-126-neutron-l3-agent | nova | True | UP | neutron-l3-agent | | a6d5e1d9-3102-4395-bd52-977372b0cfeb | Open vSwitch agent | enos-125-neutron-openvswitch-agent | None | True | UP | neutron-openvswitch-agent | | cd8d49c6-aaa6-4de7-a07f-68ef3b117e36 | Open vSwitch agent | enos-126-neutron-openvswitch-agent | None | True | UP | neutron-openvswitch-agent | | d951874e-d449-459e-bf47-bd746a45b16a | Metadata agent | enos-126-neutron-metadata-agent | None | True | UP | neutron-metadata-agent | | fe2cf638-ff26-4662-969e-5f49e6b57d4d | DHCP agent | enos-126-neutron-dhcp-agent | nova | True | UP | neutron-dhcp-agent | +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+ Observation with of_interface=ovs-ofctl ---------------------------------------- When switching to `of_interface=ovs-ofctl` everything works as expected : all the agents are running correctly : $) openstack network agent list +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+ | 21a45a3a-4a2c-4c3d-9246-26adfd6cc890 | Open vSwitch agent | enos-125-3 | None | True | UP | neutron-openvswitch-agent | | 563cb6d3-7527-4d99-a80a-bf278fbde57e | Open vSwitch agent | enos-125-1 | None | True | UP | neutron-openvswitch-agent | | 596ec4f1-0fe9-42a0-be8d-602deb8a4a4e | Open vSwitch agent | enos-126-neutron-openvswitch-agent | None | True | UP | neutron-openvswitch-agent | | 5bbb6722-4df9-465a-a433-fb8153f595b5 | Open vSwitch agent | enos-125-2 | None | True | UP | neutron-openvswitch-agent | | 797fdec3-6967-46e5-9d81-6280c888066d | L3 agent | enos-126-neutron-l3-agent | nova | True | UP | neutron-l3-agent | | b56549f1-c755-45dd-b2c9-072fdd418e78 | Open vSwitch agent | enos-125-neutron-openvswitch-agent | None | True | UP | neutron-openvswitch-agent | | cef1c889-0851-44f4-b1d3-e6339320473a | DHCP agent | enos-126-neutron-dhcp-agent | nova | True | UP | neutron-dhcp-agent | | df718b50-305c-4fcc-bce3-3dd252942dd3 | Metadata agent | enos-126-neutron-metadata-agent | None | True | UP | neutron-metadata-agent | +--------------------------------------+--------------------+------------------------------------+-------------------+-------+-------+---------------------------+ I think I can enforce `of_interface=ovs-ofctl` in the configuration file, but it doesn't seem a viable solution dur to its future deprecation. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1695995/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp