[Yahoo-eng-team] [Bug 1268955] Re: OVS agent updates the wrong port when using Xen + Neutron with HVM or PVHVM
** Changed in: neutron Status: Incomplete => Fix Committed ** Project changed: neutron => nova ** Summary changed: - OVS agent updates the wrong port when using Xen + Neutron with HVM or PVHVM + OVS agent updates the wrong port when using XenAPI + Neutron with HVM or PVHVM -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1268955 Title: OVS agent updates the wrong port when using XenAPI + Neutron with HVM or PVHVM Status in OpenStack Compute (nova): Fix Committed Bug description: Environment == - Xen Server 6.2 - OpenStack Havana installed with Packstack - Neutron OVS agent using VLAN From time to time, when an instance is started, it fails to get network connectivity. As a result the instance cannot get its IP address from DHCP and it remains unreachable. After further investigation, it appears that the OVS agent running on the compute node is updating the wrong OVS port because on startup, 2 ports exist for the same instance: vifX.0 and tapX.0. The agent updates whatever port is returned in first position (see logs below). Note that the tapX.0 port is only transient and disappears after a few seconds. Workaround == Manually update the OVS port on dom0: $ ovs-vsctl set Port vif17.0 tag=1 OVS Agent logs 2014-01-14 14:15:11.382 18268 DEBUG neutron.agent.linux.utils [-] Running command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id="98679ab6-b879-4b1b-a524-01696959d468"'] execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:43 2014-01-14 14:15:11.403 18268 DEBUG qpid.messaging.io.raw [-] SENT[3350c68]: '\x0f\x01\x00\x19\x00\x01\x00\x00\x00\x00\x00\x00\x04\n\x01\x00\x07\x00\x010\x00\x00\x00\x00\x01\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x81' writeable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:480 2014-01-14 14:15:11.649 18268 DEBUG neutron.agent.linux.utils [-] Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id="98679ab6-b879-4b1b-a524-01696959d468"'] Exit code: 0 Stdout: 'external_ids: {attached-mac="fa:16:3e:46:1e:91", iface-id="98679ab6-b879-4b1b-a524-01696959d468", iface-status=active, xs-network-uuid="b2bf90df-be17-a4ff-5c1e-3d69851f508a", xs-vif-uuid="2d2718d8-6064-e734-2737-cdcb4e06efc4", xs-vm-uuid="7f7f1918-3773-d97c-673a-37843797f70a"}\nname: "tap29.0"\nofport : 52\n\nexternal_ids: {attached-mac="fa:16:3e:46:1e:91", iface-id="98679ab6-b879-4b1b-a524-01696959d468", iface-status=inactive, xs-network-uuid="b2bf90df-be17-a4ff-5c1e-3d69851f508a", xs-vif-uuid="2d2718d8-6064-e734-2737-cdcb4e06efc4", xs-vm-uuid="7f7f1918-3773-d97c-673a-37843797f70a"}\nname: "vif29.0"\nofport : 51\n\n' Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:60 2014-01-14 14:15:11.650 18268 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Port 98679ab6-b879-4b1b-a524-01696959d468 updated. Details: {u'admin_state_up': True, u'network_id': u'ad37f107-074b-4c58-8f36-4705533afb8d', u'segmentation_id': 100, u'physical_network': u'default', u'device': u'98679ab6-b879-4b1b-a524-01696959d468', u'port_id': u'98679ab6-b879-4b1b-a524-01696959d468', u'network_type': u'vlan'} 2014-01-14 14:15:11.650 18268 DEBUG neutron.agent.linux.utils [-] Running command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'set', 'Port', 'tap29.0', 'tag=1'] execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:43 2014-01-14 14:15:11.913 18268 DEBUG neutron.agent.linux.utils [-] Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'set', 'Port', 'tap29.0', 'tag=1'] Exit code: 0 Stdout: '\n' Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:60 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1268955/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1268955] Re: OVS agent updates the wrong port when using Xen + Neutron with HVM or PVHVM
I think this will not be fixed in Havana. I wonder if Icehouse works fine for the described case. ** Changed in: neutron Status: New => Opinion -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1268955 Title: OVS agent updates the wrong port when using Xen + Neutron with HVM or PVHVM Status in OpenStack Neutron (virtual network service): Opinion Bug description: Environment == - Xen Server 6.2 - OpenStack Havana installed with Packstack - Neutron OVS agent using VLAN From time to time, when an instance is started, it fails to get network connectivity. As a result the instance cannot get its IP address from DHCP and it remains unreachable. After further investigation, it appears that the OVS agent running on the compute node is updating the wrong OVS port because on startup, 2 ports exist for the same instance: vifX.0 and tapX.0. The agent updates whatever port is returned in first position (see logs below). Note that the tapX.0 port is only transient and disappears after a few seconds. Workaround == Manually update the OVS port on dom0: $ ovs-vsctl set Port vif17.0 tag=1 OVS Agent logs 2014-01-14 14:15:11.382 18268 DEBUG neutron.agent.linux.utils [-] Running command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id="98679ab6-b879-4b1b-a524-01696959d468"'] execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:43 2014-01-14 14:15:11.403 18268 DEBUG qpid.messaging.io.raw [-] SENT[3350c68]: '\x0f\x01\x00\x19\x00\x01\x00\x00\x00\x00\x00\x00\x04\n\x01\x00\x07\x00\x010\x00\x00\x00\x00\x01\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x81' writeable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:480 2014-01-14 14:15:11.649 18268 DEBUG neutron.agent.linux.utils [-] Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id="98679ab6-b879-4b1b-a524-01696959d468"'] Exit code: 0 Stdout: 'external_ids: {attached-mac="fa:16:3e:46:1e:91", iface-id="98679ab6-b879-4b1b-a524-01696959d468", iface-status=active, xs-network-uuid="b2bf90df-be17-a4ff-5c1e-3d69851f508a", xs-vif-uuid="2d2718d8-6064-e734-2737-cdcb4e06efc4", xs-vm-uuid="7f7f1918-3773-d97c-673a-37843797f70a"}\nname: "tap29.0"\nofport : 52\n\nexternal_ids: {attached-mac="fa:16:3e:46:1e:91", iface-id="98679ab6-b879-4b1b-a524-01696959d468", iface-status=inactive, xs-network-uuid="b2bf90df-be17-a4ff-5c1e-3d69851f508a", xs-vif-uuid="2d2718d8-6064-e734-2737-cdcb4e06efc4", xs-vm-uuid="7f7f1918-3773-d97c-673a-37843797f70a"}\nname: "vif29.0"\nofport : 51\n\n' Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:60 2014-01-14 14:15:11.650 18268 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Port 98679ab6-b879-4b1b-a524-01696959d468 updated. Details: {u'admin_state_up': True, u'network_id': u'ad37f107-074b-4c58-8f36-4705533afb8d', u'segmentation_id': 100, u'physical_network': u'default', u'device': u'98679ab6-b879-4b1b-a524-01696959d468', u'port_id': u'98679ab6-b879-4b1b-a524-01696959d468', u'network_type': u'vlan'} 2014-01-14 14:15:11.650 18268 DEBUG neutron.agent.linux.utils [-] Running command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'set', 'Port', 'tap29.0', 'tag=1'] execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:43 2014-01-14 14:15:11.913 18268 DEBUG neutron.agent.linux.utils [-] Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'set', 'Port', 'tap29.0', 'tag=1'] Exit code: 0 Stdout: '\n' Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:60 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1268955/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1268955] Re: OVS agent updates the wrong port when using Xen + Neutron with HVM or PVHVM
** Changed in: neutron Status: Opinion => Incomplete -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1268955 Title: OVS agent updates the wrong port when using Xen + Neutron with HVM or PVHVM Status in neutron: Incomplete Bug description: Environment == - Xen Server 6.2 - OpenStack Havana installed with Packstack - Neutron OVS agent using VLAN From time to time, when an instance is started, it fails to get network connectivity. As a result the instance cannot get its IP address from DHCP and it remains unreachable. After further investigation, it appears that the OVS agent running on the compute node is updating the wrong OVS port because on startup, 2 ports exist for the same instance: vifX.0 and tapX.0. The agent updates whatever port is returned in first position (see logs below). Note that the tapX.0 port is only transient and disappears after a few seconds. Workaround == Manually update the OVS port on dom0: $ ovs-vsctl set Port vif17.0 tag=1 OVS Agent logs 2014-01-14 14:15:11.382 18268 DEBUG neutron.agent.linux.utils [-] Running command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id="98679ab6-b879-4b1b-a524-01696959d468"'] execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:43 2014-01-14 14:15:11.403 18268 DEBUG qpid.messaging.io.raw [-] SENT[3350c68]: '\x0f\x01\x00\x19\x00\x01\x00\x00\x00\x00\x00\x00\x04\n\x01\x00\x07\x00\x010\x00\x00\x00\x00\x01\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x81' writeable /usr/lib/python2.6/site-packages/qpid/messaging/driver.py:480 2014-01-14 14:15:11.649 18268 DEBUG neutron.agent.linux.utils [-] Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--', '--columns=external_ids,name,ofport', 'find', 'Interface', 'external_ids:iface-id="98679ab6-b879-4b1b-a524-01696959d468"'] Exit code: 0 Stdout: 'external_ids: {attached-mac="fa:16:3e:46:1e:91", iface-id="98679ab6-b879-4b1b-a524-01696959d468", iface-status=active, xs-network-uuid="b2bf90df-be17-a4ff-5c1e-3d69851f508a", xs-vif-uuid="2d2718d8-6064-e734-2737-cdcb4e06efc4", xs-vm-uuid="7f7f1918-3773-d97c-673a-37843797f70a"}\nname: "tap29.0"\nofport : 52\n\nexternal_ids: {attached-mac="fa:16:3e:46:1e:91", iface-id="98679ab6-b879-4b1b-a524-01696959d468", iface-status=inactive, xs-network-uuid="b2bf90df-be17-a4ff-5c1e-3d69851f508a", xs-vif-uuid="2d2718d8-6064-e734-2737-cdcb4e06efc4", xs-vm-uuid="7f7f1918-3773-d97c-673a-37843797f70a"}\nname: "vif29.0"\nofport : 51\n\n' Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:60 2014-01-14 14:15:11.650 18268 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Port 98679ab6-b879-4b1b-a524-01696959d468 updated. Details: {u'admin_state_up': True, u'network_id': u'ad37f107-074b-4c58-8f36-4705533afb8d', u'segmentation_id': 100, u'physical_network': u'default', u'device': u'98679ab6-b879-4b1b-a524-01696959d468', u'port_id': u'98679ab6-b879-4b1b-a524-01696959d468', u'network_type': u'vlan'} 2014-01-14 14:15:11.650 18268 DEBUG neutron.agent.linux.utils [-] Running command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'set', 'Port', 'tap29.0', 'tag=1'] execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:43 2014-01-14 14:15:11.913 18268 DEBUG neutron.agent.linux.utils [-] Command: ['/usr/bin/neutron-rootwrap-xen-dom0', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'set', 'Port', 'tap29.0', 'tag=1'] Exit code: 0 Stdout: '\n' Stderr: '' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:60 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1268955/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp