Hi Manojawa, Comments below.
BR, Billy From: Manojawa Paritala [mailto:[email protected]] Sent: Thursday, October 25, 2018 10:33 AM To: O Mahony, Billy <[email protected]> Cc: [email protected]; [email protected]; Subba Rao Kodavalla <[email protected]>; Song, Kee SangX <[email protected]>; Srinivasa Goda <[email protected]>; Kris Rajana <[email protected]> Subject: Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens Hi Billy, The issue of br-flat1 getting deleted is fixed now with the key-value pair of "datapath_types = netdev,system" in OVS neutron-openvswitch-agent.ini in the compute node. [[BO'M]] Glad to hear you solved it. Earlier, we used the key-value pair as "datapath_type = netdev". Please note the new key name we used is "datapath_types" and in the the value we mentioned "netdev,system". So, now our updated OVS section in neutron-openvswitch-agent.ini is as below. [OVS] bridge_mappings = flat:br-flat1,vxlan:br-vxlan,vlan:br-vlan,vlan1:br0,vlan2:br1 local_ip = 192.168.5.14 datapath_types = netdev,system One issue that we see now is, when we add "vlan1:br0,vlan2:br1" to bridge_mappings and restart the neutron-openvswitch-agent.ini and openvsiwtch-agent services, in the "ovs-vsctl show" output, the ports in the bridges br0 & br1 have the below error. This issue is getting resolved, if I change the datapath_type of those bridges to netdev. [[BO'M]] As both system and netdev (dpdk) type datapaths are in use here I would guess that br1 and br0 here are both being set as system type datapaths (you can check in ovsdb). But system type datapath (ie openvswitch.ko) is a kernel based datapath and does not support dpdk (userspace) type interfaces – ie dpdk[vhostuser[client]] Getting OHA/neutron-agent to use specific datapath types for specific bridges is beyond the scope of this ML. But let me know if you have any OvS issues. Bridge "br1" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "phy-br1" Interface "phy-br1" type: patch options: {peer="int-br1"} Port "vhost-user-5" Interface "vhost-user-5" type: dpdkvhostuser error: "could not add network device vhost-user-5 to ofproto (Invalid argument)" Port "port1" Interface "port1" type: dpdk options: {dpdk-devargs="0000:b1:00.1"} error: "Error attaching device '0000:b1:00.1' to DPDK" Port "vhost-user-6" Interface "vhost-user-6" type: dpdkvhostuser error: "could not add network device vhost-user-6 to ofproto (Invalid argument)" Port "br1" Interface "br1" type: internal Bridge "br0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "vhost-user-1" Interface "vhost-user-1" type: dpdkvhostuser error: "could not add network device vhost-user-1 to ofproto (Invalid argument)" Port "br0" Interface "br0" type: internal Port "port0" Interface "port0" type: dpdk options: {dpdk-devargs="0000:af:00.1"} error: "could not add network device port0 to ofproto (Invalid argument)" Port "phy-br0" Interface "phy-br0" type: patch options: {peer="int-br0"} Port "vhost-user-3" Interface "vhost-user-3" type: dpdkvhostuser error: "could not add network device vhost-user-3 to ofproto (Invalid argument)" Port "vhost-user-2" Interface "vhost-user-2" type: dpdkvhostuser error: "could not add network device vhost-user-2 to ofproto (Invalid argument)" Port "vhost-user-4" Interface "vhost-user-4" type: dpdkvhostuser error: "could not add network device vhost-user-4 to ofproto (Invalid argument)" Thanks & Regards, PVMJ On Wed, Oct 24, 2018 at 7:47 PM Manojawa Paritala <[email protected]<mailto:[email protected]>> wrote: Hi Billy, The reason for br-vlan getting deleted in my previous attempt was due to a small misconfiguration in the openstack-agent.ini. I corrected it and re-done the test. Attaching the new logs with this email. Please ignore the earlier logs. Apologies for the confusion. Thanks & Regards, PVMJ On Wed, Oct 24, 2018 at 7:26 PM Manojawa Paritala <[email protected]<mailto:[email protected]>> wrote: Hi Billy, Firstly, thank you for taking a look at the logs I sent yesterday. Please see in-line. Please let me know if I miss anything. Thanks & Regards, PVMJ On Wed, Oct 24, 2018 at 3:36 PM O Mahony, Billy <[email protected]<mailto:[email protected]>> wrote: From: Manojawa Paritala [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, October 23, 2018 5:37 PM To: O Mahony, Billy <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; Subba Rao Kodavalla <[email protected]<mailto:[email protected]>>; Song, Kee SangX <[email protected]<mailto:[email protected]>>; Srinivasa Goda <[email protected]<mailto:[email protected]>>; Kris Rajana <[email protected]<mailto:[email protected]>> Subject: Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens Hi Billy, There are no br-flat1 entries in ovsdb. As suggested I increased the log level to debug and then tried the same scenario again. Though the result was same (br-flat1 getting deleted), I observed the below 2 issues (i assume). [[BO'M]] You can just increase the log level for the bridge module (all the extra revalidator debug is making it hard to read the logs.): ovs-appctl vlog/set bridge:file:dbg Manoj :- Added the log level of debug to only bridge module and collected the logs. Attached zip file has the logs. [[BO'M]] Also is there logging from the neutron agent. OvS should not be removing the br-flat1 of it’s own accord. Something external is updating ovsdb to remove it’s record. Is ovn in the loop here? Is there and ovn-controller or similar process running on the host? I think this is important. As far as I know the decision to delete the bridge will not be made by vswitchd. It will be something external that will remove the record in ovsdb bridge table. If the neutron agent log does not mention it is doing this then maybe check the ovsdb .log file (it may not exist if it’s not configured on the ovsdb command line) – you’ll have to check the ovsdb man pages and also set. The ovsdb-tool is showing just the delete transaction so we need to find out where that delete request is coming from. Manoj :- I do not see any changes in the OVSDB logs after the configuration change. The attached zip file also contains neutron-ovs-agent logs. Issue-1 :- 1. Everything is up and running. That is all the bridges are displayed in OVS and no issues in the logs. [[BO'M]] can you add the output from vsctl show at this point so I can see the desired post agent restart state. Manoj :- Attached zip file contains the output of the ovs-vsctl show command before making changes to the neutron-ovs-agent and restarting the service. 2. I add the below entries in the OVS section of neutron's openvswitch-agent.ini file and restart the respective service. datapath_type=netdev vhostuser_socket_dir=/var/run/openvswitch 3. As mentioned earlier, bridge br-flat1 is deleted. At this point of time, observed the below. 3.1 br-int dtapath type changed from "system" to "netdev". 3.2. Not sure if it is an expected behavour, but there were MAC address changed prints only for both br-flat1 & br-int. 2018-10-23T14:39:40.253Z|41205|in_band|DBG|br-int: remote MAC address changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 2018-10-23T14:39:41.347Z|41374|in_band|DBG|br-flat1: remote MAC address changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 2018-10-23T14:39:48.229Z|41852|in_band|DBG|br-int: remote MAC address changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 2018-10-23T14:39:55.032Z|42008|in_band|DBG|br-int: remote MAC address changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 [[BO'M]] The datapath type change is expected. The MAC address changes I’m not sure. Manoj :- Ok. Thank you for the clarification. 3.3 interface state of br-int & eth8 (attached to br-flat1) are down. [[BO'M]] Can you copy the o/p from vsctl show again at this point. Manoj :- Attached zip file contains the ovs-vsctl output after changing the nutron-ovs-agent parameters and restartong the service. Please note that this time we also observed that 2 bridges got deleted. br-flat1 & br-vlan. Attaching the debug logs of ovs-vswitch. Issue-2 :- 1. Everything is up and running. That is all the bridges are displayed in OVS and no issues in the logs. I have one bridge named br0, which is of netdev type and I have attached a dpdk port, vhost-user port. Everthing is file. [[BO'M]] When you say everything is fine br-flat1 is still missing right? If that is the case lets stick with the missing bridge issue for now. (There is actually quite a few issues in the vsctl output below that we deal with after we figure out who deleted br-flat1). Manoj :- Ok Sure. 2. Now, in the OVS section ofneutron's openvswitch-agent.ini file, to the existing "bridge_mappings" key, I added an extra value "vlan1:br0". The new key-value pair now is as below. I wanted to create a new network and then map the bridge br0. So, I added this entry. bridge_mappings = flat:br-flat1,vxlan:br-vxlan,vlan:br-vlan,vlan1:br0 3. Now, when I restart the openvswitch-agent service, I observed that the datapath type of br0 changed from netdev to system. In the ovs-vsctl show output, I see the below. Bridge "br0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "vhost-user-1" Interface "vhost-user-1" type: dpdkvhostuser error: "could not add network device vhost-user-1 to ofproto (Invalid argument)" Port "br0" Interface "br0" type: internal Port "port0" Interface "port0" type: dpdk options: {dpdk-devargs="0000:af:00.1"} error: "could not add network device port0 to ofproto (Invalid argument)" Port "phy-br0" Interface "phy-br0" type: patch options: {peer="int-br0"} Attaching the logs for this issue as well. Thanks & Regards, PVMJ On Tue, Oct 23, 2018 at 5:44 PM O Mahony, Billy <[email protected]<mailto:[email protected]>> wrote: Hi Manojawa, So is there any remaining entry br-flat entry in the ovsdb? Does it give any clue to the reason – there may be a free-form ‘status’ or ‘info’ field for that purpose. I can understand the situation where a bridge might get incorrectly configured but I can’t understand why it is deleted by something other than the agent. Maybe it tries to create the bridge, there is some error so it decides to delete it. Are there more detailed log levels available for the agent? You may be able to turn on more detailed logging for the bridge logging in OvS too. /Billy. From: Manojawa Paritala [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, October 23, 2018 12:16 PM To: O Mahony, Billy <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens Hi Billy, Thank you for your reply. 1. Huge pages are properly set. Based on the dpdk configuration dpdk-socket-mem="4096,4096", 8 pages were created under /dev/hugepages. 2. dpdk-p0 is not attached to br-flat1. Actually I defined the bridge as br-flat1. 3. Yes, 'ovs-vsctl show' does not show br-flat1. As soon as I add the below entries in openvswitch-agent.ini and restart the neutron-openmvswitch-agent service, br-flat1 is getting deleted. I can see that in the ovs-vswitch logs and also in the output of "ovsdb-tool -mmm show-log" datapath_type=netdev vhostuser_socket_dir=/var/run/openvswitch 4. I do not see any errors in then neutron-openvswitch-agent logs, except for the below which are displayed after the bridge is deleted. ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-99a234e3-c943-4234-8c4d-f0fdc594df8f - - - - -] Bridge br-flat1 for physical network flat does not exist. Agent terminated! Thanks & Regards, PVMJ On Tue, Oct 23, 2018 at 3:06 PM O Mahony, Billy <[email protected]<mailto:[email protected]>> wrote: Hi, I don't see any errors relating to the dpdk interfaces. But it is also not clear where the user-space drivers are bound and the hugepage memory is set up. So double check those two items. Is the dpdk-p0 interface being attached to br-flat? Even if there are issues with the dpdk port the bridge should not be deleted (at least not automatically by OvS). Can you confirm with 'ovs-vsctl show' that the br-flat is actually not present after the agent is restarted. And that the dpdk-p0 is not reporting an error. What does the neutron-openmvswitch-agent logs say? Also run ovsdb-tool -mmm show-log which might give a clue as to when and how br-flat is being modified. Regards, Billy > -----Original Message----- > From: [email protected]<mailto:[email protected]> > [mailto:ovs-dev-<mailto:ovs-dev-> > [email protected]<mailto:[email protected]>] On Behalf Of > Manojawa Paritala > Sent: Monday, October 22, 2018 3:31 PM > To: [email protected]<mailto:[email protected]>; > [email protected]<mailto:[email protected]> > Subject: [ovs-dev] Issues configuring OVS-DPDK in openstack queens > > Hello All, > > On a 3 node (one controller + 2 compute), we configured Openstack Queens > using OSA with OVS. On all the nodes, we defined br-mgmt as linux bridge, br- > tun as private network and br-flat as external. > Installation was successful and we could create networks and instances on > Openstack. > > Below are the versions of the OVS packages used on each node. > > Controller :- openstack-vswitch - 2.9.0 > Computes :- openstack-vswitch-dpdk - 2.9.0 (as we wanted to configure dpdk on > the compute hosts) > > The openstack-vswitch-dpdk 2.9.0 package that we installed had dpdk version > 17.11.3. When we tried to enable DPDK it failed with the below error. > > dpdk|ERR|DPDK not supported in this copy of Open vSwitch > > So, we downloaded the sources for dpdk 17.11.4 and openvswitch 2.9.2, built > openvswitch with dpdk as suggested in the below official link. > No issues on Openstack or OVS. > http://docs.openvswitch.org/en/latest/intro/install/dpdk/ > > Then, we added the below parameters to OVS and everything looked ok. > No issues in Openstack or OVS. > > $ovs-vsctl get Open_vSwitch . other_config {dpdk-extra="-n 2", > dpdk-init="true", > dpdk-lcore-mask="0x300000000000", dpdk-socket-mem="4096,4096", pmd- > cpu-mask="0xf00003c0000", vhost-iommu-support="true"} > > Then on the compute node, in openvswitch_agent.ini file - OVS section, I added > the below (based on the link > https://docs.openstack.org/neutron/pike/contributor/internals/ovs_vhostuser.h > tml > ) > and restarted neutron-openmvswitch-agent service. > > datapath_type=netdev > vhostuser_socket_dir=/var/run/openvswitch > > After the above change, bridge br-flat is getting deleted from OVS. > Attached are the logs after I restart the neutron-openmvswitch-agent service > on > the compute now. Not sure what the issue is. > > Can any of you please let me know if we are missing anything? > > Best Regards, > PVMJ _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
