Hi Billy, The reason for br-vlan getting deleted in my previous attempt was due to a small misconfiguration in the openstack-agent.ini. I corrected it and re-done the test. Attaching the new logs with this email. Please ignore the earlier logs.
Apologies for the confusion. Thanks & Regards, PVMJ On Wed, Oct 24, 2018 at 7:26 PM Manojawa Paritala <manojaw...@biarca.com> wrote: > Hi Billy, > > Firstly, thank you for taking a look at the logs I sent yesterday. > > Please see in-line. Please let me know if I miss anything. > > Thanks & Regards, > PVMJ > > On Wed, Oct 24, 2018 at 3:36 PM O Mahony, Billy <billy.o.mah...@intel.com> > wrote: > >> >> >> >> >> *From:* Manojawa Paritala [mailto:manojaw...@biarca.com] >> *Sent:* Tuesday, October 23, 2018 5:37 PM >> *To:* O Mahony, Billy <billy.o.mah...@intel.com> >> *Cc:* ovs-disc...@openvswitch.org; ovs-dev@openvswitch.org; Subba Rao >> Kodavalla <s...@biarca.com>; Song, Kee SangX <kee.sangx.s...@intel.com>; >> Srinivasa Goda <sri...@biarca.com>; Kris Rajana <kr...@biarca.com> >> *Subject:* Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens >> >> >> >> Hi Billy, >> >> >> >> There are no br-flat1 entries in ovsdb. >> >> >> >> As suggested I increased the log level to debug and then tried the same >> scenario again. Though the result was same (br-flat1 getting deleted), I >> observed the below 2 issues (i assume). >> >> *[[BO'M]] You can just increase the log level for the bridge module (all >> the extra revalidator debug is making it hard to read the logs.):* >> >> ovs-appctl vlog/set bridge:file:dbg >> >> >> > *Manoj :- Added the log level of debug to only bridge module and collected > the logs. Attached zip file has the logs.* > > >> *[[BO'M]] Also is there logging from the neutron agent. OvS should not be >> removing the br-flat1 of it’s own accord. Something external is updating >> ovsdb to remove it’s record. Is ovn in the loop here? Is there and >> ovn-controller or similar process running on the host?* >> >> *I think this is important. As far as I know the decision to delete the >> bridge will not be made by vswitchd. It will be something external that >> will remove the record in ovsdb bridge table. If the neutron agent log does >> not mention it is doing this then maybe check the ovsdb .log file (it may >> not exist if it’s not configured on the ovsdb command line) – you’ll have >> to check the ovsdb man pages and also set. The ovsdb-tool is showing just >> the delete transaction so we need to find out where that delete request is >> coming from.* >> >> >> > *Manoj :- I do not see any changes in the OVSDB logs after the > configuration change. The attached zip file also contains neutron-ovs-agent > logs.* > > >> Issue-1 :- >> >> 1. Everything is up and running. That is all the bridges are displayed in >> OVS and no issues in the logs. >> >> >> >> *[[BO'M]] can you add the output from vsctl show at this point so I can >> see the desired post agent restart state.* >> > *Manoj :- Attached zip file contains the output of the ovs-vsctl show > command before making changes to the neutron-ovs-agent and restarting the > service.* > > >> 2. I add the below entries in the OVS section of neutron's >> openvswitch-agent.ini file and restart the respective service. >> >> >> >> datapath_type=netdev >> >> vhostuser_socket_dir=/var/run/openvswitch >> >> >> >> 3. As mentioned earlier, bridge br-flat1 is deleted. At this point of >> time, observed the below. >> >> >> >> 3.1 br-int dtapath type changed from "system" to "netdev". >> >> 3.2. Not sure if it is an expected behavour, but there were MAC address >> changed prints only for both br-flat1 & br-int. >> >> >> >> 2018-10-23T14:39:40.253Z|41205|in_band|DBG|br-int: remote MAC address >> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 >> >> 2018-10-23T14:39:41.347Z|41374|in_band|DBG|br-flat1: remote MAC address >> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 >> >> 2018-10-23T14:39:48.229Z|41852|in_band|DBG|br-int: remote MAC address >> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 >> >> 2018-10-23T14:39:55.032Z|42008|in_band|DBG|br-int: remote MAC address >> changed from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01 >> >> *[[BO'M]] The datapath type change is expected. The MAC address changes >> I’m not sure.* >> > *Manoj :- Ok. Thank you for the clarification.* > >> >> >> 3.3 interface state of br-int & eth8 (attached to br-flat1) are down. >> >> *[[BO'M]] Can you copy the o/p from vsctl show again at this point.* >> > *Manoj :- Attached zip file contains the ovs-vsctl output after changing > the nutron-ovs-agent parameters and restartong the service. Please note > that this time we also observed that 2 bridges got deleted. br-flat1 & > br-vlan. * > >> >> >> Attaching the debug logs of ovs-vswitch. >> >> >> >> >> >> Issue-2 :- >> >> 1. Everything is up and running. That is all the bridges are displayed in >> OVS and no issues in the logs. I have one bridge named br0, which is of >> netdev type and I have attached a dpdk port, vhost-user port. Everthing is >> file. >> >> >> >> >> >> *[[BO'M]] When you say everything is fine br-flat1 is still missing >> right? If that is the case lets stick with the missing bridge issue for >> now. (There is actually quite a few issues in the vsctl output below that >> we deal with after we figure out who deleted br-flat1).* >> > *Manoj :- Ok Sure. * > >> >> >> 2. Now, in the OVS section ofneutron's openvswitch-agent.ini file, to the >> existing "bridge_mappings" key, I added an extra value "vlan1:br0". The new >> key-value pair now is as below. I wanted to create a new network and then >> map the bridge br0. So, I added this entry. >> >> >> >> bridge_mappings = flat:br-flat1,vxlan:br-vxlan,vlan:br-vlan,vlan1:br0 >> >> >> >> 3. Now, when I restart the openvswitch-agent service, I observed that the >> datapath type of br0 changed from netdev to system. In the ovs-vsctl show >> output, I see the below. >> >> >> >> Bridge "br0" >> >> Controller "tcp:127.0.0.1:6633" >> >> is_connected: true >> >> fail_mode: secure >> >> Port "vhost-user-1" >> >> Interface "vhost-user-1" >> >> type: dpdkvhostuser >> >> error: "could not add network device vhost-user-1 to >> ofproto (Invalid argument)" >> >> Port "br0" >> >> Interface "br0" >> >> type: internal >> >> Port "port0" >> >> Interface "port0" >> >> type: dpdk >> >> options: {dpdk-devargs="0000:af:00.1"} >> >> error: "could not add network device port0 to ofproto >> (Invalid argument)" >> >> Port "phy-br0" >> >> Interface "phy-br0" >> >> type: patch >> >> options: {peer="int-br0"} >> >> >> >> Attaching the logs for this issue as well. >> >> >> >> >> >> Thanks & Regards, >> >> PVMJ >> >> >> >> On Tue, Oct 23, 2018 at 5:44 PM O Mahony, Billy <billy.o.mah...@intel.com> >> wrote: >> >> Hi Manojawa, >> >> >> >> So is there any remaining entry br-flat entry in the ovsdb? Does it give >> any clue to the reason – there may be a free-form ‘status’ or ‘info’ field >> for that purpose. >> >> >> >> I can understand the situation where a bridge might get incorrectly >> configured but I can’t understand why it is deleted by something other than >> the agent. >> >> >> >> Maybe it tries to create the bridge, there is some error so it decides to >> delete it. Are there more detailed log levels available for the agent? You >> may be able to turn on more detailed logging for the bridge logging in OvS >> too. >> >> >> >> /Billy. >> >> >> >> >> >> *From:* Manojawa Paritala [mailto:manojaw...@biarca.com] >> *Sent:* Tuesday, October 23, 2018 12:16 PM >> *To:* O Mahony, Billy <billy.o.mah...@intel.com> >> *Cc:* ovs-disc...@openvswitch.org; ovs-dev@openvswitch.org >> *Subject:* Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens >> >> >> >> Hi Billy, >> >> >> >> Thank you for your reply. >> >> >> >> 1. Huge pages are properly set. Based on the dpdk configuration >> dpdk-socket-mem="4096,4096", 8 pages were created under /dev/hugepages. >> >> 2. dpdk-p0 is not attached to br-flat1. Actually I defined the bridge as >> br-flat1. >> >> 3. Yes, 'ovs-vsctl show' does not show br-flat1. As soon as I add the >> below entries in openvswitch-agent.ini and restart the >> neutron-openmvswitch-agent service, br-flat1 is getting deleted. I can see >> that in the ovs-vswitch logs and also in the output of "ovsdb-tool -mmm >> show-log" >> >> >> >> datapath_type=netdev >> >> vhostuser_socket_dir=/var/run/openvswitch >> >> >> >> 4. I do not see any errors in then neutron-openvswitch-agent logs, except >> for the below which are displayed after the bridge is deleted. >> >> >> >> ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent >> [req-99a234e3-c943-4234-8c4d-f0fdc594df8f - - - - -] Bridge br-flat1 for >> physical network flat does not exist. Agent terminated! >> >> >> >> Thanks & Regards, >> >> PVMJ >> >> >> >> On Tue, Oct 23, 2018 at 3:06 PM O Mahony, Billy <billy.o.mah...@intel.com> >> wrote: >> >> Hi, >> >> I don't see any errors relating to the dpdk interfaces. But it is also >> not clear where the user-space drivers are bound and the hugepage memory is >> set up. So double check those two items. >> >> Is the dpdk-p0 interface being attached to br-flat? Even if there are >> issues with the dpdk port the bridge should not be deleted (at least not >> automatically by OvS). >> >> Can you confirm with 'ovs-vsctl show' that the br-flat is actually not >> present after the agent is restarted. And that the dpdk-p0 is not reporting >> an error. >> >> What does the neutron-openmvswitch-agent logs say? >> >> Also run ovsdb-tool -mmm show-log which might give a clue as to when and >> how br-flat is being modified. >> >> Regards, >> Billy >> >> > -----Original Message----- >> > From: ovs-dev-boun...@openvswitch.org [mailto:ovs-dev- >> > boun...@openvswitch.org] On Behalf Of Manojawa Paritala >> > Sent: Monday, October 22, 2018 3:31 PM >> > To: ovs-disc...@openvswitch.org; ovs-dev@openvswitch.org >> > Subject: [ovs-dev] Issues configuring OVS-DPDK in openstack queens >> > >> > Hello All, >> > >> > On a 3 node (one controller + 2 compute), we configured Openstack Queens >> > using OSA with OVS. On all the nodes, we defined br-mgmt as linux >> bridge, br- >> > tun as private network and br-flat as external. >> > Installation was successful and we could create networks and instances >> on >> > Openstack. >> > >> > Below are the versions of the OVS packages used on each node. >> > >> > Controller :- openstack-vswitch - 2.9.0 >> > Computes :- openstack-vswitch-dpdk - 2.9.0 (as we wanted to configure >> dpdk on >> > the compute hosts) >> > >> > The openstack-vswitch-dpdk 2.9.0 package that we installed had dpdk >> version >> > 17.11.3. When we tried to enable DPDK it failed with the below error. >> > >> > dpdk|ERR|DPDK not supported in this copy of Open vSwitch >> > >> > So, we downloaded the sources for dpdk 17.11.4 and openvswitch 2.9.2, >> built >> > openvswitch with dpdk as suggested in the below official link. >> > No issues on Openstack or OVS. >> > http://docs.openvswitch.org/en/latest/intro/install/dpdk/ >> > >> > Then, we added the below parameters to OVS and everything looked ok. >> > No issues in Openstack or OVS. >> > >> > $ovs-vsctl get Open_vSwitch . other_config {dpdk-extra="-n 2", >> dpdk-init="true", >> > dpdk-lcore-mask="0x300000000000", dpdk-socket-mem="4096,4096", pmd- >> > cpu-mask="0xf00003c0000", vhost-iommu-support="true"} >> > >> > Then on the compute node, in openvswitch_agent.ini file - OVS section, >> I added >> > the below (based on the link >> > >> https://docs.openstack.org/neutron/pike/contributor/internals/ovs_vhostuser.h >> > tml >> > ) >> > and restarted neutron-openmvswitch-agent service. >> > >> > datapath_type=netdev >> > vhostuser_socket_dir=/var/run/openvswitch >> > >> > After the above change, bridge br-flat is getting deleted from OVS. >> > Attached are the logs after I restart the neutron-openmvswitch-agent >> service on >> > the compute now. Not sure what the issue is. >> > >> > Can any of you please let me know if we are missing anything? >> > >> > Best Regards, >> > PVMJ >> >>
_______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
