From: Manojawa Paritala [mailto:[email protected]]
Sent: Tuesday, October 23, 2018 5:37 PM
To: O Mahony, Billy <[email protected]>
Cc: [email protected]; [email protected]; Subba Rao Kodavalla
<[email protected]>; Song, Kee SangX <[email protected]>; Srinivasa Goda
<[email protected]>; Kris Rajana <[email protected]>
Subject: Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens
Hi Billy,
There are no br-flat1 entries in ovsdb.
As suggested I increased the log level to debug and then tried the same
scenario again. Though the result was same (br-flat1 getting deleted), I
observed the below 2 issues (i assume).
[[BO'M]] You can just increase the log level for the bridge module (all the
extra revalidator debug is making it hard to read the logs.):
ovs-appctl vlog/set bridge:file:dbg
[[BO'M]] Also is there logging from the neutron agent. OvS should not be
removing the br-flat1 of it’s own accord. Something external is updating ovsdb
to remove it’s record. Is ovn in the loop here? Is there and ovn-controller or
similar process running on the host?
I think this is important. As far as I know the decision to delete the bridge
will not be made by vswitchd. It will be something external that will remove
the record in ovsdb bridge table. If the neutron agent log does not mention it
is doing this then maybe check the ovsdb .log file (it may not exist if it’s
not configured on the ovsdb command line) – you’ll have to check the ovsdb man
pages and also set. The ovsdb-tool is showing just the delete transaction so we
need to find out where that delete request is coming from.
Issue-1 :-
1. Everything is up and running. That is all the bridges are displayed in OVS
and no issues in the logs.
[[BO'M]] can you add the output from vsctl show at this point so I can see the
desired post agent restart state.
2. I add the below entries in the OVS section of neutron's
openvswitch-agent.ini file and restart the respective service.
datapath_type=netdev
vhostuser_socket_dir=/var/run/openvswitch
3. As mentioned earlier, bridge br-flat1 is deleted. At this point of time,
observed the below.
3.1 br-int dtapath type changed from "system" to "netdev".
3.2. Not sure if it is an expected behavour, but there were MAC address changed
prints only for both br-flat1 & br-int.
2018-10-23T14:39:40.253Z|41205|in_band|DBG|br-int: remote MAC address changed
from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
2018-10-23T14:39:41.347Z|41374|in_band|DBG|br-flat1: remote MAC address changed
from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
2018-10-23T14:39:48.229Z|41852|in_band|DBG|br-int: remote MAC address changed
from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
2018-10-23T14:39:55.032Z|42008|in_band|DBG|br-int: remote MAC address changed
from 00:00:00:00:00:00 to 00:a0:c9:0e:01:01
[[BO'M]] The datapath type change is expected. The MAC address changes I’m not
sure.
3.3 interface state of br-int & eth8 (attached to br-flat1) are down.
[[BO'M]] Can you copy the o/p from vsctl show again at this point.
Attaching the debug logs of ovs-vswitch.
Issue-2 :-
1. Everything is up and running. That is all the bridges are displayed in OVS
and no issues in the logs. I have one bridge named br0, which is of netdev type
and I have attached a dpdk port, vhost-user port. Everthing is file.
[[BO'M]] When you say everything is fine br-flat1 is still missing right? If
that is the case lets stick with the missing bridge issue for now. (There is
actually quite a few issues in the vsctl output below that we deal with after
we figure out who deleted br-flat1).
2. Now, in the OVS section ofneutron's openvswitch-agent.ini file, to the
existing "bridge_mappings" key, I added an extra value "vlan1:br0". The new
key-value pair now is as below. I wanted to create a new network and then map
the bridge br0. So, I added this entry.
bridge_mappings = flat:br-flat1,vxlan:br-vxlan,vlan:br-vlan,vlan1:br0
3. Now, when I restart the openvswitch-agent service, I observed that the
datapath type of br0 changed from netdev to system. In the ovs-vsctl show
output, I see the below.
Bridge "br0"
Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>"
is_connected: true
fail_mode: secure
Port "vhost-user-1"
Interface "vhost-user-1"
type: dpdkvhostuser
error: "could not add network device vhost-user-1 to ofproto
(Invalid argument)"
Port "br0"
Interface "br0"
type: internal
Port "port0"
Interface "port0"
type: dpdk
options: {dpdk-devargs="0000:af:00.1"}
error: "could not add network device port0 to ofproto (Invalid
argument)"
Port "phy-br0"
Interface "phy-br0"
type: patch
options: {peer="int-br0"}
Attaching the logs for this issue as well.
Thanks & Regards,
PVMJ
On Tue, Oct 23, 2018 at 5:44 PM O Mahony, Billy
<[email protected]<mailto:[email protected]>> wrote:
Hi Manojawa,
So is there any remaining entry br-flat entry in the ovsdb? Does it give any
clue to the reason – there may be a free-form ‘status’ or ‘info’ field for that
purpose.
I can understand the situation where a bridge might get incorrectly configured
but I can’t understand why it is deleted by something other than the agent.
Maybe it tries to create the bridge, there is some error so it decides to
delete it. Are there more detailed log levels available for the agent? You may
be able to turn on more detailed logging for the bridge logging in OvS too.
/Billy.
From: Manojawa Paritala
[mailto:[email protected]<mailto:[email protected]>]
Sent: Tuesday, October 23, 2018 12:16 PM
To: O Mahony, Billy <[email protected]<mailto:[email protected]>>
Cc: [email protected]<mailto:[email protected]>;
[email protected]<mailto:[email protected]>
Subject: Re: [ovs-dev] Issues configuring OVS-DPDK in openstack queens
Hi Billy,
Thank you for your reply.
1. Huge pages are properly set. Based on the dpdk configuration
dpdk-socket-mem="4096,4096", 8 pages were created under /dev/hugepages.
2. dpdk-p0 is not attached to br-flat1. Actually I defined the bridge as
br-flat1.
3. Yes, 'ovs-vsctl show' does not show br-flat1. As soon as I add the below
entries in openvswitch-agent.ini and restart the neutron-openmvswitch-agent
service, br-flat1 is getting deleted. I can see that in the ovs-vswitch logs
and also in the output of "ovsdb-tool -mmm show-log"
datapath_type=netdev
vhostuser_socket_dir=/var/run/openvswitch
4. I do not see any errors in then neutron-openvswitch-agent logs, except for
the below which are displayed after the bridge is deleted.
ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-99a234e3-c943-4234-8c4d-f0fdc594df8f - - - - -] Bridge br-flat1 for
physical network flat does not exist. Agent terminated!
Thanks & Regards,
PVMJ
On Tue, Oct 23, 2018 at 3:06 PM O Mahony, Billy
<[email protected]<mailto:[email protected]>> wrote:
Hi,
I don't see any errors relating to the dpdk interfaces. But it is also not
clear where the user-space drivers are bound and the hugepage memory is set up.
So double check those two items.
Is the dpdk-p0 interface being attached to br-flat? Even if there are issues
with the dpdk port the bridge should not be deleted (at least not automatically
by OvS).
Can you confirm with 'ovs-vsctl show' that the br-flat is actually not present
after the agent is restarted. And that the dpdk-p0 is not reporting an error.
What does the neutron-openmvswitch-agent logs say?
Also run ovsdb-tool -mmm show-log which might give a clue as to when and how
br-flat is being modified.
Regards,
Billy
> -----Original Message-----
> From: [email protected]<mailto:[email protected]>
> [mailto:ovs-dev-<mailto:ovs-dev->
> [email protected]<mailto:[email protected]>] On Behalf Of
> Manojawa Paritala
> Sent: Monday, October 22, 2018 3:31 PM
> To: [email protected]<mailto:[email protected]>;
> [email protected]<mailto:[email protected]>
> Subject: [ovs-dev] Issues configuring OVS-DPDK in openstack queens
>
> Hello All,
>
> On a 3 node (one controller + 2 compute), we configured Openstack Queens
> using OSA with OVS. On all the nodes, we defined br-mgmt as linux bridge, br-
> tun as private network and br-flat as external.
> Installation was successful and we could create networks and instances on
> Openstack.
>
> Below are the versions of the OVS packages used on each node.
>
> Controller :- openstack-vswitch - 2.9.0
> Computes :- openstack-vswitch-dpdk - 2.9.0 (as we wanted to configure dpdk on
> the compute hosts)
>
> The openstack-vswitch-dpdk 2.9.0 package that we installed had dpdk version
> 17.11.3. When we tried to enable DPDK it failed with the below error.
>
> dpdk|ERR|DPDK not supported in this copy of Open vSwitch
>
> So, we downloaded the sources for dpdk 17.11.4 and openvswitch 2.9.2, built
> openvswitch with dpdk as suggested in the below official link.
> No issues on Openstack or OVS.
> http://docs.openvswitch.org/en/latest/intro/install/dpdk/
>
> Then, we added the below parameters to OVS and everything looked ok.
> No issues in Openstack or OVS.
>
> $ovs-vsctl get Open_vSwitch . other_config {dpdk-extra="-n 2",
> dpdk-init="true",
> dpdk-lcore-mask="0x300000000000", dpdk-socket-mem="4096,4096", pmd-
> cpu-mask="0xf00003c0000", vhost-iommu-support="true"}
>
> Then on the compute node, in openvswitch_agent.ini file - OVS section, I added
> the below (based on the link
> https://docs.openstack.org/neutron/pike/contributor/internals/ovs_vhostuser.h
> tml
> )
> and restarted neutron-openmvswitch-agent service.
>
> datapath_type=netdev
> vhostuser_socket_dir=/var/run/openvswitch
>
> After the above change, bridge br-flat is getting deleted from OVS.
> Attached are the logs after I restart the neutron-openmvswitch-agent service
> on
> the compute now. Not sure what the issue is.
>
> Can any of you please let me know if we are missing anything?
>
> Best Regards,
> PVMJ
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
