Re: [ovs-discuss] OVS misconfiguration issue
Hi All, The issue here is we are trying to send the packets on the same device in a loop. While sending on a device, a spinlock for the tx queue has to be acquired in dev_queue_xmit function. This is where we are trying to acquire the same lock again, which is leading to the kernel crash. This issue becomes worse if internal ports are involved in the configuration. I think, we should avoid these loops in the vport send functions. But in case of tunneling, the tunnel send function should take care of these checks. Please share your thoughts on this issue. On Mon, Apr 30, 2018 at 11:17 AM, Neelakantam Gaddam wrote: > Hi All, > > > > OVS misconfiguration leading to spinlock recursion in dev_queue_xmit. > > > > We are running ovs-2.8.1 with openvswitch kernel modules on two hosts > connected back to back. We are running OVS on MIPS64 platform. > > > > We are using the below configuration. > > > > ovs-vsctl add-br br0 > > ovs-vsctl add-bond br0 bond0 p1p1 p1p2 > > ovs-vsctl set port bond0 lacp=active bond_mode=balance-tcp > > ifconfig br0 100.0.0.1 up > > ovs-vsctl add-port br0 veth0 > > ovs-vsctl add-port br0 vx0 -- set interface vx0 type=vxlan > options:local_ip=100.0.0.1 options:remote_ip=100.0.0.2 option:key=flow > > > > ovs-ofctl add-flow br0 "table=0, priority=1, cookie=100, tun_id=100, > in_port=4, action=output:3" > > ovs-ofctl add-flow br0 "table=0, priority=1, cookie=100, in_port=3, > actions=set_field:100->tun_id output:4" > > > > > > When this configuration is applied on both hosts, we are seeing the below > spinlock recursion bug. > > > > [] show_stack+0x6c/0xf8 > > > [] do_raw_spin_lock+0x168/0x170 > > > [] dev_queue_xmit+0x43c/0x470 > > > [] ip_finish_output+0x250/0x490 > > > [] rpl_iptunnel_xmit+0x134/0x218 [openvswitch] > > > [] rpl_vxlan_xmit+0x430/0x538 [openvswitch] > > > [] do_execute_actions+0x18f8/0x19e8 [openvswitch] > > > [] ovs_execute_actions+0x90/0x208 [openvswitch] > > > [] ovs_dp_process_packet+0xb0/0x1a8 [openvswitch] > > > [] ovs_vport_receive+0x78/0x130 [openvswitch] > > > [] internal_dev_xmit+0x34/0x98 [openvswitch] > > > [] dev_hard_start_xmit+0x2e8/0x4f8 > > > [] sch_direct_xmit+0xf0/0x238 > > > [] dev_queue_xmit+0x1d8/0x470 > > > [] arp_process+0x614/0x628 > > > [] __netif_receive_skb_core+0x2e8/0x5d8 > > > [] process_backlog+0xc0/0x1b0 > > > [] net_rx_action+0x154/0x240 > > > [] __do_softirq+0x1d0/0x218 > > > [] do_softirq+0x68/0x70 > > > [] local_bh_enable+0xa8/0xb0 > > > [] netif_rx_ni+0x20/0x30 > > > > The packet path traced is : netif_rx->arp->dev_queue_xmit(internal > port)->vxlan_xmit->dev_queue_xmit(internal port). According to the > configuration, this packet path is valid. But we should not hit the crash. > > > Questions: > > >- Is it a kernel bug or ovs bug ? >- How OVS handles these kind of misconfigurations especially packet >loops involved? > > > > Any suggestion or help is greatly appreciated. > > > > > > > > Thanks > > > -- > Thanks & Regards > Neelakantam Gaddam > -- Thanks & Regards Neelakantam Gaddam ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovs-vswitchd 2.7.3 crash
On Tue, May 01, 2018 at 03:41:56PM -0400, Nicolas Bouliane via discuss wrote: > Hi, > > We experienced a crash and we generated the backtrace from the core dump > with that command: gdb -se "ovs-vswitchd" -ex "bt full" -ex "thread apply > all bt" --batch -c /core > /tmp/backtrace.txt > > The file is in attachement. > > # ovs-vswitchd --version > ovs-vswitchd (Open vSwitch) 2.7.3 > > # cat /proc/version > Linux version 4.9.77-1-generic (root@c509c24b315f) (gcc version 7.2.0 (GCC) > ) #4+1516907002do~691dc34 SMP Thu Jan 25 19:04:34 UTC 2018 > > # lsmod | grep switch > openvswitch 139264 50 vport_stt > nf_nat_ipv616384 1 openvswitch > nf_nat_ipv416384 2 openvswitch,iptable_nat > nf_nat 24576 3 openvswitch,nf_nat_ipv6,nf_nat_ipv4 > udp_tunnel 16384 1 openvswitch > nf_defrag_ipv6 36864 2 nf_conntrack_ipv6,openvswitch > nf_conntrack 110592 9 > nf_conntrack_ipv6,nf_conntrack_ftp,openvswitch,nf_conntrack_ipv4,nf_conntrack_netlink,xt_CT,nf_nat_ipv6,nf_nat_ipv4,nf_nat > libcrc32c 16384 2 openvswitch,raid456 > > Let me know if you need more information. We are curious to know if it is a > know bug, or if it is something new. It's not obviously a known bug. It might be related to memory corruption, because SIGBUS can be caused by reading or writing a wild pointer and calloc(), which is at the top of the backtrace, can definitely do either one if there's been memory corruption. OVS 2.7.4 has a few memory corruption fixes, relative to OVS 2.7.3, so possibly it's a bug that's already fixed. Is this something that you see repeatedly? ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] Geneve and IPv6 (was: Re: could not add network device ovn-xxxxxx-0 to ofproto (Address family not supported by protocol))
Hi Greg.
We've had multiple reports now that Geneve kernel-based tunnels do not
work in Open vSwitch if IPv6 is not enabled. Do you have an idea
whether we should consider this a bug in OVS or a bug in the
documentation? That is, should we plan to fix it (when we can) or
should we plan to document that Geneve requires IPv6? (Or is something
else going on?)
No rush.
Thanks,
Ben.
On Mon, Apr 30, 2018 at 10:25:06AM +0200, Matthias May wrote:
> On 27/04/18 14:20, Paul Greenberg wrote:
> >
> > When IPv6 is disables in `/boot/grub2/grub.cfg`:
> >
> > ```
> > linux16 ... ipv6.disable=1
> > ```
> >
> > Then GENEVE interface does not come up properly.
> >
> > ```
> > # ovs-vsctl show
> > Port "ovn-14a54f-0"
> > Interface "ovn-14a54f-0"
> > type: geneve
> > options: {csum="true", key=flow, remote_ip="10.10.10.10"}
> > error: "could not add network device ovn-14a54f-0 to
> > ofproto (Address family not supported by protocol)"
> > ```
> >
> > Additionally, run the following commands:
> >
> > ```
> > # cat /proc/version
> > Linux version 3.10.0-514.21.1.el7.x86_64
> > (mockbu...@x86-020.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623
> > (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Apr 22 02:41:35 EDT 2017
> >
> > # ovs-vswitchd --version
> > ovs-vswitchd (Open vSwitch) 2.9.0
> > ```
> >
> > When IPv6 is enabled, then OVN works properly.
> > ___
> > discuss mailing list
> > disc...@openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> >
>
> Hi
> I remember running into the same when playing around with GRE on a system
> where IPv6 was disabled.
>
> BR
> Matthias
> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVS as a destination
On Tue, May 01, 2018 at 01:25:40PM -0700, Justin Pettit wrote: > > > On May 1, 2018, at 11:21 AM, Sh j wrote: > > > > If the destination of a packet be an OVS, is there any way to set the OVS > > to get the packet without sending any packet_in messages to the controller > > for handling this packet? > > I'm not sure that I understand the question. Are you asking is there a way > for OVS to send some packets to the controller without sending an OpenFlow > packet-in? I suppose you could mirror the packet to a system through a > dedicated port, VLAN, or tunnel. However, if you want to be using OpenFlow, > packet-in is the natural way for sending packets to a controller. Another way to interpret the question is, "Can I set up a flow without receiving a packet-in?" If that's the question, then the answer is "yes: the flow table can be modified independent of traffic." ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] bug: ovs-vsctl: Fix segfault when attempting to del-port from parent bridge
Greetings! I encountered a bug in ovs-vsctl that may be worth sharing with you. The code path in the error case when bridge's del-port is attempted for the parent (instead of the fake-bridge) segfaults. Here are some simple steps for reproducing this issue: PARENT_BRIDGE=br0 ; FAKE_BRIDGE=br0c ; VLAN_TAG=666 ovs-vsctl add-br $PARENT_BRIDGE ovs-vsctl add-br $FAKE_BRIDGE $PARENT_BRIDGE $VLAN_TAG ovs-vsctl add-port $FAKE_BRIDGE p1 ovs-vsctl del-port $PARENT_BRIDGE p1 I submitted more info and a proposed fix to ovs-dev [1]. Please see if it makes sense and let me know if that is useful. Thanks, -- flaviof [1]: https://patchwork.ozlabs.org/patch/907245/ https://patchwork.ozlabs.org/patch/907246/ signature.asc Description: Message signed with OpenPGP ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OpenVswitch on a host with only one cellular modemconnected to internet
You need to make sure that the cellular interface is getting an IP address, and that that IP address can route to the controller - which is unlikely by default. There's nothing special about OVS in this case - can you ping your controller when ethernet is off and apn is on? On Tue, May 1, 2018 at 4:27 PM, Justin Pettit wrote: > > > > On Apr 30, 2018, at 7:59 PM, EMANUEL FERNANDO MONTOYA GOMEZ < > emanuel.mont...@udea.edu.co> wrote: > > > > I am working in a project on which I want to have a couple of hosts that > has 2 ethernet interfaces and 1 cellular modem, but only the cellular modem > has internet access. > > I haven't been able to make them connect to controller. > > When I try using the ethernet interfaces it works perfectly, but I need > to make it work using the cellular one. > > I use nmcli to connect to cellular APN and it works (when not using > openvswitch) > > I appreciate any suggestion you may give on this topic. I haven't found > information on this specific subject. > > I'm not familiar with this particular use-case. However, as long as the > cellular APN shows up as a Linux system device and delivers Ethernet > frames, I don't know why it wouldn't work; OVS doesn't really care what > kind of interface is attached. > > --Justin > > > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > -- Blue Lang VP, Engineering *| *Veracity 3423 Piedmont Rd NE Suite 400 Atlanta, GA 30305 Cell: (770) 265-1381 <+17702651381> https://www.linkedin.com/in/bluelang/ b...@veracity.io www.veracity.io ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OpenVswitch on a host with only one cellular modemconnected to internet
> On Apr 30, 2018, at 7:59 PM, EMANUEL FERNANDO MONTOYA GOMEZ > wrote: > > I am working in a project on which I want to have a couple of hosts that has > 2 ethernet interfaces and 1 cellular modem, but only the cellular modem has > internet access. > I haven't been able to make them connect to controller. > When I try using the ethernet interfaces it works perfectly, but I need to > make it work using the cellular one. > I use nmcli to connect to cellular APN and it works (when not using > openvswitch) > I appreciate any suggestion you may give on this topic. I haven't found > information on this specific subject. I'm not familiar with this particular use-case. However, as long as the cellular APN shows up as a Linux system device and delivers Ethernet frames, I don't know why it wouldn't work; OVS doesn't really care what kind of interface is attached. --Justin ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] OVS as a destination
> On May 1, 2018, at 11:21 AM, Sh j wrote: > > If the destination of a packet be an OVS, is there any way to set the OVS to > get the packet without sending any packet_in messages to the controller for > handling this packet? I'm not sure that I understand the question. Are you asking is there a way for OVS to send some packets to the controller without sending an OpenFlow packet-in? I suppose you could mirror the packet to a system through a dedicated port, VLAN, or tunnel. However, if you want to be using OpenFlow, packet-in is the natural way for sending packets to a controller. --Justin ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] OVS as a destination
Hello, If the destination of a packet be an OVS, is there any way to set the OVS to get the packet without sending any packet_in messages to the controller for handling this packet? Thank you ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
