Thanks Ben! I will try to walk through this today and see how much miles I can
get out of it.
John
-----Original Message-----
From: Ben Pfaff <b...@ovn.org>
Sent: Thursday, February 28, 2019 9:02 PM
To: John Carew <johnwca...@gmail.com>
Cc: ovs-discuss@openvswitch.org
Subject: Re: [ovs-discuss] Router not allowing traffic in reverse
OK.
The OVS FAQ has some advice:
Q: I have a sophisticated network setup involving Open vSwitch, VMs or multiple
hosts, and other components. The behavior isn't what I expect. Help!
A: To debug network behavior problems, trace the path of a packet,
hop-by-hop, from its origin in one host to a remote host. If that's
correct, then trace the path of the response packet back to the origin.
The open source tool called ``plotnetcfg`` can help to understand the
relationship between the networking devices on a single host.
Usually a simple ICMP echo request and reply (``ping``) packet is good
enough. Start by initiating an ongoing ``ping`` from the origin host to a
remote host. If you are tracking down a connectivity problem, the "ping"
will not display any successful output, but packets are still being sent.
(In this case the packets being sent are likely ARP rather than ICMP.)
Tools available for tracing include the following:
- ``tcpdump`` and ``wireshark`` for observing hops across network devices,
such as Open vSwitch internal devices and physical wires.
- ``ovs-appctl dpif/dump-flows <br>`` in Open vSwitch 1.10 and later or
``ovs-dpctl dump-flows <br>`` in earlier versions. These tools allow one
to observe the actions being taken on packets in ongoing flows.
See ovs-vswitchd(8) for ``ovs-appctl dpif/dump-flows`` documentation,
ovs-dpctl(8) for ``ovs-dpctl dump-flows`` documentation, and "Why are
there so many different ways to dump flows?" above for some background.
- ``ovs-appctl ofproto/trace`` to observe the logic behind how ovs-vswitchd
treats packets. See ovs-vswitchd(8) for documentation. You can out more
details about a given flow that ``ovs-dpctl dump-flows`` displays, by
cutting and pasting a flow from the output into an ``ovs-appctl
ofproto/trace`` command.
- SPAN, RSPAN, and ERSPAN features of physical switches, to observe what
goes on at these physical hops.
Starting at the origin of a given packet, observe the packet at each hop in
turn. For example, in one plausible scenario, you might:
1. ``tcpdump`` the ``eth`` interface through which an ARP egresses a VM,
from inside the VM.
2. ``tcpdump`` the ``vif`` or ``tap`` interface through which the ARP
ingresses the host machine.
3. Use ``ovs-dpctl dump-flows`` to spot the ARP flow and observe the host
interface through which the ARP egresses the physical machine. You may
need to use ``ovs-dpctl show`` to interpret the port numbers. If the
output seems surprising, you can use ``ovs-appctl ofproto/trace`` to
observe details of how ovs-vswitchd determined the actions in the
``ovs-dpctl dump-flows`` output.
4. ``tcpdump`` the ``eth`` interface through which the ARP egresses the
physical machine.
5. ``tcpdump`` the ``eth`` interface through which the ARP ingresses the
physical machine, at the remote host that receives the ARP.
6. Use ``ovs-dpctl dump-flows`` to spot the ARP flow on the remote host
remote host that receives the ARP and observe the VM ``vif`` or ``tap``
interface to which the flow is directed. Again, ``ovs-dpctl show`` and
``ovs-appctl ofproto/trace`` might help.
7. ``tcpdump`` the ``vif`` or ``tap`` interface to which the ARP is
directed.
8. ``tcpdump`` the ``eth`` interface through which the ARP ingresses a VM,
from inside the VM.
It is likely that during one of these steps you will figure out the
problem. If not, then follow the ARP reply back to the origin, in reverse.
On Thu, Feb 28, 2019 at 06:34:17PM -0600, John Carew wrote:
> I think there is some misunderstanding, I don't think OVS is working
> incorrectly or has a problem. I believe it is working exactly how it is
> configured. I believe OpenStack is just configuring OVS, and it has put in a
> flow rule or something that is allowing outbound for a network/subnet but not
> inbound. I'm asking how to figure out where in OVS it is stopping at.
>
> > On Feb 28, 2019, at 5:51 PM, Ben Pfaff <b...@ovn.org> wrote:
> >
> > Do you have a reason to believe that it's a problem in OVS, rather
> > than a problem in what OpenStack is telling OVS to do?
> >
> > On Thu, Feb 28, 2019 at 04:48:44PM -0600, John Carew wrote:
> >> I agree, but I want to diagnose what is causing it in ovs first before I
> >> go to them.
> >>
> >> John
> >>
> >>>> On Feb 28, 2019, at 4:33 PM, Ben Pfaff <b...@ovn.org> wrote:
> >>>>
> >>>> On Wed, Feb 27, 2019 at 09:26:16PM -0600, John Carew wrote:
> >>>> I have setup OpenStack with OVS. I have a single Hyper-V server
> >>>> running the controller and three CentOS instances(10.0.0.x) on a
> >>>> private subnet. I created a router in OpenStack with SNAT
> >>>> disabled, as I only want it to route traffic between the private
> >>>> subnet(10.0.0.x) and the external subnet(172.16.1.x)/internet.
> >>>> All of the instances can ping each other along with the external
> >>>> network(172.16.1.x). From the external network, I can ping the
> >>>> interface of the ovs router on the external network. I can not
> >>>> though ping inside the private network. A trace route stops at
> >>>> the IP of the OVS router. With wireshark, I do not see anything
> >>>> coming from the external pc’s IP. If I trace route it, I see
> >>>> packets making all the way to the OVS router and then stop. Since
> >>>> I can ping one way, and not the other; I believe there is
> >>>> something in the router/OVS that is stopping the packets to route
> >>>> into the private subnet. What do I need to look at? (I have
> >>>> disabled all firewalls on all OSes involved.)
> >>>
> >>> An OpenStack mailing list might be a better place to track this down.
> >>> OpenStack programs and configures OVS; OVS by itself doesn't have
> >>> much to do with what's going on.
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
