On Thu, Feb 13, 2020 at 03:07:33PM +0100, Ilya Maximets wrote:
> On 2/13/20 2:52 PM, Yi Yang (杨燚)-云服务集团 wrote:
> > Thanks Ilya, iperf3 udp should be single direction, source IP address and
> > destination IP address are two VMs' IP, udp bandwidth will be 0 if they are
> > wrong, but obviously UDP loss rate is 0, so it isn't the case you're
> > saying, do we have way to disable MAC learning or MAC broadcast?
>
> NORMAL action acts like an L2 learning switch. If you don't
> want to use MAC learning, remove flow with NORMAL action and
> add direct forwarding flow like output:. But
> I don't think that you want to do that in OpenStack setup.
Also iperf3 establishes the control connection which uses TCP in
both directions. So, in theory, the FDB should be updated.
> > Is NORMAL action or MAC learning slow path process? If so, ovs-vswitchd
> > daemon should have high cpu utilization.
>
> It's not a slow path, so there will be no cpu usage by ovs-vswitchd
> userspace process. To confirm that you're flooding packets, you
> may dump installed datapath flows with the following command:
>
> ovs-appctl dpctl/dump-flows
>
> In case of flood, you will see datapath flow with big number of
> output ports like this:
>
> <...> actions:,,...
I'd suggest to look at the fdb: ovs-appctl fdb/show
and port stats to see if there is traffic moving as well.
Maybe it's not your UDP test packet, but another unrelated
traffic in the network.
HTH,
fbl
>
> >
> > -邮件原件-
> > 发件人: Ilya Maximets [mailto:i.maxim...@ovn.org]
> > 发送时间: 2020年2月13日 21:23
> > 收件人: Flavio Leitner ; Yi Yang (杨燚)-云服务集团
> >
> > 抄送: ovs-discuss@openvswitch.org; ovs-...@openvswitch.org; Ilya Maximets
> >
> > 主题: Re: [ovs-dev] OVS performance issue: why small udp packet pps
> > performance between VMs is highly related with number of ovs ports and
> > number of VMs?
> >
> > On 2/13/20 12:48 PM, Flavio Leitner wrote:
> >> On Thu, Feb 13, 2020 at 09:18:38AM +, Yi Yang (杨燚)-云服务集团 wrote:
> >>> Hi, all
> >>>
> >>> We find ovs has serious performance issue, we only launch one VM in
> >>> one compute, and do iperf small udp pps performance test between
> >>> these two VMs, we can see about 18 pps (packets per second, -l
> >>> 16), but
> >>>
> >>> 1) if we add 100 veth ports in br-int bridge, respectively, then the pps
> >>> performance will be about 5 pps.
> >>> 2) If we launch one more VM in every compute node, but don’t run any
> >>> workload, the pps performance will be about 9 pps. (note, no
> >>> above veth ports in this test)
> >>> 3) If we launch two more VMs in every compute node (totally 3 VMs
> >>> every compute nodes), but don’t run any workload , the pps
> >>> performance will be about 5 pps (note, no above veth ports in
> >>> this test)
> >>>
> >>> Anybody can help explain why it is so? Is there any known way to
> >>> optimized this? I really think ovs performance is bad (we can draw
> >>> such conclusion from our test result at least), I don’t want to
> >>> defame ovs ☺
> >>>
> >>> BTW, we used ovs kernel datapath and vhost, we can see every port has a
> >>> vhost kernel thread, it is running with 100% cpu utilization if we run
> >>> iperf in VM, bu for those idle VMs, the corresponding vhost still has
> >>> about 30% cpu utilization, I don’t understand why.
> >>>
> >>> In addition, we find udp performance is also very bad for small UDP
> >>> packet for physical NIC. But it can reach 26 pps for –l 80 which
> >>> enough covers vxlan header (8 bytes) + inner eth header (14) + ipudp
> >>> header (28) + 16 = 66, if we consider performance overhead ovs bridge
> >>> introduces, pps performance between VMs should be able to reach 20
> >>> pps at least, other VMs and ports shouldn’t have so big hurt against it
> >>> because they are idle, no any workload there.
> >>
> >> What do you have in the flow table? It sounds like the traffic is
> >> being broadcast to all ports. Check the FDB to see if OvS is learning
> >> the mac addresses.
> >>
> >> It's been a while since I don't run performance tests with kernel
> >> datapath, but it should be no different than Linux bridge with just
> >> action NORMAL in the flow table.
> >>
> >
> > I agree that if your performance heavily depends on the number of ports
> > than you're most likely just flooding all the packets to all the ports.
> > Since you're using UDP traffic, please, be sure that you're sending some
> > packets in backward direction, so OVS and all other switches (if any) will
> > learn/not forget to which port packets should be sent. Also, check if your
> > IP addresses are correct. If for some reason it's not possible for OVS to
> > learn MAC addresses correctly, avoid using action:NORMAL.
> >
> > Best regards, Ilya Maximets.
> >
>
--
fbl
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss