Re: [ovs-discuss] OVN scalability?

[Wei Li]

Hi Han,


On 04/19/2018 06:07 AM, Han Zhou wrote:
> On Wed, Apr 18, 2018 at 1:41 PM, Mike Spreitzer  wrote:
>> How big a system can OVN manage?  I am told some local experimenters did
>> some tests about two years ago, and ran into CPU saturation at the south
>> DB at around a hundred hypervisors.
>>
>> Thanks,
>> Mike
>>
> In our testing env (setup with https://github.com/openvswitch/ovn-scale-test),
> with 1k hypervisors the central node CPU (including NB DB, SB DB and
> ovn-northd) is still low.
We observed a high cpu utilization in our environment(about 15 HVs) when doing
a high rate of port insertion/deletion. It seems ovn-northd recompute 
everything on
NB change, do you have any optimization suggestion on this scenario?

> CPU is rather busy on HVs, but there is an ongoing optimization with very
> promising test result:
> https://mail.openvswitch.org/pipermail/ovs-dev/2018-April/346136.html
Is there will be a similar incremental processing for ovn-northd in your 
roadmap?

Thanks,
Wei

>
> Thanks,
> Han
>
>
>
> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] ovs-dpdk performance not stable

[michael me]

Hi Ian and everyone,

Thank you for clarifying, i was just trying to understand :)

my bad about the 1 queue, though, i changed it to two queues and still the
performance was poor around 60mpps.
My findings are:
1. changed the PMD to a core where no other services were running.
2. added many queues (around 64).
3. after a few tests i could see that i would loose traffic i would reset
the VM and then again i would be able to get double to throughput for about
a test or two (each test 3 min)

thank you for answering,
Michael



On Fri, Apr 20, 2018 at 12:25 PM, Mooney, Sean K 
wrote:

>
>
>
>
> *From:* Stokes, Ian
> *Sent:* Thursday, April 19, 2018 9:51 PM
> *To:* michael me <1michaelmesgu...@gmail.com>
> *Cc:* ovs-discuss@openvswitch.org; Mooney, Sean K  >
> *Subject:* RE: [ovs-discuss] ovs-dpdk performance not stable
>
>
>
> Hi Michael,
>
>
>
> “It will be split between queues based on des tip so it’s important that
> test traffic varies if you want traffic to be dispersed evenly among the
> queues at this level."
>
>
>
> “Des tip” should be destination IP (apologies, I replied before having a
> morning coffee J).
>
>
>
> By varying the traffic I mean changing the destination IP, if using the
> same IP I believe the rss hash will evaluate to the same queue  on the NIC.
>
>
>
> I’m not an expert on Openstack so I’m not too sure how to enable multi
> queue for vhost interfaces in that case.
>
>
>
> @ Sean (cc’d): Is there a specific way to enable vhost multi queue for
> open stack?
>
> *[Mooney, Sean K] yes to enable vhost multi queue in openstack you need to
> set an image metadata key to request it. that will result in 1 queue per
> vCPU of the guest.*
>
> *The key should be defiend here
> https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json
> 
> but its missing the key you want to add*
>
> *hw_vif_mutliqueue_enabled its is documented here
> https://docs.openstack.org/neutron/pike/admin/config-ovs-dpdk.html
> . I
> should probably open a bug to add it to the glance*
>
> *default metadata refs.*
>
>
>
> I haven’t run MQ with a single PMD, so I’m not sure why you have better
> performance. Leave this with me to investigate further. I suspect as you
> have multiple queues more traffic is enqueued at the NIC leval
>
> *[Mooney, Sean K] for kernel virtio-net in the guest I belive there is a
> performance improvement due to reduction in intenal contentiuon from locks
> in the guset kernel networking stack but with dpdk in the guest I think the*
>
> *Perfromace would normally be the same however if the bottleneck you are
> hitting is on vswitch tx to the guest then perhaps that will also benefit
> form multiqueu howver unless you guest has more queues/cores then host *
>
> *pmds you would still have to use spin locks in the vhost pmd as you
> clould not setup a 1:1 pmd mapping to allow lockless enqueue in to the
> guest.*
>
>
>
> The problem with only 1 queue for the VM is that is creates a bottleneck
> in terms of transmitting traffic from the host to the VM (in your case 8
> queues trying to enqueue to 1 queue).
>
>
>
> How are you isolating core 0? Are you using isolcpus? Normally I would
> suggest isolating core 2 (i.e. the pmd core) with isolcpu.
>
>
>
> When you say you set txq =1 , why is that?
>
>
>
> Typically txq is set automatically, it will be number of PMDs +1 (in your
> case 2 txqs in total). The +1 is to account for traffic from kernel space.
>
>
>
> Thanks
>
> Ian
>
>
>
> *From:* michael me [mailto:1michaelmesgu...@gmail.com
> <1michaelmesgu...@gmail.com>]
> *Sent:* Thursday, April 19, 2018 7:12 PM
> *To:* Stokes, Ian 
> *Cc:* ovs-discuss@openvswitch.org
> *Subject:* Re: [ovs-discuss] ovs-dpdk performance not stable
>
>
>
> Hi Ian,
>
>
>
> Thank you for you answers!
>
>
>
> it is correct that i am using ovs-vsctl set Interface dpdk0
> options:n_rxq=8 commands for the queues.
>
> Could you please expand on the sentence "  It will be split between
> queues based on des tip so it’s important that test traffic varies if you
> want traffic to be dispersed evenly among the queues at this level."
>
> It might be a typo, or i might just not know what you mean by "des tip",
> could you please clarify for me?
>
> Additionally, what do you mean by varying the traffic? do you mean to
> somehow not have the packets at a constant frame rate?
>
>
>
> Regarding the Vhost user queues, i am using Openstack and i did not find
> yet a way to create multiple queues (i updated the image's
> metadata hw_vif_multiqueue_enabled=true) but i don't know how to set the
> queue amount especially that in the VM that i am running i do not have
> ethtool.
>
>
>
> Regarding the multiple queues while using one core for the PMD:
>
> i did get much better performance when i had two cores for the PMD,
> however, i am not at the luxury to be able 

[ovs-discuss] Questions about modified ofproto.c

[林志仁]

 I modified ofproto.c   "handle_flow_stats_request"   function.  I added a
while loop to check the flow table every 2 seconds.

I want to ask "handle_openflow_"  can handle other request, like packet -
in?
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN scalability?

[Han Zhou]

On Mon, Apr 23, 2018 at 12:21 AM, Wei Li  wrote:
>
> Hi Han,
>
>
> On 04/19/2018 06:07 AM, Han Zhou wrote:
> > On Wed, Apr 18, 2018 at 1:41 PM, Mike Spreitzer 
wrote:
> >> How big a system can OVN manage?  I am told some local experimenters
did
> >> some tests about two years ago, and ran into CPU saturation at the
south
> >> DB at around a hundred hypervisors.
> >>
> >> Thanks,
> >> Mike
> >>
> > In our testing env (setup with
https://github.com/openvswitch/ovn-scale-test),
> > with 1k hypervisors the central node CPU (including NB DB, SB DB and
> > ovn-northd) is still low.
> We observed a high cpu utilization in our environment(about 15 HVs) when
doing
> a high rate of port insertion/deletion. It seems ovn-northd recompute
everything on
> NB change, do you have any optimization suggestion on this scenario?
>
What's the rate of port insertion/deletion in your testing? Since northd
processed NB DB data only, the number of HVs actually doesn't matter. In
our testing the rate was around 1 port/sec, so it was not that hot.
It might trigger hot CPU on northd if NB DB changes very frequently, but it
doesn't mean there is real problem. northd doesn't react to each change,
but instead it computes against the latest data in NB DB. So what matters
is the size of total data in NB DB, and how long it takes for northd to
finish a round of computing for this size of data.

> > CPU is rather busy on HVs, but there is an ongoing optimization with
very
> > promising test result:
> > https://mail.openvswitch.org/pipermail/ovs-dev/2018-April/346136.html
> Is there will be a similar incremental processing for ovn-northd in your
roadmap?
>
Incremental processing for northd has been discussed in this community and
I think it is in the roadmap :) According to previous discussions, the
approach for northd incremental processing will be different from what I am
doing in ovn-controller though. Unlike ovn-controller, the input of northd
is purely NB DB, so the proposal was to rewrite northd with some domain
specific language (a datalog approach) so that the incremental processing
happens in a more clean way. But I am not sure about the progress of this
work. Ben may have more information.

> Thanks,
> Wei
>
> >
> > Thanks,
> > Han
> >
> >
> >
> > ___
> > discuss mailing list
> > disc...@openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] [ovs-dev] Delete Flows matching specific cookie - OpenFlow 1.3.5 Spec support by OpenVSwitch

[Darrell Ball]

//You can verify that OVS supports deletion by cookie specification by checking 
these tests:

 932: ofproto.at:1746ofproto - del flows based on cookie
 933: ofproto.at:1767ofproto - del flows based on cookie mask

// Run the tests using “make check” like this:
sudo make check TESTSUITEFLAGS='932-933' -C _gcc

//Look at what is being tested here:
tests/ofproto.at

Thanks Darrell


On 4/23/18, 4:17 PM, "ovs-dev-boun...@openvswitch.org on behalf of Carlos 
Ferreira"  
wrote:

Hello to all

According to the OpenFlow 1.3.5 spec, page 44 specifies the following:


"Modify and delete commands can also be filtered by cookie value, if
the cookie_mask field contains
a value other than 0. This constraint is that the bits specified by
the cookie_mask in both
the cookie field of the flow mod and a flow entry’s cookie value must
be equal. In other words,
(flow entry.cookie&flow mod.cookie mask) == (flow mod.cookie&flow
mod.cookie mask)."


Now, using the Ryu python-based controller, I tried to delete a flow
by specifying the flow's cookie value, but the procedure was not
successful.

The following code is a test-example which I used.

from ryu.base.app_manager import RyuApp
from ryu.controller.dpset import EventDP
from ryu.controller.handler import MAIN_DISPATCHER
from ryu.controller.handler import set_ev_cls
from ryu.ofproto import ofproto_v1_3
from ryu.ofproto import ether, inet

class MPLS_Testing(RyuApp):
  OFP_VERSIONS = [ofproto_v1_3.OFP_VERSION]

  @set_ev_cls(EventDP, MAIN_DISPATCHER)
  def switch_connect_event(self, ev):
ofp_parser = ev.dp.ofproto_parser
ofp = ev.dp.ofproto
datapath_obj = ev.dp
if ev.enter:
  datapath_obj.send_msg(  # Removes all flows registered in this 
switch.
ofp_parser.OFPFlowMod(
  datapath=datapath_obj,
  table_id=ofp.OFPTT_ALL,
  command=ofp.OFPFC_DELETE,
  out_port=ofp.OFPP_ANY,
  out_group=ofp.OFPG_ANY,
)
  )
  add_label_flow = ofp_parser.OFPFlowMod(
datapath=datapath_obj,
cookie=1,
table_id=0,
command=ofp.OFPFC_ADD,
match=ofp_parser.OFPMatch(
  in_port=1
),
instructions=[
  ofp_parser.OFPInstructionActions(
ofp.OFPIT_APPLY_ACTIONS,
[
  ofp_parser.OFPActionPushMpls(),
  ofp_parser.OFPActionSetField(mpls_label=16),
]
  ),
  ofp_parser.OFPInstructionGotoTable(table_id=1),
]
  )
  datapath_obj.send_msg(add_label_flow)

  add_label_flow2 = ofp_parser.OFPFlowMod(
datapath=datapath_obj,
cookie=2,
table_id=1,
command=ofp.OFPFC_ADD,
match=ofp_parser.OFPMatch(
  in_port=1
),
instructions=[
  ofp_parser.OFPInstructionActions(
ofp.OFPIT_APPLY_ACTIONS,
[
  ofp_parser.OFPActionPushMpls(),
  ofp_parser.OFPActionSetField(mpls_label=12),
]
  ),
  ofp_parser.OFPInstructionGotoTable(table_id=2),
]
  )
  datapath_obj.send_msg(add_label_flow2)

  # Deletes flow with cookie equal to 2.
  datapath_obj.send_msg(
ofp_parser.OFPFlowMod(
  cookie=2,
  cookie_mask=0x,
  datapath=datapath_obj,
  command=ofp.OFPFC_DELETE,
  out_port=ofp.OFPP_ANY,
  out_group=ofp.OFPG_ANY,
)
  )

Can anyone tell me if OpenVSwitch 2.9 supports cookie match when
deleting a flow from the tables? OpenFlow 1.3.5 spec clearly states
that a Delete command could also filter flows using the cookie value,
when the cookie_mask is different than zero. Currently, I'm kinda lost
here.

Thank you!

-- 

Carlos Miguel Ferreira
Researcher at Telecommunications Institute
Aveiro - Portugal
Work E-mail - c...@av.it.pt
Skype & GTalk -> carlosmf...@gmail.com
LinkedIn -> 
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_in_carlosmferreira&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=ND4lqlTir-Ukyp2WkESK_WJ7j0b1dAUdptpdObaFU7Y&s=saz4uZw1U4C3pxnJSK9cRxTNXucuBUokJFvgmL3P3Wg&e=
___
dev mailing list
d...@openvswitch

[ovs-discuss] interface mac setting

[Tonghao Zhang]

One question, why now ovs can only set the mac of internal interface,
for example:

ovs-vsctl set Interface p1 type=internal mac=\"00:11:22:33:44:55\"
ovs-vsctl get Interface p1 mac_in_use

And the doc about 'mac' is not explained, why we cannot set it for
other type interface. So it's a bug?

diff --git a/vswitchd/bridge.c b/vswitchd/bridge.c
index d90997e..28dd7f8 100644
--- a/vswitchd/bridge.c
+++ b/vswitchd/bridge.c
@@ -4416,10 +4416,6 @@ iface_set_mac(const struct bridge *br, const
struct port *port, struct iface *if
 struct eth_addr ea, *mac = NULL;
 struct iface *hw_addr_iface;

-if (strcmp(iface->type, "internal")) {
-return;
-}
-
 if (iface->cfg->mac && eth_addr_from_string(iface->cfg->mac, &ea)) {
 mac = &ea;
 } else if (port->cfg->fake_bridge) {
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] [ovs-dev] Delete Flows matching specific cookie - OpenFlow 1.3.5 Spec support by OpenVSwitch

[Darrell Ball]

On 4/23/18, 6:53 PM, "Carlos Ferreira"  wrote:

@IWAMOTO

Hello IWAMOTO,
No, I do not get any kind of error from Ryu. I have also verified the
structure of the Openflow messages sent to the switch using Wireshark
and no anomalies were found. The packet was well structured.

@Darrell,

Darrel, thank you for the info. I tried to use the del-flows using the
"ovs-ofctl del-flows" command with the specific cookie and in fact, it
did work. I'm lost here, because it does not seems to work if the
communication is done via OpenFlow with a remote controller.

Maybe you want to check what Ryu is sending the switch

http://www.openvswitch.org//support/dist-docs/ovs-ofctl.8.pdf

“
OpenFlow Switch Monitoring Commands

monitor switch [miss-len] [invalid_ttl] [watch:[spec...]]
Connects to switch and prints to the console all OpenFlow messages received. 
Usually, switch
should specify the name of a bridge in the ovs−vswitchd database.
“

eg) ovs-ofctl monitor br0 65534 invalid_ttl --detach --no-chdir --pidfile 2> 
ofctl_monitor.log




On 24 April 2018 at 02:42, Darrell Ball  wrote:
> //You can verify that OVS supports deletion by cookie specification by 
checking these tests:
>
>  932: ofproto.at:1746ofproto - del flows based on cookie
>  933: ofproto.at:1767ofproto - del flows based on cookie mask
>
> // Run the tests using “make check” like this:
> sudo make check TESTSUITEFLAGS='932-933' -C _gcc
>
> //Look at what is being tested here:
> tests/ofproto.at
>
> Thanks Darrell
>
>
> On 4/23/18, 4:17 PM, "ovs-dev-boun...@openvswitch.org on behalf of Carlos 
Ferreira"  
wrote:
>
> Hello to all
>
> According to the OpenFlow 1.3.5 spec, page 44 specifies the following:
>
>
> "Modify and delete commands can also be filtered by cookie value, if
> the cookie_mask field contains
> a value other than 0. This constraint is that the bits specified by
> the cookie_mask in both
> the cookie field of the flow mod and a flow entry’s cookie value must
> be equal. In other words,
> (flow entry.cookie&flow mod.cookie mask) == (flow mod.cookie&flow
> mod.cookie mask)."
>
>
> Now, using the Ryu python-based controller, I tried to delete a flow
> by specifying the flow's cookie value, but the procedure was not
> successful.
>
> The following code is a test-example which I used.
>
> from ryu.base.app_manager import RyuApp
> from ryu.controller.dpset import EventDP
> from ryu.controller.handler import MAIN_DISPATCHER
> from ryu.controller.handler import set_ev_cls
> from ryu.ofproto import ofproto_v1_3
> from ryu.ofproto import ether, inet
>
> class MPLS_Testing(RyuApp):
>   OFP_VERSIONS = [ofproto_v1_3.OFP_VERSION]
>
>   @set_ev_cls(EventDP, MAIN_DISPATCHER)
>   def switch_connect_event(self, ev):
> ofp_parser = ev.dp.ofproto_parser
> ofp = ev.dp.ofproto
> datapath_obj = ev.dp
> if ev.enter:
>   datapath_obj.send_msg(  # Removes all flows registered in 
this switch.
> ofp_parser.OFPFlowMod(
>   datapath=datapath_obj,
>   table_id=ofp.OFPTT_ALL,
>   command=ofp.OFPFC_DELETE,
>   out_port=ofp.OFPP_ANY,
>   out_group=ofp.OFPG_ANY,
> )
>   )
>   add_label_flow = ofp_parser.OFPFlowMod(
> datapath=datapath_obj,
> cookie=1,
> table_id=0,
> command=ofp.OFPFC_ADD,
> match=ofp_parser.OFPMatch(
>   in_port=1
> ),
> instructions=[
>   ofp_parser.OFPInstructionActions(
> ofp.OFPIT_APPLY_ACTIONS,
> [
>   ofp_parser.OFPActionPushMpls(),
>   ofp_parser.OFPActionSetField(mpls_label=16),
> ]
>   ),
>   ofp_parser.OFPInstructionGotoTable(table_id=1),
> ]
>   )
>   datapath_obj.send_msg(add_label_flow)
>
>   add_label_flow2 = ofp_parser.OFPFlowMod(
> datapath=datapath_obj,
> cookie=2,
> table_id=1,
> command=ofp.OFPFC_ADD,
> match=ofp_parser.OFPMatch(
>   in_port=1
> ),
> instructions=[
>   ofp_parser.OFPInstructionActions(
> 

[ovs-discuss] Query regarding packet_in reason sent from OvS for “Table-miss” rule.

[Rohith Basavaraja]

Hi,

Currently in OvS if we hit "Table-miss" rules (associated with Controller 
action) then we send
PACKET_IN message to controller with reason as OFPR_NO_MATCH.

“Table-miss” rule is one whose priority is 0 and its catch all rule.

But if we hit same "Table-miss" rule after executing group entry we will send 
the reason as
OFPR_ACTION (for OF1.3 and below) and OFPR_GROUP (for OF1.4 and above).

This is because once we execute group entry we set ctx->in_group and later when 
we hit the
"Table-miss" rule, Since ctx->in_group  is set we send reason as OFPR_ACTION 
(for OF1.3) and
OFPR_GROUP (for OF1.4 and above).

Since in the packet_in message we are sending the table-id of the corresponding 
“Table-miss” rule
Shouldn’t we send the reason for packet-in as OFPR_NO_MATCH ( the reason 
appropriate to the
Corresponding rule?)

For eg: for the following pipeline we will send the reason as OFPR_ACTION even 
if we hit
The “Table-miss” rule.

cookie=0x800, duration=761.189s, table=0, n_packets=1401, n_bytes=67954, 
priority=4,in_port=9,vlan_tci=0x/0x1fff 
actions=write_metadata:0x678700/0xff01,goto_table:17

cookie=0x681, duration=768.848s, table=17, n_packets=1418, n_bytes=68776, 
priority=10,metadata=0x678700/0xff00 
actions=write_metadata:0xe0678700/0xfffe,goto_table:60

cookie=0x680, duration=24944.312s, table=60, n_packets=58244, 
n_bytes=2519520, priority=0 actions=resubmit(,17)
cookie=0x804, duration=785.733s, table=17, n_packets=1450, n_bytes=69724, 
priority=10,metadata=0xe0678700/0xff00 
actions=write_metadata:0x67871d4d00/0xfffe,goto_table:43

cookie=0x822002d, duration=24960.795s, table=43, n_packets=53097, 
n_bytes=2230074, priority=100,arp,arp_op=1 actions=group:6000
group_id=6000,type=all,bucket=actions=CONTROLLER:65535,bucket=actions=resubmit(,48),bucket=actions=resubmit(,81)

cookie=0x850, duration=24977.323s, table=48, n_packets=58309, 
n_bytes=2522634, priority=0 actions=resubmit(,49),resubmit(,50)

cookie=0x805, duration=24984.679s, table=50, n_packets=6, n_bytes=264, 
priority=0 
actions=CONTROLLER:65535,learn(table=49,hard_timeout=10,priority=0,cookie=0x860,NXM_OF_ETH_SRC[],load:0x1->NXM_NX_REG4[0..7]),goto_table:51
(Note that Table50 entry is a “Table-miss” rule).

Currently we are sending table_id as 50 and packet_in reason as OFPR_ACTION.
Shouldn’t we send packet_in reason as OFPR_NO_MATCH in this case?


Thanks
Rohith



___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss