Re: [ovs-discuss] OVN scale

2020-07-28 Thread Han Zhou 
On Mon, Jul 27, 2020 at 10:16 AM Tony Liu  wrote:

> Hi Han,
>
> Just some updates here.
>
> I tried with 4K networks on single router. Configuration was done without
> any issues. I checked both
> nb-db and sb-db, they all look good. It's just that router configuration
> is huge (in Neutron DB, nb-db
> and flow table in sb-db), because it contains all 4K ports. Also, the
> pipeline of router datapath in sb-db
> is quite big.
>
> I see ovn-northd master and sb-db leader are busy, taking 90+% CPU. There
> are only 3 compute nodes
> and 2 gateway nodes. Does that monitor setting "ovn-monitor-all" matters
> in such case? Any idea what
> they are busy with, without any configuration updates from OpenStack? The
> nb-db is not busy though.
>

Did you create logical switch ports in your test? Did you do port-binding
on compute nodes? If yes, then "ovn-monitor-all" would matter, since all
networks are connected to the same router. With "ovn-monitor-all" = true,
it would avoid the huge monitor condition change messages.

Normally, if there is no NB-DB change, all components should be idle.


> Probably because nb-db is busy, ovn-controller can't connect to it
> consistently. It keeps being
> disconnected and reconnecting. Restarting ovn-controller seems help. I am
> able to launch a few VMs
> on different networks and they are connected via the router.
>
> If you are seeing ovn-controller disconnected to sb-db due to probe
timeout, you can disable/adjust the probe interval. See this slide:
https://www.slideshare.net/hanzhou1978/large-scale-overlay-networks-with-ovn-problems-and-solutions/16



> Now, I have problem on external access. The router is set as gateway to a
> provider/underlay network
> on an interface on the gateway node. The router is allocated an underlay
> address from that provider
> network. My understanding is that, the br-ex on gateway node holding the
> active router will broadcast
> ARP to announce that router underlay address in case of failover. Also, it
> will respond ARP request for
> that router underlay address. But when I run tcpdump on that underlay
> interface on gateway node,
> I see ARP request coming in, but no ARP response going out. I checked the
> flow table in sb-db, it seems
> ok. I also checked flow on br-ex by "ovs-ofctl dump-flows br-ex", I don't
> see anything about ARP there.
> How should I look into it?
>

"br-ex" is not managed by OVN, so you won't see any flows there. Did you
use OpenStack commands to setup the gateway? Did you see port-binding of
the gateway port in SB DB?


> Again, the case is to support 4K networks with external access (security
> group is disabled),
> 4K routers (one for each network), 50 routers (one for 80 networks), 1
> router (for all 4K networks)...
> All networks are isolated by ACL on the logical router. Which option
> should work better?
> Any comment is appreciated.
>
>
If the 4K networks don't need to communicate with each other, then what
would scale the best (in theory) is: 4K routers (one for each network) with
ovn-monitor-all=false. This way, each HV only need to process a small
proportion of the data. (the monitor condition change message should also
be small because each HV only monitor the networks that have related VMs on
the HV).

Thanks,
Han


> Thanks!
>
> Tony
>
>
> --
> *From:* discuss  on behalf of Tony
> Liu 
> *Sent:* July 21, 2020 09:09 PM
> *To:* Daniel Alvarez 
> *Cc:* ovs-discuss@openvswitch.org 
> *Subject:* Re: [ovs-discuss] OVN scale
>
> [root@ovn-db-2 ~]# ovn-nbctl list nb_global
> _uuid   : b7b3aa05-f7ed-4dbc-979f-10445ac325b8
> connections : []
> external_ids: {"neutron:liveness_check_at"="2020-07-22
> 04:03:17.726917+00:00"}
> hv_cfg  : 312
> ipsec   : false
> name: ""
> nb_cfg  : 2636
> options : {mac_prefix="ca:e8:07",
> svc_monitor_mac="4e:d0:3a:80:d4:b7"}
> sb_cfg  : 2005
> ssl : []
>
> [root@ovn-db-2 ~]# ovn-sbctl list sb_global
> _uuid   : 3720bc1d-b0da-47ce-85ca-96fa8d398489
> connections : []
> external_ids: {}
> ipsec   : false
> nb_cfg  : 312
> options : {mac_prefix="ca:e8:07",
> svc_monitor_mac="4e:d0:3a:80:d4:b7"}
> ssl : []
>
> The NBDB and SBDB is definitely out of sync. Is there any way to force
> ovn-northd sync them?
>
> Thanks!
>
> Tony
>
> --
> *From:* Tony Liu 
> *Sent:* July 21, 2020 08:39 PM
> *To:* Daniel Alvarez 
> *Cc:* Cory Hawkless ; ovs-discuss@openvswitch.org <
> ovs-discuss@openvswitch.org>; Dumitru Ceara 
> *Subject:* Re: [ovs-discuss] OVN scale
>
> When create a network (and subnet) on OpenStack, a GW port and service
> port (for DHCP and metadata)
> are also created. They are created in Neutron and onv-nb-db by ML2 driver.
> Then ovn-northd will translate
> such update from NBDB to SBDB. My question here is that, with 20.03, is

[ovs-discuss] Display OpenFlow port number, interface name and bridge name in single OvS CLI command

2020-07-28 Thread Matteo Olivi 
Hello everyone,
I have an OvS bridge *X *and some network interfaces connected to it via
OpenFlow ports.
For each interface connected to *X*, I want to display the name and the
number of its OpenFlow port.
I've been using the following command:
"ovs-vsctl -f table -- --columns=ofport,name list Interface"

The problem with the command above is that it lists the interface name and
OpenFlow port
number for the interfaces connected to all the bridges on the host, while I
only want the interfaces
connected to bridge *X*. Is there a single command to obtain what I need?
Or, is there a single
database table where rows store the information that I need, i.e. the
triplet (bridge name, interface name, OpenFlow port number) ?

Thanks,
Matteo.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] Display OpenFlow port number, interface name and bridge name in single OvS CLI command

2020-07-28 Thread Tony Liu 

for p in $(ovs-vsctl list-ports br-int); do \
ovs-vsctl -f table --columns=ofport,name list interface $p; \
done

Tony

From: discuss  on behalf of Matteo Olivi 

Sent: July 28, 2020 11:14 AM
To: ovs-discuss@openvswitch.org 
Subject: [ovs-discuss] Display OpenFlow port number, interface name and bridge 
name in single OvS CLI command

Hello everyone,
I have an OvS bridge X and some network interfaces connected to it via OpenFlow 
ports.
For each interface connected to X, I want to display the name and the number of 
its OpenFlow port.
I've been using the following command:
"ovs-vsctl -f table -- --columns=ofport,name list Interface"

The problem with the command above is that it lists the interface name and 
OpenFlow port
number for the interfaces connected to all the bridges on the host, while I 
only want the interfaces
connected to bridge X. Is there a single command to obtain what I need? Or, is 
there a single
database table where rows store the information that I need, i.e. the triplet 
(bridge name, interface name, OpenFlow port number) ?

Thanks,
Matteo.
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] OVN: resync nb-db to sb-db

2020-07-28 Thread Tony Liu 
Hi,

When I run a script to create bunch of networks and routers from OpenStack, for 
whatever reason,
nb-db is fully updated, but sb-db is only partially updated. For example, there 
are 500 logical routers
in nb-db, but only 218 datapath bindings in sb-db. In this case, is there any 
way to resync nb-db to
sb-db? Any advices to avoid such failure?


Thanks!

Tony

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] max_speed and curr_speed is incorrect

2020-07-28 Thread Ryan Chen 
Hi, 

I want to collect ports’ curr_speed and max_speed from ovs to calculate ports’ 
bandwidth on Mininet. However, no matter how I set the bandwidth, the value of 
max_speed is always 0 and value of curr_speed is always 1000. 

I’m using ovs 2.13.0 in Mininet 2.3.0b6. Could anyone tell me how can I get the 
correct speed from my ovs switches?


Thanks

___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] [ovs-dev] OVN nb-db and sb-db election timer

2020-07-28 Thread Han Zhou 
On Mon, Jul 27, 2020 at 1:40 PM Tony Liu  wrote:
>
> Hi,
>
> During scaling test, when sb-db is busy, followers believe the leader is
dead and started election
> request. Some inconsistency happens during such leader switch. Two
datapath bindings are created
> for the same logical switch. To avoid such case, I was recommended to
increase election timer x10.
> 4K networks are created successfully with that setting.
>
> Is it necessary to set big election timer for nb-db as well? The nb-db
doesn't seem very busy during
> the test, sb-db is always busy and taking 90+% CPU.
>
NB-DB is usually not busy, but you can still adjust the timer according to
your use case.

> With that big election timer, in case real problem happens, like the
leader node goes down, is it going
> to take a while for the new leader to be elected?
>

Yes, it will take longer to detect real failures when election timer is
bigger. During the gap, control plane operations may be impacted/delayed,
but dataplane is not impacted - what's running will continue working, but
changes during this time will not be reflected until the failover is done.

>
> Thanks!
>
> Tony
>
> ___
> dev mailing list
> d...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] [ovs-dev] OVN: resync nb-db to sb-db

2020-07-28 Thread Han Zhou 
On Tue, Jul 28, 2020 at 2:09 PM Tony Liu  wrote:
>
> Hi,
>
> When I run a script to create bunch of networks and routers from
OpenStack, for whatever reason,
> nb-db is fully updated, but sb-db is only partially updated. For example,
there are 500 logical routers
> in nb-db, but only 218 datapath bindings in sb-db. In this case, is there
any way to resync nb-db to
> sb-db? Any advices to avoid such failure?
>
>
> Thanks!
>
> Tony
>
This looks like a bug. Please share the version, and logs of ovn-northd.
You can also enable debug logs with: ovn-appctl -t ovn-northd vlog/set
file:dbg, if it can be reproduced.

Normally, SB and NB will be always in sync if ovn-northd is running.

Thanks,
Han
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

[ovs-discuss] OVN: configuration in Neutron DB?

2020-07-28 Thread Tony Liu 
Hi,

In case of integration with OpenStack, for example, when a client requests to 
create a network,
is this network configuration saved in both Neutron DB and OVN DB, or OVN DB 
only?
Also, when a client gets a network from Neutron API, is the configuration read 
from Neutron DB
or OVN DB?

Other than coding, is there any doc about how Neutron OVN ML2 driver works?

I have this script to create 256 routers and set each of them as gateway.
router()
{
local op=$1

for c in `seq 0 1 255`; do
echo "INFO: $op router-$c..."
openstack router $op router-$c
if [ "$op" == "create" ]; then
openstack router set \
--external-gateway public \
--fixed-ip ip-address=10.6.33.$c \
--disable-snat \
router-$c
fi
done
}
I see lots failures from Neutron log when get/show a router. It seems like 
that, when setting a router,
the router is not completely ready yet. Is it possible?

After running that script, I see some logical routers in ovn-nb-db don't have 
gw_port_id. And there
are some duplications. Here is an example. Each of them has unique UUID.

external_ids: {"neutron:gw_port_id"="", "neutron:revision_number"="1", 
"neutron:router_name"=router-255}
external_ids: {"neutron:gw_port_id"="", "neutron:revision_number"="1", 
"neutron:router_name"=router-232}
external_ids: {"neutron:gw_port_id"="", "neutron:revision_number"="0", 
"neutron:router_name"=router-158}
external_ids: {"neutron:gw_port_id"="", "neutron:revision_number"="0", 
"neutron:router_name"=router-158}
external_ids: 
{"neutron:gw_port_id"="e52dda53-c914-4ea7-840b-8632a5770680", 
"neutron:revision_number"="2", "neutron:router_name"=router-158}

I enabled nb-db debug logging and searched, eg. router-158, it only shows in a 
jsonrpc reply message
including 3 router-158, as the above.

Any clues?


Thanks!

Tony



___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN: configuration in Neutron DB?

2020-07-28 Thread Tony Liu 
Quick update. I changed the script to create 256 routers first, then set each 
of them as gateway.
There is no create and set back to back. It seems working fine now.

It would be good someone can clarify my questions. It seems that it's not 
guaranteed that the
object is ready when client get OK response of creation request. Is this 
expected?


Thanks!

Tony


From: dev  on behalf of Tony Liu 

Sent: July 28, 2020 10:37 PM
To: ovs-discuss@openvswitch.org ; 
ovs-...@openvswitch.org 
Subject: [ovs-dev] OVN: configuration in Neutron DB?

Hi,

In case of integration with OpenStack, for example, when a client requests to 
create a network,
is this network configuration saved in both Neutron DB and OVN DB, or OVN DB 
only?
Also, when a client gets a network from Neutron API, is the configuration read 
from Neutron DB
or OVN DB?

Other than coding, is there any doc about how Neutron OVN ML2 driver works?

I have this script to create 256 routers and set each of them as gateway.
router()
{
local op=$1

for c in `seq 0 1 255`; do
echo "INFO: $op router-$c..."
openstack router $op router-$c
if [ "$op" == "create" ]; then
openstack router set \
--external-gateway public \
--fixed-ip ip-address=10.6.33.$c \
--disable-snat \
router-$c
fi
done
}
I see lots failures from Neutron log when get/show a router. It seems like 
that, when setting a router,
the router is not completely ready yet. Is it possible?

After running that script, I see some logical routers in ovn-nb-db don't have 
gw_port_id. And there
are some duplications. Here is an example. Each of them has unique UUID.

external_ids: {"neutron:gw_port_id"="", "neutron:revision_number"="1", 
"neutron:router_name"=router-255}
external_ids: {"neutron:gw_port_id"="", "neutron:revision_number"="1", 
"neutron:router_name"=router-232}
external_ids: {"neutron:gw_port_id"="", "neutron:revision_number"="0", 
"neutron:router_name"=router-158}
external_ids: {"neutron:gw_port_id"="", "neutron:revision_number"="0", 
"neutron:router_name"=router-158}
external_ids: 
{"neutron:gw_port_id"="e52dda53-c914-4ea7-840b-8632a5770680", 
"neutron:revision_number"="2", "neutron:router_name"=router-158}

I enabled nb-db debug logging and searched, eg. router-158, it only shows in a 
jsonrpc reply message
including 3 router-158, as the above.

Any clues?


Thanks!

Tony



___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN: configuration in Neutron DB?

2020-07-28 Thread Numan Siddique 
Adding Daniel and Lucas. Maybe you can also include opendev ML to get
appropriate responses from the OpenStack side.

Please see below for few comments.


On Wed, Jul 29, 2020 at 12:02 PM Tony Liu  wrote:

> Quick update. I changed the script to create 256 routers first, then set
> each of them as gateway.
> There is no create and set back to back. It seems working fine now.
>
> It would be good someone can clarify my questions. It seems that it's
> not guaranteed that the
> object is ready when client get OK response of creation request. Is this
> expected?
>
>
> Thanks!
>
> Tony
>
> --
> *From:* dev  on behalf of Tony Liu <
> tonyliu0...@hotmail.com>
> *Sent:* July 28, 2020 10:37 PM
> *To:* ovs-discuss@openvswitch.org ;
> ovs-...@openvswitch.org 
> *Subject:* [ovs-dev] OVN: configuration in Neutron DB?
>
> Hi,
>
> In case of integration with OpenStack, for example, when a client requests
> to create a network,
> is this network configuration saved in both Neutron DB and OVN DB, or OVN
> DB only?
>

The neutron API first saves in the neutron db and the neutron OVN mechanism
driver will talk
to the Northbound ovsdb-server and create corresponding OVN logical
resources.


> Also, when a client gets a network from Neutron API, is the configuration
> read from Neutron DB
> or OVN DB?
>

I think its read from the neutron DB.



>
> Other than coding, is there any doc about how Neutron OVN ML2 driver works?
>

You can refer here -
https://docs.openstack.org/neutron/latest/admin/ovn/refarch/refarch.html



>
> I have this script to create 256 routers and set each of them as gateway.
> router()
> {
> local op=$1
>
> for c in `seq 0 1 255`; do
> echo "INFO: $op router-$c..."
> openstack router $op router-$c
> if [ "$op" == "create" ]; then
> openstack router set \
> --external-gateway public \
> --fixed-ip ip-address=10.6.33.$c \
> --disable-snat \
> router-$c
> fi
> done
> }
> I see lots failures from Neutron log when get/show a router. It seems like
> that, when setting a router,
> the router is not completely ready yet. Is it possible?
>
> After running that script, I see some logical routers in ovn-nb-db don't
> have gw_port_id. And there
> are some duplications. Here is an example. Each of them has unique UUID.
>
> external_ids: {"neutron:gw_port_id"="",
> "neutron:revision_number"="1", "neutron:router_name"=router-255}
> external_ids: {"neutron:gw_port_id"="",
> "neutron:revision_number"="1", "neutron:router_name"=router-232}
> external_ids: {"neutron:gw_port_id"="",
> "neutron:revision_number"="0", "neutron:router_name"=router-158}
> external_ids: {"neutron:gw_port_id"="",
> "neutron:revision_number"="0", "neutron:router_name"=router-158}
> external_ids:
> {"neutron:gw_port_id"="e52dda53-c914-4ea7-840b-8632a5770680",
> "neutron:revision_number"="2", "neutron:router_name"=router-158}
>
> I enabled nb-db debug logging and searched, eg. router-158, it only shows
> in a jsonrpc reply message
> including 3 router-158, as the above.
>
> Any clues?
>
>
Maybe Daniel/Lucas can comment.

Thanks
Numan


>
> Thanks!
>
> Tony
>
>
>
> ___
> dev mailing list
> d...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVN: configuration in Neutron DB?

2020-07-28 Thread Tony Liu 
Thanks Numan!

Tony

From: Numan Siddique 
Sent: July 28, 2020 11:41 PM
To: Tony Liu 
Cc: ovs-discuss@openvswitch.org ; 
ovs-...@openvswitch.org ; Daniel Alvarez Sanchez 
; Lucas Alvares Gomes Martins 
Subject: Re: [ovs-discuss] OVN: configuration in Neutron DB?


Adding Daniel and Lucas. Maybe you can also include opendev ML to get 
appropriate responses from the OpenStack side.

Please see below for few comments.


On Wed, Jul 29, 2020 at 12:02 PM Tony Liu 
mailto:tonyliu0...@hotmail.com>> wrote:
Quick update. I changed the script to create 256 routers first, then set each 
of them as gateway.
There is no create and set back to back. It seems working fine now.

It would be good someone can clarify my questions. It seems that it's not 
guaranteed that the
object is ready when client get OK response of creation request. Is this 
expected?


Thanks!

Tony


From: dev 
mailto:ovs-dev-boun...@openvswitch.org>> on 
behalf of Tony Liu mailto:tonyliu0...@hotmail.com>>
Sent: July 28, 2020 10:37 PM
To: ovs-discuss@openvswitch.org 
mailto:ovs-discuss@openvswitch.org>>; 
ovs-...@openvswitch.org 
mailto:ovs-...@openvswitch.org>>
Subject: [ovs-dev] OVN: configuration in Neutron DB?

Hi,

In case of integration with OpenStack, for example, when a client requests to 
create a network,
is this network configuration saved in both Neutron DB and OVN DB, or OVN DB 
only?

The neutron API first saves in the neutron db and the neutron OVN mechanism 
driver will talk
to the Northbound ovsdb-server and create corresponding OVN logical resources.

Also, when a client gets a network from Neutron API, is the configuration read 
from Neutron DB
or OVN DB?

I think its read from the neutron DB.



Other than coding, is there any doc about how Neutron OVN ML2 driver works?

You can refer here - 
https://docs.openstack.org/neutron/latest/admin/ovn/refarch/refarch.html



I have this script to create 256 routers and set each of them as gateway.
router()
{
local op=$1

for c in `seq 0 1 255`; do
echo "INFO: $op router-$c..."
openstack router $op router-$c
if [ "$op" == "create" ]; then
openstack router set \
--external-gateway public \
--fixed-ip ip-address=10.6.33.$c \
--disable-snat \
router-$c
fi
done
}
I see lots failures from Neutron log when get/show a router. It seems like 
that, when setting a router,
the router is not completely ready yet. Is it possible?

After running that script, I see some logical routers in ovn-nb-db don't have 
gw_port_id. And there
are some duplications. Here is an example. Each of them has unique UUID.

external_ids: {"neutron:gw_port_id"="", "neutron:revision_number"="1", 
"neutron:router_name"=router-255}
external_ids: {"neutron:gw_port_id"="", "neutron:revision_number"="1", 
"neutron:router_name"=router-232}
external_ids: {"neutron:gw_port_id"="", "neutron:revision_number"="0", 
"neutron:router_name"=router-158}
external_ids: {"neutron:gw_port_id"="", "neutron:revision_number"="0", 
"neutron:router_name"=router-158}
external_ids: 
{"neutron:gw_port_id"="e52dda53-c914-4ea7-840b-8632a5770680", 
"neutron:revision_number"="2", "neutron:router_name"=router-158}

I enabled nb-db debug logging and searched, eg. router-158, it only shows in a 
jsonrpc reply message
including 3 router-158, as the above.

Any clues?


Maybe Daniel/Lucas can comment.

Thanks
Numan


Thanks!

Tony



___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss