Thanks Han for the correction. Just one more minor typo in the tutorial below.
I hit some roadblocks to start ic controller on different az but got my setup running with workarounds in current code and have posted in comments which needs fix for sure. I tried with 2 different ovn setups with 2 AZs where each ovn az uses1 hv, 1 gw and 1 lport bound to hv. I am able to access the workloads across azs. Tested-by: Aliasgar Ginwala <aginw...@ebay.com <gvrose8...@gmail.com>> On Sun, Oct 20, 2019 at 5:55 PM Han Zhou <hz...@ovn.org> wrote: > Added tutorial, and also updated NEWS and TODO. > > Signed-off-by: Han Zhou <hz...@ovn.org> > --- > Documentation/automake.mk | 1 + > Documentation/tutorials/index.rst | 1 + > Documentation/tutorials/ovn-interconnection.rst | 181 > ++++++++++++++++++++++++ > NEWS | 5 + > TODO.rst | 10 ++ > 5 files changed, 198 insertions(+) > create mode 100644 Documentation/tutorials/ovn-interconnection.rst > > diff --git a/Documentation/automake.mk b/Documentation/automake.mk > index 5968d69..15d261d 100644 > --- a/Documentation/automake.mk > +++ b/Documentation/automake.mk > @@ -20,6 +20,7 @@ DOC_SOURCE = \ > Documentation/tutorials/ovn-sandbox.rst \ > Documentation/tutorials/ovn-ipsec.rst \ > Documentation/tutorials/ovn-rbac.rst \ > + Documentation/tutorials/ovn-interconnection.rst \ > Documentation/topics/index.rst \ > Documentation/topics/testing.rst \ > Documentation/topics/high-availability.rst \ > diff --git a/Documentation/tutorials/index.rst > b/Documentation/tutorials/index.rst > index 1cf083e..4ff6e16 100644 > --- a/Documentation/tutorials/index.rst > +++ b/Documentation/tutorials/index.rst > @@ -43,3 +43,4 @@ vSwitch. > ovn-openstack > ovn-rbac > ovn-ipsec > + ovn-interconnection > diff --git a/Documentation/tutorials/ovn-interconnection.rst > b/Documentation/tutorials/ovn-interconnection.rst > new file mode 100644 > index 0000000..1320d41 > --- /dev/null > +++ b/Documentation/tutorials/ovn-interconnection.rst > @@ -0,0 +1,181 @@ > +.. > + Licensed under the Apache License, Version 2.0 (the "License"); you > may > + not use this file except in compliance with the License. You may > obtain > + a copy of the License at > + > + http://www.apache.org/licenses/LICENSE-2.0 > + > + Unless required by applicable law or agreed to in writing, software > + distributed under the License is distributed on an "AS IS" BASIS, > WITHOUT > + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > See the > + License for the specific language governing permissions and > limitations > + under the License. > + > + Convention for heading levels in OVN documentation: > + > + ======= Heading 0 (reserved for the title in a document) > + ------- Heading 1 > + ~~~~~~~ Heading 2 > + +++++++ Heading 3 > + ''''''' Heading 4 > + > + Avoid deeper levels because they do not render well. > + > +=================== > +OVN Interconnection > +=================== > + > +This document provides a guide for interconnecting multiple OVN > deployements > +with OVN managed tunneling. More details about the OVN Interconnectiong > design > +can be found in ``ovn-architecture``\(7) manpage. > + > +This document assumes two or more OVN deployments are setup and runs > normally, > +possibly at different data-centers, and the gateway chassises of each OVN > +are with IP addresses that are reachable between each other. > + > +Setup Interconnection Databases > +------------------------------- > + > +To interconnect different OVNs, you need to create global OVSDB databases > that > +store interconnection data. The databases can be setup on any nodes that > are > +accessible from all the central nodes of each OVN deployment. It is > +recommended that the global databases are setup with HA, with nodes in > +different avaialbility zones, to avoid single point of failure. > + > +1. Install OVN packages on each global database node. > + > +2. Start OVN IC-NB and IC-SB databases. > + > + On each global database node :: > + > + $ ovn-ctl [options] start_ic_ovsdb > + > + Options depends on the HA mode you use. See details with :: > + > + $ ovn-ctl --help. > + > +Register OVN to Interconnection Databases > +----------------------------------------- > + > +For each OVN deployment, set an availability zone name :: > + > + $ ovn-nbctl set NB_Global . name=<availability zone name> > + > +The name should be unique across all OVN deployments, e.g. ovn-east, > +ovn-west, etc. > + > +For each OVN deployment, start the ``ovn-ic`` daemon on central nodes :: > + > + $ ovn-ctl --ovninb-db=<IC-NB> --ovnisb-db=<IC-SB> \ > + --ovnnb-db=<NB> --ovnsb-db=<SB> [more options] start_ic > + > +An example of ``<IC-NB>`` is ``tcp:<global db hostname>:6645``, or for > +clustered DB: ``tcp:<node1>:6645,tcp:<node2>:6645,tcp:<node3>:6645``. > +``<IC-SB>`` is similar, but usually with a different port number, > typically, > +6646. > + > +For ``<NB>`` and ``<SB>``, use same connection methods as for starting > +``northd``. > + > +Verify each OVN registration from global IC-SB database, using > +``ovn-isbctl``, either on a global DB node or other nodes but with > property > +DB connection method specified in options :: > + > + $ ovn-isbctl show > + > +Configure Gateways > +------------------ > + > +For each OVN deployment, specify some chassises as interconnection > gateways. > +The number of gateways you need depends on the scale and bandwidth you > need for > +the traffic between the OVN deployments. > + > +For a node to work as an interconnection gateway, it must firstly be > installed > +and configured as a regular OVN chassis, with OVS and ``ovn-controller`` > +running. To make a chassis as an interconnection gateway, simply run the > +command on the chassis :: > + > + $ ovs-vsctl set open_vswitch . external_ids:is-interconn=true > + > +After configuring gateways, verify from the global IC-SB database :: > + > + $ ovn-isbctl show > + > +Create Transit Logical Switches > +------------------------------- > + > +Transit Logical Switches, or Transit Switches, are virtual switches for > +connecting logical routers in different OVN setups. :: > + > + $ ovn-inbctl ts-add <name> > + > +After creating a transit switch, it can be seen from each OVN deployment's > +Northbound database, which can be seen using :: > + > + $ ovn-nbctl find logical_switch other_config:interconn-ts=<name> > + > +You will also see it with simply ``ovn-nbctl ls-list``. > + > +If there are multiple tenants that require traffic being isolated from > each > +other, then multiple transit switches can be created accordingly. > + > +Connect Logical Routers to Transit Switches > +------------------------------------------- > + > +Connect logical routers from each OVN deployment to the desired transit > +switches just as if they are regular logical switches, which includes > below > +steps (from each OVN, for each logical router you want to connect). > + > +Assume a transit switch named ``ts1`` is already created in ``IC-NB`` and > a > +logical router ``lr1`` created in current OVN deployment. > + > +1. Create a logical router port. :: > + > + $ ovn-nbctl lrp-add lr1 lrp-lr1-ts1 aa:aa:aa:aa:aa:01 > 169.254.100.1/24 > + > + (The mac and IP are examples.) > + > +2. Create a logical switch port on the transit switch and peer with the > logical > + router port. :: > + > + $ ovn-nbctl lsp-add ts1 lsp-ts1-lr1 -- \ > + ovn-nbctl lsp-set-addresses lsp-ts1-lr1 router -- \ > + ovn-nbctl lsp-set-type lsp-ts1-lr1 router -- \ > + ovn-nbctl lsp-set-options lsp-ts1-lr1 > + > +3. Assign gateway(s) for the logical router port. :: > + > + $ ovn-nbctl lrp-set-gateway-chassis lrp-lr1-ts1 <gateway name> > [priority] > + > + Optionally, you can assign more gateways and specify priorities, to > achieve > + HA, just as usual for a distributed gateway port. > + > +Similarly in another OVN deployment, you can connect a logical router > (e.g. > +lr2) to the same transit switch the same way, with a different IP (e.g. > +169.254.100.2) on the same subnet. > + > +The ports connected to transit switches will be automatically populated to > +``IC-SB`` database, which can be verified by :: > + > + $ ovn-isbctl show > + > +Create Static Routes > +-------------------- > + > +Now that you have all the physical and logical topologies ready, simply > create > +static routes between the OVN deployments so that packets can be > forwarded by > +the logical routers through transit switches to the remote OVN. > + > +For example, in ovn-east, there are workloads using 10.0.1.0/24 under > lr1, and > +in ovn-west, there are workloads using 10.0.2.0/24 under lr2. > + > +In ovn-east, add below route :: > + > + $ ovn-nbctl lr-route-add lr1 10.0.2.0/24 169.254.100.2 > + > +In ovs-west, add below route :: > + > + $ ovn-nbctl lr-route-add lr1 10.0.1.0/24 169.254.100.1 > should be lr2. > + > +Now the traffic should be able to go through between the workloads through > +tunnels crossing gateway nodes of ovn-east and ovn-west. > diff --git a/NEWS b/NEWS > index 72e52b9..41a0d95 100644 > --- a/NEWS > +++ b/NEWS > @@ -3,6 +3,11 @@ Post-v2.12.0 > - This is the first release after OVN is split from Open vSwitch > project. > > + - OVN Interconnection: > + * Support for L3 interconnection of multiple OVN deployments with > tunnels > + managed by OVN. See instructions in > + Documentation/tutorials/ovn-interconnection.rst. > + > v2.12.0 - 03 Sep 2019 > --------------------- > - DPDK: > diff --git a/TODO.rst b/TODO.rst > index ed55ea2..5de1420 100644 > --- a/TODO.rst > +++ b/TODO.rst > @@ -155,3 +155,13 @@ OVN To-do List > > match(ip4.src == {IP1, IP2, IP3} && ip4.dst == {IP4, IP5, IP6} && > tcp.dst >= 1000 && tcp.dst <= 2000) actions=allow > + > +* OVN Interconnection > + > + * Packaging for RHEL, Debian, etc. > + > + * Gateway HA enhancement. Currently gateway HA relies on each OVN's BFD > + monitoring detecting gateway failover and updating port-binding in > + SB DB, which then is synced to IC-SB DB by ovn-ic. This may have > longer > + latency for failover than monitoring between each gateway pairs > + acrossing OVN. > -- > 2.1.0 > > _______________________________________________ > dev mailing list > d...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev