Would it make sense to move the control plane for this piece into the cluster? (vm in a mangement tenant?)
Thanks, Kevin ________________________________________ From: Florian Engelmann [florian.engelm...@everyware.ch] Sent: Thursday, October 25, 2018 7:39 AM To: openstack-operators@lists.openstack.org Subject: Re: [Openstack-operators] [octavia][rocky] Octavia and VxLAN without DVR It looks like devstack implemented some o-hm0 interface to connect the physical control host to a VxLAN. In our case there is no VxLAN at the control nodes nor is OVS. Is it a option to deploy those Octavia services needing this conenction to the compute or network nodes and use o-hm0? Am 10/25/18 um 10:22 AM schrieb Florian Engelmann: > Or could I create lb-mgmt-net as VxLAN and connect the control nodes to > this VxLAN? How to do something like that? > > Am 10/25/18 um 10:03 AM schrieb Florian Engelmann: >> Hmm - so right now I can't see any routed option because: >> >> The gateway connected to the VLAN provider networks (bond1 on the >> network nodes) is not able to route any traffic to my control nodes in >> the spine-leaf layer3 backend network. >> >> And right now there is no br-ex at all nor any "streched" L2 domain >> connecting all compute nodes. >> >> >> So the only solution I can think of right now is to create an overlay >> VxLAN in the spine-leaf backend network, connect all compute and >> control nodes to this overlay L2 network, create a OVS bridge >> connected to that network on the compute nodes and allow the Amphorae >> to get an IPin this network as well. >> Not to forget about DHCP... so the network nodes will need this bridge >> as well. >> >> Am 10/24/18 um 10:01 PM schrieb Erik McCormick: >>> >>> >>> On Wed, Oct 24, 2018, 3:33 PM Engelmann Florian >>> <florian.engelm...@everyware.ch >>> <mailto:florian.engelm...@everyware.ch>> wrote: >>> >>> On the network nodes we've got a dedicated interface to deploy VLANs >>> (like the provider network for internet access). What about creating >>> another VLAN on the network nodes, give that bridge a IP which is >>> part of the subnet of lb-mgmt-net and start the octavia worker, >>> healthmanager and controller on the network nodes binding to that >>> IP? >>> >>> The problem with that is you can't out an IP in the vlan interface >>> and also use it as an OVS bridge, so the Octavia processes would have >>> nothing to bind to. >>> >>> >>> ------------------------------------------------------------------------ >>> *From:* Erik McCormick <emccorm...@cirrusseven.com >>> <mailto:emccorm...@cirrusseven.com>> >>> *Sent:* Wednesday, October 24, 2018 6:18 PM >>> *To:* Engelmann Florian >>> *Cc:* openstack-operators >>> *Subject:* Re: [Openstack-operators] [octavia][rocky] Octavia and >>> VxLAN without DVR >>> >>> >>> On Wed, Oct 24, 2018, 12:02 PM Florian Engelmann >>> <florian.engelm...@everyware.ch >>> <mailto:florian.engelm...@everyware.ch>> wrote: >>> >>> Am 10/24/18 um 2:08 PM schrieb Erik McCormick: >>> > >>> > >>> > On Wed, Oct 24, 2018, 3:14 AM Florian Engelmann >>> > <florian.engelm...@everyware.ch >>> <mailto:florian.engelm...@everyware.ch> >>> <mailto:florian.engelm...@everyware.ch >>> <mailto:florian.engelm...@everyware.ch>>> >>> > wrote: >>> > >>> > Ohoh - thank you for your empathy :) >>> > And those great details about how to setup this mgmt >>> network. >>> > I will try to do so this afternoon but solving that >>> routing "puzzle" >>> > (virtual network to control nodes) I will need our >>> network guys to help >>> > me out... >>> > >>> > But I will need to tell all Amphorae a static route to >>> the gateway that >>> > is routing to the control nodes? >>> > >>> > >>> > Just set the default gateway when you create the neutron >>> subnet. No need >>> > for excess static routes. The route on the other connection >>> won't >>> > interfere with it as it lives in a namespace. >>> >>> >>> My compute nodes have no br-ex and there is no L2 domain spread >>> over all >>> compute nodes. As far as I understood lb-mgmt-net is a provider >>> network >>> and has to be flat or VLAN and will need a "physical" gateway >>> (as there >>> is no virtual router). >>> So the question - is it possible to get octavia up and running >>> without a >>> br-ex (L2 domain spread over all compute nodes) on the compute >>> nodes? >>> >>> >>> Sorry, I only meant it was *like* br-ex on your network nodes. You >>> don't need that on your computes. >>> >>> The router here would be whatever does routing in your physical >>> network. Setting the gateway in the neutron subnet simply adds that >>> to the DHCP information sent to the amphorae. >>> >>> This does bring up another thingI forgot though. You'll probably >>> want to add the management network / bridge to your network nodes or >>> wherever you run the DHCP agents. When you create the subnet, be >>> sure to leave some space in the address scope for the physical >>> devices with static IPs. >>> >>> As for multiple L2 domains, I can't think of a way to go about that >>> for the lb-mgmt network. It's a single network with a single subnet. >>> Perhaps you could limit load balancers to an AZ in a single rack? >>> Seems not very HA friendly. >>> >>> >>> >>> > >>> > >>> > >>> > Am 10/23/18 um 6:57 PM schrieb Erik McCormick: >>> > > So in your other email you said asked if there was a >>> guide for >>> > > deploying it with Kolla ansible... >>> > > >>> > > Oh boy. No there's not. I don't know if you've seen my >>> recent >>> > mails on >>> > > Octavia, but I am going through this deployment >>> process with >>> > > kolla-ansible right now and it is lacking in a few >>> areas. >>> > > >>> > > If you plan to use different CA certificates for >>> client and server in >>> > > Octavia, you'll need to add that into the playbook. >>> Presently it only >>> > > copies over ca_01.pem, cacert.key, and client.pem and >>> uses them for >>> > > everything. I was completely unable to make it work >>> with only one CA >>> > > as I got some SSL errors. It passes gate though, so I >>> aasume it must >>> > > work? I dunno. >>> > > >>> > > Networking comments and a really messy kolla-ansible / >>> octavia >>> > how-to below... >>> > > >>> > > On Tue, Oct 23, 2018 at 10:09 AM Florian Engelmann >>> > > <florian.engelm...@everyware.ch >>> <mailto:florian.engelm...@everyware.ch> >>> > <mailto:florian.engelm...@everyware.ch >>> <mailto:florian.engelm...@everyware.ch>>> wrote: >>> > >> >>> > >> Am 10/23/18 um 3:20 PM schrieb Erik McCormick: >>> > >>> On Tue, Oct 23, 2018 at 7:53 AM Florian Engelmann >>> > >>> <florian.engelm...@everyware.ch >>> <mailto:florian.engelm...@everyware.ch> >>> > <mailto:florian.engelm...@everyware.ch >>> <mailto:florian.engelm...@everyware.ch>>> wrote: >>> > >>>> >>> > >>>> Hi, >>> > >>>> >>> > >>>> We did test Octavia with Pike (DVR deployment) and >>> everything was >>> > >>>> working right our of the box. We changed our >>> underlay network to a >>> > >>>> Layer3 spine-leaf network now and did not deploy >>> DVR as we >>> > don't wanted >>> > >>>> to have that much cables in a rack. >>> > >>>> >>> > >>>> Octavia is not working right now as the lb-mgmt-net >>> does not >>> > exist on >>> > >>>> the compute nodes nor does a br-ex. >>> > >>>> >>> > >>>> The control nodes running >>> > >>>> >>> > >>>> octavia_worker >>> > >>>> octavia_housekeeping >>> > >>>> octavia_health_manager >>> > >>>> octavia_api >>> > >>>> >>> > Amphorae-VMs, z.b. >>> > >>> > lb-mgmt-net 172.16.0.0/16 <http://172.16.0.0/16> >>> <http://172.16.0.0/16> default GW >>> > >>>> and as far as I understood octavia_worker, >>> > octavia_housekeeping and >>> > >>>> octavia_health_manager have to talk to the amphora >>> instances. >>> > But the >>> > >>>> control nodes are spread over three different >>> leafs. So each >>> > control >>> > >>>> node in a different L2 domain. >>> > >>>> >>> > >>>> So the question is how to deploy a lb-mgmt-net >>> network in our >>> > setup? >>> > >>>> >>> > >>>> - Compute nodes have no "stretched" L2 domain >>> > >>>> - Control nodes, compute nodes and network nodes >>> are in L3 >>> > networks like >>> > >>>> api, storage, ... >>> > >>>> - Only network nodes are connected to a L2 domain >>> (with a >>> > separated NIC) >>> > >>>> providing the "public" network >>> > >>>> >>> > >>> You'll need to add a new bridge to your compute >>> nodes and create a >>> > >>> provider network associated with that bridge. In my >>> setup this is >>> > >>> simply a flat network tied to a tagged interface. In >>> your case it >>> > >>> probably makes more sense to make a new VNI and >>> create a vxlan >>> > >>> provider network. The routing in your switches >>> should handle >>> > the rest. >>> > >> >>> > >> Ok that's what I try right now. But I don't get how >>> to setup >>> > something >>> > >> like a VxLAN provider Network. I thought only vlan >>> and flat is >>> > supported >>> > >> as provider network? I guess it is not possible to >>> use the tunnel >>> > >> interface that is used for tenant networks? >>> > >> So I have to create a separated VxLAN on the control >>> and compute >>> > nodes like: >>> > >> >>> > >> # ip link add vxoctavia type vxlan id 42 dstport 4790 >>> group >>> > 239.1.1.1 >>> > >> dev vlan3535 ttl 5 >>> > >> # ip addr add 172.16.1.11/20 <http://172.16.1.11/20> >>> <http://172.16.1.11/20> dev vxoctavia >>> > >> # ip link set vxoctavia up >>> > >> >>> > >> and use it like a flat provider network, true? >>> > >> >>> > > This is a fine way of doing things, but it's only half >>> the battle. >>> > > You'll need to add a bridge on the compute nodes and >>> bind it to that >>> > > new interface. Something like this if you're using >>> openvswitch: >>> > > >>> > > docker exec openvswitch_db >>> > > /usr/local/bin/kolla_ensure_openvswitch_configured >>> br-mgmt vxoctavia >>> > > >>> > > Also you'll want to remove the IP address from that >>> interface as it's >>> > > going to be a bridge. Think of it like your public >>> (br-ex) interface >>> > > on your network nodes. >>> > > >>> > > From there you'll need to update the bridge mappings >>> via kolla >>> > > overrides. This would usually be in >>> /etc/kolla/config/neutron. Create >>> > > a subdirectory for your compute inventory group and >>> create an >>> > > ml2_conf.ini there. So you'd end up with something >>> like: >>> > > >>> > > [root@kolla-deploy ~]# cat >>> > /etc/kolla/config/neutron/compute/ml2_conf.ini >>> > > [ml2_type_flat] >>> > > flat_networks = mgmt-net >>> > > >>> > > [ovs] >>> > > bridge_mappings = mgmt-net:br-mgmt >>> > > >>> > > run kolla-ansible --tags neutron reconfigure to push >>> out the new >>> > > configs. Note that there is a bug where the neutron >>> containers >>> > may not >>> > > restart after the change, so you'll probably need to >>> do a 'docker >>> > > container restart neutron_openvswitch_agent' on each >>> compute node. >>> > > >>> > > At this point, you'll need to create the provider >>> network in the >>> > admin >>> > > project like: >>> > > >>> > > openstack network create --provider-network-type flat >>> > > --provider-physical-network mgmt-net lb-mgmt-net >>> > > >>> > > And then create a normal subnet attached to this >>> network with some >>> > > largeish address scope. I wouldn't use 172.16.0.0/16 >>> <http://172.16.0.0/16> >>> > <http://172.16.0.0/16> because docker >>> > > uses that by default. I'm not sure if it matters since >>> the network >>> > > traffic will be isolated on a bridge, but it makes me >>> paranoid so I >>> > > avoided it. >>> > > >>> > > For your controllers, I think you can just let >>> everything >>> > function off >>> > > your api interface since you're routing in your >>> spines. Set up a >>> > > gateway somewhere from that lb-mgmt network and save >>> yourself the >>> > > complication of adding an interface to your >>> controllers. If you >>> > choose >>> > > to use a separate interface on your controllers, >>> you'll need to make >>> > > sure this patch is in your kolla-ansible install or >>> cherry pick it. >>> > > >>> > > >>> > >>> https://github.com/openstack/kolla-ansible/commit/0b6e401c4fdb9aa4ff87d0bfd4b25c91b86e0d60#diff-6c871f6865aecf0057a5b5f677ae7d59 >>> >>> > > >>> > > I don't think that's been backported at all, so unless >>> you're running >>> > > off master you'll need to go get it. >>> > > >>> > > From here on out, the regular Octavia instruction >>> should serve you. >>> > > Create a flavor, Create a security group, and capture >>> their UUIDs >>> > > along with the UUID of the provider network you made. >>> Override >>> > them in >>> > > globals.yml with: >>> > > >>> > > octavia_amp_boot_network_list: <uuid> >>> > > octavia_amp_secgroup_list: <uuid> >>> > > octavia_amp_flavor_id: <uuid> >>> > > >>> > > This is all from my scattered notes and bad memory. >>> Hopefully it >>> > makes >>> > > sense. Corrections welcome. >>> > > >>> > > -Erik >>> > > >>> > > >>> > > >>> > >> >>> > >> >>> > >>> >>> > >>> -Erik >>> > >>>> >>> > >>>> All the best, >>> > >>>> Florian >>> > >>>> _______________________________________________ >>> > >>>> OpenStack-operators mailing list >>> > >>>> OpenStack-operators@lists.openstack.org >>> <mailto:OpenStack-operators@lists.openstack.org> >>> > <mailto:OpenStack-operators@lists.openstack.org >>> <mailto:OpenStack-operators@lists.openstack.org>> >>> > >>>> >>> > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >>> > >>> > -- >>> > >>> > EveryWare AG >>> > Florian Engelmann >>> > Systems Engineer >>> > Zurlindenstrasse 52a >>> > CH-8003 Zürich >>> > >>> > tel: +41 44 466 60 00 >>> > fax: +41 44 466 60 10 >>> > mail: mailto:florian.engelm...@everyware.ch >>> <mailto:florian.engelm...@everyware.ch> >>> > <mailto:florian.engelm...@everyware.ch >>> <mailto:florian.engelm...@everyware.ch>> >>> > web: http://www.everyware.ch >>> > >>> >>> -- >>> EveryWare AG >>> Florian Engelmann >>> Systems Engineer >>> Zurlindenstrasse 52a >>> CH-8003 Zürich >>> >>> tel: +41 44 466 60 00 >>> fax: +41 44 466 60 10 >>> mail: mailto:florian.engelm...@everyware.ch >>> <mailto:florian.engelm...@everyware.ch> >>> web: http://www.everyware.ch >>> >> >> >> _______________________________________________ >> OpenStack-operators mailing list >> OpenStack-operators@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> > > > _______________________________________________ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > -- EveryWare AG Florian Engelmann Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelm...@everyware.ch web: http://www.everyware.ch _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators