Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints
currently we are testing what is needed to get consul + registrator and kolla/kolla-ansible play together nicely. To get the services created in consul by registrator all kolla containers running relevant services (eg. keystone, nova, cinder, ... but also mariadb, memcached, es, ...) need to "--expose" their ports. Registrator will use those "exposed" ports to add a service to consul. I there any (existing) option to add those ports to the container bootstrap? What about "docker_common_options"? command should look like: docker run -d --expose 5000/tcp --expose 35357/tcp --name=keystone ... After testing registrator I recognized the project seems to be unmaintained. So we won't use registrator. I just need to find another method to register a container (service) in consul after the contaier has started. I would like to do so without changing any kolla container or kolla-ansible code. Am 10/10/18 um 9:18 AM schrieb Florian Engelmann: by "another storage system" you mean the KV store of consul? That's just someting consul brings with it... consul is very strong in doing health checks Am 10/9/18 um 6:09 PM schrieb Fox, Kevin M: etcd is an already approved openstack dependency. Could that be used instead of consul so as to not add yet another storage system? coredns with the https://coredns.io/plugins/etcd/ plugin would maybe do what you need? Thanks, Kevin ________ From: Florian Engelmann [florian.engelm...@everyware.ch] Sent: Monday, October 08, 2018 3:14 AM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints Hi, I would like to start a discussion about some changes and additions I would like to see in in kolla and kolla-ansible. 1. Keepalived is a problem in layer3 spine leaf networks as any floating IP can only exist in one leaf (and VRRP is a problem in layer3). I would like to use consul and registrar to get rid of the "internal" floating IP and use consuls DNS service discovery to connect all services with each other. 2. Using "ports" for external API (endpoint) access is a major headache if a firewall is involved. I would like to configure the HAProxy (or fabio) for the external access to use "Host:" like, eg. "Host: keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with HTTPS. Any customer would just need HTTPS access and not have to open all those ports in his firewall. For some enterprise customers it is not possible to request FW changes like that. 3. HAProxy is not capable to handle "read/write" split with Galera. I would like to introduce ProxySQL to be able to scale Galera. 4. HAProxy is fine but fabio integrates well with consul, statsd and could be connected to a vault cluster to manage secure certificate access. 5. I would like to add vault as Barbican backend. 6. I would like to add an option to enable tokenless authentication for all services with each other to get rid of all the openstack service passwords (security issue). What do you think about it? All the best, Florian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- EveryWare AG Florian Engelmann Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelm...@everyware.ch web: http://www.everyware.ch smime.p7s Description: S/MIME cryptographic signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints
On 17.10.2018 15:45, Florian Engelmann wrote: On 10.10.2018 09:06, Florian Engelmann wrote: Now I get you. I would say all configuration templates need to be changed to allow, eg. $ grep http /etc/kolla/cinder-volume/cinder.conf glance_api_servers = http://10.10.10.5:9292 auth_url = http://internal.somedomain.tld:35357 www_authenticate_uri = http://internal.somedomain.tld:5000 auth_url = http://internal.somedomain.tld:35357 auth_endpoint = http://internal.somedomain.tld:5000 to look like: glance_api_servers = http://glance.service.somedomain.consul:9292 auth_url = http://keystone.service.somedomain.consul:35357 www_authenticate_uri = http://keystone.service.somedomain.consul:5000 auth_url = http://keystone.service.somedomain.consul:35357 auth_endpoint = http://keystone.service.somedomain.consul:5000 The idea with Consul looks interesting. But I don't get your issue with VIP address and spine-leaf network. What we have: - controller1 behind leaf1 A/B pair with MLAG - controller2 behind leaf2 A/B pair with MLAG - controller3 behind leaf3 A/B pair with MLAG The VIP address is active on one controller server. When the server fail then the VIP will move to another controller server. Where do you see a SPOF in this configuration? So leaf1 2 and 3 have to share the same L2 domain, right (in IPv4 network)? Yes, they share L2 domain but we have ARP and ND suppression enabled. It is an EVPN network where there is a L3 with VxLANs between leafs and spines. So we don't care where a server is connected. It can be connected to any leaf. Ok that sounds very interesting. Is it possible to share some internals? Which switch vendor/model do you use? How does you IP address schema look like? If VxLAN is used between spine and leafs are you using VxLAN networking for Openstack as well? Where is your VTEP? But we wanna deploy a layer3 spine-leaf network were every leaf is it's own L2 domain and everything above is layer3. eg: leaf1 = 10.1.1.0/24 leaf2 = 10.1.2.0/24 leaf2 = 10.1.3.0/24 So a VIP like, eg. 10.1.1.10 could only exist in leaf1 In my opinion it's a very constrained environment, I don't like the idea. Regards, Piotr __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- EveryWare AG Florian Engelmann Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelm...@everyware.ch web: http://www.everyware.ch smime.p7s Description: S/MIME cryptographic signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints
No, I mean, Consul would be an extra dependency in a big list of dependencies OpenStack already has. OpenStack has so many it is causing operators to reconsider adoption. I'm asking, if existing dependencies can be made to solve the problem without adding more? Stateful dependencies are much harder to deal with then stateless ones, as they take much more operator care/attention. Consul is stateful as is etcd, and etcd is already a dependency. Can etcd be used instead so as not to put more load on the operators? While etcd is a strong KV store it lacks many features consul has. Using consul for DNS based service discovery is very easy to implement without making it a dependency. So we will start with a "external" consul and see how to handle the service registration without modifying the kolla containers or any kolla-ansible code. All the best, Flo Thanks, Kevin ____ From: Florian Engelmann [florian.engelm...@everyware.ch] Sent: Wednesday, October 10, 2018 12:18 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints by "another storage system" you mean the KV store of consul? That's just someting consul brings with it... consul is very strong in doing health checks Am 10/9/18 um 6:09 PM schrieb Fox, Kevin M: etcd is an already approved openstack dependency. Could that be used instead of consul so as to not add yet another storage system? coredns with the https://coredns.io/plugins/etcd/ plugin would maybe do what you need? Thanks, Kevin ________ From: Florian Engelmann [florian.engelm...@everyware.ch] Sent: Monday, October 08, 2018 3:14 AM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints Hi, I would like to start a discussion about some changes and additions I would like to see in in kolla and kolla-ansible. 1. Keepalived is a problem in layer3 spine leaf networks as any floating IP can only exist in one leaf (and VRRP is a problem in layer3). I would like to use consul and registrar to get rid of the "internal" floating IP and use consuls DNS service discovery to connect all services with each other. 2. Using "ports" for external API (endpoint) access is a major headache if a firewall is involved. I would like to configure the HAProxy (or fabio) for the external access to use "Host:" like, eg. "Host: keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with HTTPS. Any customer would just need HTTPS access and not have to open all those ports in his firewall. For some enterprise customers it is not possible to request FW changes like that. 3. HAProxy is not capable to handle "read/write" split with Galera. I would like to introduce ProxySQL to be able to scale Galera. 4. HAProxy is fine but fabio integrates well with consul, statsd and could be connected to a vault cluster to manage secure certificate access. 5. I would like to add vault as Barbican backend. 6. I would like to add an option to enable tokenless authentication for all services with each other to get rid of all the openstack service passwords (security issue). What do you think about it? All the best, Florian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- EveryWare AG Florian Engelmann Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelm...@everyware.ch web: http://www.everyware.ch __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- EveryWare AG Florian Engelmann Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelm...@everyware.ch web: http://www.everyware.ch smime.p7s Description: S/MIME cryptographic signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints
currently we are testing what is needed to get consul + registrator and kolla/kolla-ansible play together nicely. To get the services created in consul by registrator all kolla containers running relevant services (eg. keystone, nova, cinder, ... but also mariadb, memcached, es, ...) need to "--expose" their ports. Registrator will use those "exposed" ports to add a service to consul. I there any (existing) option to add those ports to the container bootstrap? What about "docker_common_options"? command should look like: docker run -d --expose 5000/tcp --expose 35357/tcp --name=keystone ... Am 10/10/18 um 9:18 AM schrieb Florian Engelmann: by "another storage system" you mean the KV store of consul? That's just someting consul brings with it... consul is very strong in doing health checks Am 10/9/18 um 6:09 PM schrieb Fox, Kevin M: etcd is an already approved openstack dependency. Could that be used instead of consul so as to not add yet another storage system? coredns with the https://coredns.io/plugins/etcd/ plugin would maybe do what you need? Thanks, Kevin ________ From: Florian Engelmann [florian.engelm...@everyware.ch] Sent: Monday, October 08, 2018 3:14 AM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints Hi, I would like to start a discussion about some changes and additions I would like to see in in kolla and kolla-ansible. 1. Keepalived is a problem in layer3 spine leaf networks as any floating IP can only exist in one leaf (and VRRP is a problem in layer3). I would like to use consul and registrar to get rid of the "internal" floating IP and use consuls DNS service discovery to connect all services with each other. 2. Using "ports" for external API (endpoint) access is a major headache if a firewall is involved. I would like to configure the HAProxy (or fabio) for the external access to use "Host:" like, eg. "Host: keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with HTTPS. Any customer would just need HTTPS access and not have to open all those ports in his firewall. For some enterprise customers it is not possible to request FW changes like that. 3. HAProxy is not capable to handle "read/write" split with Galera. I would like to introduce ProxySQL to be able to scale Galera. 4. HAProxy is fine but fabio integrates well with consul, statsd and could be connected to a vault cluster to manage secure certificate access. 5. I would like to add vault as Barbican backend. 6. I would like to add an option to enable tokenless authentication for all services with each other to get rid of all the openstack service passwords (security issue). What do you think about it? All the best, Florian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- EveryWare AG Florian Engelmann Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelm...@everyware.ch web: http://www.everyware.ch smime.p7s Description: S/MIME cryptographic signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints
On 10.10.2018 09:06, Florian Engelmann wrote: Now I get you. I would say all configuration templates need to be changed to allow, eg. $ grep http /etc/kolla/cinder-volume/cinder.conf glance_api_servers = http://10.10.10.5:9292 auth_url = http://internal.somedomain.tld:35357 www_authenticate_uri = http://internal.somedomain.tld:5000 auth_url = http://internal.somedomain.tld:35357 auth_endpoint = http://internal.somedomain.tld:5000 to look like: glance_api_servers = http://glance.service.somedomain.consul:9292 auth_url = http://keystone.service.somedomain.consul:35357 www_authenticate_uri = http://keystone.service.somedomain.consul:5000 auth_url = http://keystone.service.somedomain.consul:35357 auth_endpoint = http://keystone.service.somedomain.consul:5000 The idea with Consul looks interesting. But I don't get your issue with VIP address and spine-leaf network. What we have: - controller1 behind leaf1 A/B pair with MLAG - controller2 behind leaf2 A/B pair with MLAG - controller3 behind leaf3 A/B pair with MLAG The VIP address is active on one controller server. When the server fail then the VIP will move to another controller server. Where do you see a SPOF in this configuration? So leaf1 2 and 3 have to share the same L2 domain, right (in IPv4 network)? But we wanna deploy a layer3 spine-leaf network were every leaf is it's own L2 domain and everything above is layer3. eg: leaf1 = 10.1.1.0/24 leaf2 = 10.1.2.0/24 leaf2 = 10.1.3.0/24 So a VIP like, eg. 10.1.1.10 could only exist in leaf1 smime.p7s Description: S/MIME cryptographic signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][glance][cinder][nova][keystone] healthcheck
Thank you very much for your detailed answer. Keystone healthcheck is working fine and as you said out of the box. I got trouble with, eg. neutron-server and cinder-api. While nova is happy with: [filter:healthcheck] paste.filter_factory = oslo_middleware:Healthcheck.factory backends = disable_by_file disable_by_file_path = /var/log/kolla/nova/healthcheck_disable and some changes to the pipeline: [pipeline:oscomputeversions] pipeline = healthcheck cors faultwrap request_log http_proxy_to_wsgi oscomputeversionapp I was not able to get the same thing working with neutron-server as it's paste configuration is very different: [composite:neutron] use = egg:Paste#urlmap /: neutronversions_composite /v2.0: neutronapi_v2_0 [composite:neutronapi_v2_0] use = call:neutron.auth:pipeline_factory noauth = cors http_proxy_to_wsgi request_id catch_errors extensions neutronapiapp_v2_0 keystone = cors http_proxy_to_wsgi request_id catch_errors authtoken keystonecontext extensions neutronapiapp_v2_0 [composite:neutronversions_composite] use = call:neutron.auth:pipeline_factory noauth = cors http_proxy_to_wsgi neutronversions keystone = cors http_proxy_to_wsgi neutronversions [filter:request_id] paste.filter_factory = oslo_middleware:RequestId.factory [filter:catch_errors] paste.filter_factory = oslo_middleware:CatchErrors.factory [filter:cors] paste.filter_factory = oslo_middleware.cors:filter_factory oslo_config_project = neutron [filter:http_proxy_to_wsgi] paste.filter_factory = oslo_middleware.http_proxy_to_wsgi:HTTPProxyToWSGI.factory [filter:keystonecontext] paste.filter_factory = neutron.auth:NeutronKeystoneContext.factory [filter:authtoken] paste.filter_factory = keystonemiddleware.auth_token:filter_factory [filter:extensions] paste.filter_factory = neutron.api.extensions:plugin_aware_extension_middleware_factory [app:neutronversions] paste.app_factory = neutron.pecan_wsgi.app:versions_factory [app:neutronapiapp_v2_0] paste.app_factory = neutron.api.v2.router:APIRouter.factory [filter:osprofiler] paste.filter_factory = osprofiler.web:WsgiMiddleware.factory #[filter:healthcheck] #paste.filter_factory = oslo_middleware:Healthcheck.factory #backends = disable_by_file #disable_by_file_path = /var/log/kolla/neutron/healthcheck_disable I did read the oslo middleware documentation a couple of times but I still don't get what to do to enable the healthcheck API with neutron-server. Is there any "tutorial" that could help? Am 10/12/18 um 7:50 PM schrieb Morgan Fainberg: Keystone no longer uses paste (since Rocky) as paste is unmaintained. The healthcheck app is permanently enabled for keystone at /healthcheck. We chose to make it a default bit of functionality in how we have Keystone deployed. We also have unit tests in place to ensure we don't regress and healthcheck changes behavior down the line (future releases). You should be able to configure additional bits for healthcheck in keystone.conf (e.g. detailed mode, disable-by-file, etc). Cheers, --Morgan On Fri, Oct 12, 2018 at 3:07 AM Florian Engelmann mailto:florian.engelm...@everyware.ch>> wrote: Hi, I tried to configure the healthcheck framework (/healthcheck) for nova, cinder, glance and keystone but it looks like paste is not used with keystone anymore? https://github.com/openstack/keystone/commit/8bf335bb015447448097a5c08b870da8e537a858 In our rocky deployment the healthcheck is working for keystone only and I failed to configure it for, eg. nova-api. Nova seems to use paste? Is there any example nova api-paste.ini with the olso healthcheck middleware enabled? To documentation is hard to understand - at least for me. Thank you for your help. All the best, Florian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- EveryWare AG Florian Engelmann Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelm...@everyware.ch web: http://www.everyware.ch smime.p7s Description: S/MIME cryptographic signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo][glance][cinder][nova][keystone] healthcheck
Hi, I tried to configure the healthcheck framework (/healthcheck) for nova, cinder, glance and keystone but it looks like paste is not used with keystone anymore? https://github.com/openstack/keystone/commit/8bf335bb015447448097a5c08b870da8e537a858 In our rocky deployment the healthcheck is working for keystone only and I failed to configure it for, eg. nova-api. Nova seems to use paste? Is there any example nova api-paste.ini with the olso healthcheck middleware enabled? To documentation is hard to understand - at least for me. Thank you for your help. All the best, Florian smime.p7s Description: S/MIME cryptographic signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints
by "another storage system" you mean the KV store of consul? That's just someting consul brings with it... consul is very strong in doing health checks Am 10/9/18 um 6:09 PM schrieb Fox, Kevin M: etcd is an already approved openstack dependency. Could that be used instead of consul so as to not add yet another storage system? coredns with the https://coredns.io/plugins/etcd/ plugin would maybe do what you need? Thanks, Kevin ____ From: Florian Engelmann [florian.engelm...@everyware.ch] Sent: Monday, October 08, 2018 3:14 AM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints Hi, I would like to start a discussion about some changes and additions I would like to see in in kolla and kolla-ansible. 1. Keepalived is a problem in layer3 spine leaf networks as any floating IP can only exist in one leaf (and VRRP is a problem in layer3). I would like to use consul and registrar to get rid of the "internal" floating IP and use consuls DNS service discovery to connect all services with each other. 2. Using "ports" for external API (endpoint) access is a major headache if a firewall is involved. I would like to configure the HAProxy (or fabio) for the external access to use "Host:" like, eg. "Host: keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with HTTPS. Any customer would just need HTTPS access and not have to open all those ports in his firewall. For some enterprise customers it is not possible to request FW changes like that. 3. HAProxy is not capable to handle "read/write" split with Galera. I would like to introduce ProxySQL to be able to scale Galera. 4. HAProxy is fine but fabio integrates well with consul, statsd and could be connected to a vault cluster to manage secure certificate access. 5. I would like to add vault as Barbican backend. 6. I would like to add an option to enable tokenless authentication for all services with each other to get rid of all the openstack service passwords (security issue). What do you think about it? All the best, Florian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- EveryWare AG Florian Engelmann Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelm...@everyware.ch web: http://www.everyware.ch smime.p7s Description: S/MIME cryptographic signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints
Am 10/9/18 um 1:47 PM schrieb Mark Goddard:
On Tue, 9 Oct 2018 at 12:03, Florian Engelmann
mailto:florian.engelm...@everyware.ch>>
wrote:
Am 10/9/18 um 11:04 AM schrieb Mark Goddard:
> Thanks for these suggestions Florian, there are some interesting
ideas
> in here. I'm a little concerned about the maintenance overhead of
adding
> support for all of these things, and wonder if some of them could be
> done without explicit support in kolla and kolla-ansible. The kolla
> projects have been able to move quickly by providing a flexible
> configuration mechanism that avoids the need to maintain support for
> every OpenStack feature. Other thoughts inline.
>
I do understand your apprehensions Mark. For some of the suggested
changes/additions I agree. But adding those components without
kolla/kolla-ansible integration feels not right.
I'm not entirely against adding some of these things, if enough people
in the community want them. I'd just like to make sure that if they can
be done in a sane way without changes, then we do that and document how
to do it instead.
Yes I agree and that is a very important role/job.
>
> On Mon, 8 Oct 2018 at 11:15, Florian Engelmann
> mailto:florian.engelm...@everyware.ch>
<mailto:florian.engelm...@everyware.ch
<mailto:florian.engelm...@everyware.ch>>>
> wrote:
>
> Hi,
>
> I would like to start a discussion about some changes and
additions I
> would like to see in in kolla and kolla-ansible.
>
> 1. Keepalived is a problem in layer3 spine leaf networks as any
> floating
> IP can only exist in one leaf (and VRRP is a problem in
layer3). I
> would
> like to use consul and registrar to get rid of the "internal"
floating
> IP and use consuls DNS service discovery to connect all
services with
> each other.
>
>
> Without reading up, I'm not sure exactly how this fits together. If
> kolla-ansible made the API host configurable for each service rather
> than globally, would that be a step in the right direction?
No that would not help. The problem is HA. Right now there is a
"central" floating IP (kolla_internal_vip_address) that is used for all
services to connect to (each other). Keepalived is failing that IP over
if the "active" host fails. In a layer3 (CLOS/Spine-Leaf) network this
IP is only available in one leaf/rack. So that rack is a "SPOF".
Using service discovery fits perfect in a CLOS network and scales much
better as a HA solution.
Right, but what I'm saying as a thought experiment is, if we gave you
the required variables in kolla-ansible (e.g. nova_internal_fqdn) to
make this possible with an externally managed consul service, could that
work?
Now I get you. I would say all configuration templates need to be
changed to allow, eg.
$ grep http /etc/kolla/cinder-volume/cinder.conf
glance_api_servers = http://10.10.10.5:9292
auth_url = http://internal.somedomain.tld:35357
www_authenticate_uri = http://internal.somedomain.tld:5000
auth_url = http://internal.somedomain.tld:35357
auth_endpoint = http://internal.somedomain.tld:5000
to look like:
glance_api_servers = http://glance.service.somedomain.consul:9292
auth_url = http://keystone.service.somedomain.consul:35357
www_authenticate_uri = http://keystone.service.somedomain.consul:5000
auth_url = http://keystone.service.somedomain.consul:35357
auth_endpoint = http://keystone.service.somedomain.consul:5000
>
>
> 2. Using "ports" for external API (endpoint) access is a
major headache
> if a firewall is involved. I would like to configure the
HAProxy (or
> fabio) for the external access to use "Host:" like, eg. "Host:
> keystone.somedomain.tld", "Host: nova.somedomain.tld", ...
with HTTPS.
> Any customer would just need HTTPS access and not have to
open all
> those
> ports in his firewall. For some enterprise customers it is
not possible
> to request FW changes like that.
>
> 3. HAProxy is not capable to handle "read/write" split with
Galera. I
> would like to introduce ProxySQL to be able to scale Galera.
>
>
> It's now possible to use an external database server with
kolla-ansible,
> instead of deploying a mariadb/galera cluster. This could be
implemented
> how you like, see
>
https://docs.openstack.org/kolla-ansible/latest/reference/external-mariadb-guide.html.
Yes I agree. And this is what we will do
Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints
Am 10/9/18 um 1:23 PM schrieb Jay Pipes: On 10/09/2018 06:34 AM, Florian Engelmann wrote: Am 10/9/18 um 11:41 AM schrieb Jay Pipes: On 10/09/2018 04:34 AM, Christian Berendt wrote: On 8. Oct 2018, at 19:48, Jay Pipes wrote: Why not send all read and all write traffic to a single haproxy endpoint and just have haproxy spread all traffic across each Galera node? Galera, after all, is multi-master synchronous replication... so it shouldn't matter which node in the Galera cluster you send traffic to. Probably because of MySQL deadlocks in Galera: —snip— Galera cluster has known limitations, one of them is that it uses cluster-wide optimistic locking. This may cause some transactions to rollback. With an increasing number of writeable masters, the transaction rollback rate may increase, especially if there is write contention on the same dataset. It is of course possible to retry the transaction and perhaps it will COMMIT in the retries, but this will add to the transaction latency. However, some designs are deadlock prone, e.g sequence tables. —snap— Source: https://severalnines.com/resources/tutorials/mysql-load-balancing-haproxy-tutorial Have you seen the above in production? Yes of course. Just depends on the application and how high the workload gets. Please read about deadloks and nova in the following report by Intel: http://galeracluster.com/wp-content/uploads/2017/06/performance_analysis_and_tuning_in_china_mobiles_openstack_production_cloud_2.pdf I have read the above. It's a synthetic workload analysis, which is why I asked if you'd seen this in production. For the record, we addressed much of the contention/races mentioned in the above around scheduler resource consumption in the Ocata and Pike releases of Nova. I'm aware that the report above identifies the quota handling code in Nova as the primary culprit of the deadlock issues but again, it's a synthetic workload that is designed to find breaking points. It doesn't represent a realistic production workload. You can read about the deadlock issue in depth on my blog here: http://www.joinfu.com/2015/01/understanding-reservations-concurrency-locking-in-nova/ That explains where the source of the problem comes from (it's the use of SELECT FOR UPDATE, which has been removed from Nova's quota-handling code in the Rocky release). Thank you very much for your link. Great article!!! Took my a while to read it and understand everything but helped a lot!!! If just Nova is affected we could also create an additional HAProxy listener using all Galera nodes with round-robin for all other services? I fail to see the point of using Galera with a single writer. At that point, why bother with Galera at all? Just use a single database node with a single slave for backup purposes. From my point of view Galera is easy to manage and great for HA. Having to handle a manual failover in production with mysql master/slave never was fun... Indeed writing to a single node and not using the other nodes (even for read, like it is done in kolla-ansible) is not the best solution. Galera is slower than a standalone MySQL. Using ProxySQL would enable us to use caching and read/write split to speed up database queries while HA and management are still good. Anyway - proxySQL would be a great extension. I don't disagree that proxySQL is a good extension. However, it adds yet another services to the mesh that needs to be deployed, configured and maintained. True. I guess we will start with an external MySQL installation to collect some experience. smime.p7s Description: S/MIME cryptographic signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints
Am 10/9/18 um 11:04 AM schrieb Mark Goddard: Thanks for these suggestions Florian, there are some interesting ideas in here. I'm a little concerned about the maintenance overhead of adding support for all of these things, and wonder if some of them could be done without explicit support in kolla and kolla-ansible. The kolla projects have been able to move quickly by providing a flexible configuration mechanism that avoids the need to maintain support for every OpenStack feature. Other thoughts inline. I do understand your apprehensions Mark. For some of the suggested changes/additions I agree. But adding those components without kolla/kolla-ansible integration feels not right. On Mon, 8 Oct 2018 at 11:15, Florian Engelmann mailto:florian.engelm...@everyware.ch>> wrote: Hi, I would like to start a discussion about some changes and additions I would like to see in in kolla and kolla-ansible. 1. Keepalived is a problem in layer3 spine leaf networks as any floating IP can only exist in one leaf (and VRRP is a problem in layer3). I would like to use consul and registrar to get rid of the "internal" floating IP and use consuls DNS service discovery to connect all services with each other. Without reading up, I'm not sure exactly how this fits together. If kolla-ansible made the API host configurable for each service rather than globally, would that be a step in the right direction? No that would not help. The problem is HA. Right now there is a "central" floating IP (kolla_internal_vip_address) that is used for all services to connect to (each other). Keepalived is failing that IP over if the "active" host fails. In a layer3 (CLOS/Spine-Leaf) network this IP is only available in one leaf/rack. So that rack is a "SPOF". Using service discovery fits perfect in a CLOS network and scales much better as a HA solution. 2. Using "ports" for external API (endpoint) access is a major headache if a firewall is involved. I would like to configure the HAProxy (or fabio) for the external access to use "Host:" like, eg. "Host: keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with HTTPS. Any customer would just need HTTPS access and not have to open all those ports in his firewall. For some enterprise customers it is not possible to request FW changes like that. 3. HAProxy is not capable to handle "read/write" split with Galera. I would like to introduce ProxySQL to be able to scale Galera. It's now possible to use an external database server with kolla-ansible, instead of deploying a mariadb/galera cluster. This could be implemented how you like, see https://docs.openstack.org/kolla-ansible/latest/reference/external-mariadb-guide.html. Yes I agree. And this is what we will do in our first production deployment. But I would love to see ProxySQL in Kolla as well. As a side note: Kolla-ansible does use: option mysql-check user haproxy post-41 to check Galera, but that check does not fail if the node is out of sync with the other nodes! http://galeracluster.com/documentation-webpages/monitoringthecluster.html 4. HAProxy is fine but fabio integrates well with consul, statsd and could be connected to a vault cluster to manage secure certificate access. As above. 5. I would like to add vault as Barbican backend. Does this need explicit support in kolla and kolla-ansible, or could it be done through configuration of barbican.conf? Are there additional packages required in the barbican container? If so, see https://docs.openstack.org/kolla/latest/admin/image-building.html#package-customisation. True but the vault (and consul) containers could be deployed and managed by kolla-ansible. 6. I would like to add an option to enable tokenless authentication for all services with each other to get rid of all the openstack service passwords (security issue). Again, could this be done without explicit support? We did not investigate here. Changes to the apache configuration are needed. I guess we will have to change the kolla container itself to do so? Is it possible to "inject" files in a container using kolla-ansible? smime.p7s Description: S/MIME cryptographic signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints
Am 10/9/18 um 11:41 AM schrieb Jay Pipes: On 10/09/2018 04:34 AM, Christian Berendt wrote: On 8. Oct 2018, at 19:48, Jay Pipes wrote: Why not send all read and all write traffic to a single haproxy endpoint and just have haproxy spread all traffic across each Galera node? Galera, after all, is multi-master synchronous replication... so it shouldn't matter which node in the Galera cluster you send traffic to. Probably because of MySQL deadlocks in Galera: —snip— Galera cluster has known limitations, one of them is that it uses cluster-wide optimistic locking. This may cause some transactions to rollback. With an increasing number of writeable masters, the transaction rollback rate may increase, especially if there is write contention on the same dataset. It is of course possible to retry the transaction and perhaps it will COMMIT in the retries, but this will add to the transaction latency. However, some designs are deadlock prone, e.g sequence tables. —snap— Source: https://severalnines.com/resources/tutorials/mysql-load-balancing-haproxy-tutorial Have you seen the above in production? Yes of course. Just depends on the application and how high the workload gets. Please read about deadloks and nova in the following report by Intel: http://galeracluster.com/wp-content/uploads/2017/06/performance_analysis_and_tuning_in_china_mobiles_openstack_production_cloud_2.pdf If just Nova is affected we could also create an additional HAProxy listener using all Galera nodes with round-robin for all other services? Anyway - proxySQL would be a great extension. smime.p7s Description: S/MIME cryptographic signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints
Hi, I would like to start a discussion about some changes and additions I would like to see in in kolla and kolla-ansible. 1. Keepalived is a problem in layer3 spine leaf networks as any floating IP can only exist in one leaf (and VRRP is a problem in layer3). I would like to use consul and registrar to get rid of the "internal" floating IP and use consuls DNS service discovery to connect all services with each other. 2. Using "ports" for external API (endpoint) access is a major headache if a firewall is involved. I would like to configure the HAProxy (or fabio) for the external access to use "Host:" like, eg. "Host: keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with HTTPS. Any customer would just need HTTPS access and not have to open all those ports in his firewall. For some enterprise customers it is not possible to request FW changes like that. 3. HAProxy is not capable to handle "read/write" split with Galera. I would like to introduce ProxySQL to be able to scale Galera. 4. HAProxy is fine but fabio integrates well with consul, statsd and could be connected to a vault cluster to manage secure certificate access. 5. I would like to add vault as Barbican backend. 6. I would like to add an option to enable tokenless authentication for all services with each other to get rid of all the openstack service passwords (security issue). What do you think about it? All the best, Florian smime.p7s Description: S/MIME cryptographic signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [kolla] ceph osd deploy fails
Hi,
I tried to deploy Rocky in a multinode setup but ceph-osd fails with:
failed: [xxx-poc2] (item=[0, {u'fs_uuid': u'', u'bs_wal_label':
u'', u'external_journal': False, u'bs_blk_label': u'',
u'bs_db_partition_num': u'', u'journal_device': u'', u'journal': u'',
u'partition': u'/dev/nvme0n1', u'bs_wal_partition_num': u'',
u'fs_label': u'', u'journal_num': 0, u'bs_wal_device': u'',
u'partition_num': u'1', u'bs_db_label': u'', u'bs_blk_partition_num':
u'', u'device': u'/dev/nvme0n1', u'bs_db_device': u'',
u'partition_label': u'KOLLA_CEPH_OSD_BOOTSTRAP_BS', u'bs_blk_device':
u''}]) => {
"changed": true,
"item": [
0,
{
"bs_blk_device": "",
"bs_blk_label": "",
"bs_blk_partition_num": "",
"bs_db_device": "",
"bs_db_label": "",
"bs_db_partition_num": "",
"bs_wal_device": "",
"bs_wal_label": "",
"bs_wal_partition_num": "",
"device": "/dev/nvme0n1",
"external_journal": false,
"fs_label": "",
"fs_uuid": "",
"journal": "",
"journal_device": "",
"journal_num": 0,
"partition": "/dev/nvme0n1",
"partition_label": "KOLLA_CEPH_OSD_BOOTSTRAP_BS",
"partition_num": "1"
}
]
}
MSG:
Container exited with non-zero return code 2
We tried to debug the error message by starting the container with a
modified endpoint but we are stuck at the following point right now:
docker run -e "HOSTNAME=10.0.153.11" -e "JOURNAL_DEV=" -e
"JOURNAL_PARTITION=" -e "JOURNAL_PARTITION_NUM=0" -e
"KOLLA_BOOTSTRAP=null" -e "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" -e
"KOLLA_SERVICE_NAME=bootstrap-osd-0" -e "OSD_BS_BLK_DEV=" -e
"OSD_BS_BLK_LABEL=" -e "OSD_BS_BLK_PARTNUM=" -e "OSD_BS_DB_DEV=" -e
"OSD_BS_DB_LABEL=" -e "OSD_BS_DB_PARTNUM=" -e "OSD_BS_DEV=/dev/nvme0n1"
-e "OSD_BS_LABEL=KOLLA_CEPH_OSD_BOOTSTRAP_BS" -e "OSD_BS_PARTNUM=1" -e
"OSD_BS_WAL_DEV=" -e "OSD_BS_WAL_LABEL=" -e "OSD_BS_WAL_PARTNUM=" -e
"OSD_DEV=/dev/nvme0n1" -e "OSD_FILESYSTEM=xfs" -e "OSD_INITIAL_WEIGHT=1"
-e "OSD_PARTITION=/dev/nvme0n1" -e "OSD_PARTITION_NUM=1" -e
"OSD_STORETYPE=bluestore" -e "USE_EXTERNAL_JOURNAL=false" -v
"/etc/kolla//ceph-osd/:/var/lib/kolla/config_files/:ro" -v
"/etc/localtime:/etc/localtime:ro" -v "/dev/:/dev/" -v
"kolla_logs:/var/log/kolla/" -ti --privileged=true --entrypoint
/bin/bash
10.0.128.7:5000/openstack/openstack-kolla-cfg/ubuntu-source-ceph-osd:7.0.0.3
cat /var/lib/kolla/config_files/ceph.client.admin.keyring >
/etc/ceph/ceph.client.admin.keyring
cat /var/lib/kolla/config_files/ceph.conf > /etc/ceph/ceph.conf
(bootstrap-osd-0)[root@985e2dee22bc /]# /usr/bin/ceph-osd -d
--public-addr 10.0.153.11 --cluster-addr 10.0.153.11
usage: ceph-osd -i [flags]
--osd-data PATH data directory
--osd-journal PATH
journal file or block device
--mkfscreate a [new] data directory
--mkkey generate a new secret key. This is normally used in
combination with --mkfs
--convert-filestore
run any pending upgrade operations
--flush-journal flush all data out of journal
--mkjournal initialize a new journal
--check-wants-journal
check whether a journal is desired
--check-allows-journal
check whether a journal is allowed
--check-needs-journal
check whether a journal is required
--debug_osdset debug level (e.g. 10)
--get-device-fsid PATH
get OSD fsid for the given block device
--conf/-c FILEread configuration from the given configuration file
--id/-i IDset ID portion of my name
--name/-n TYPE.ID set name
--cluster NAMEset cluster name (default: ceph)
--setuser USERset uid to user or uid (and gid to user's gid)
--setgroup GROUP set gid to group or gid
--version show version and quit
-drun in foreground, log to stderr.
-frun in foreground, log to usual location.
--debug_ms N set message debug level (e.g. 1)
2018-09-26 12:28:07.801066 7fbda64b4e40 0 ceph version 12.2.4
(52085d5249a80c5f5121a76d6288429f35e4e77b) luminous
