subject:"\[openstack\-dev\] \[kolla\] add service discovery, proxysql, vault, fabio and FQDN endpoints"

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-19 Thread Piotr Misiak



On 19.10.2018 10:21, Florian Engelmann wrote:


On 17.10.2018 15:45, Florian Engelmann wrote:

On 10.10.2018 09:06, Florian Engelmann wrote:
Now I get you. I would say all configuration templates need to be 
changed to allow, eg.


$ grep http /etc/kolla/cinder-volume/cinder.conf
glance_api_servers = http://10.10.10.5:9292
auth_url = http://internal.somedomain.tld:35357
www_authenticate_uri = http://internal.somedomain.tld:5000
auth_url = http://internal.somedomain.tld:35357
auth_endpoint = http://internal.somedomain.tld:5000

to look like:

glance_api_servers = http://glance.service.somedomain.consul:9292
auth_url = http://keystone.service.somedomain.consul:35357
www_authenticate_uri = http://keystone.service.somedomain.consul:5000
auth_url = http://keystone.service.somedomain.consul:35357
auth_endpoint = http://keystone.service.somedomain.consul:5000



The idea with Consul looks interesting.

But I don't get your issue with VIP address and spine-leaf network.

What we have:
- controller1 behind leaf1 A/B pair with MLAG
- controller2 behind leaf2 A/B pair with MLAG
- controller3 behind leaf3 A/B pair with MLAG

The VIP address is active on one controller server.
When the server fail then the VIP will move to another controller 
server.

Where do you see a SPOF in this configuration?



So leaf1 2 and 3 have to share the same L2 domain, right (in IPv4 
network)?



Yes, they share L2 domain but we have ARP and ND suppression enabled.

It is an EVPN network where there is a L3 with VxLANs between leafs 
and spines.


So we don't care where a server is connected. It can be connected to 
any leaf.


Ok that sounds very interesting. Is it possible to share some 
internals? Which switch vendor/model do you use? How does you IP 
address schema look like?
If VxLAN is used between spine and leafs are you using VxLAN 
networking for Openstack as well? Where is your VTEP?




We have Mellanox switches with Cumulus Linux installed. Here you have a 
documentation: 
https://docs.cumulusnetworks.com/display/DOCS/Ethernet+Virtual+Private+Network+-+EVPN


EVPN is a well known standard and it is also supported by Juniper, Cisco 
etc.


We have standard VLANs between servers and leaf switches, they are 
mapped to VxLANs between leafs and spines. In our env every leaf switch 
is a VTEP. Servers have MLAG/CLAG connections to two leaf switches. We 
also have anycast gateways on leaf switches. From servers point of view 
our network is like a very big switch with hundreds of ports and 
standard VLANs.


We are using VxLAN networking for OpenStack, but it is configured on top 
of network VxLANs, we dont mix them.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-19 Thread Fox, Kevin M

Adding a stateless service provider on the existing etcd key value store would 
be pretty easy with something like coredns I think without adding another 
stateful storage dependency.

I don't really have a horse in the game other then I'm an operator and we're 
feeling overwhelmed by all the state stuff to maintain.

If consul is entirely optional, its probably fine to add the feature. But I 
worry operators may avoid it.

Thanks,
Kevin

From: Florian Engelmann [florian.engelm...@everyware.ch]
Sent: Friday, October 19, 2018 1:17 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, 
fabio and FQDN endpoints

> No, I mean, Consul would be an extra dependency in a big list of dependencies 
> OpenStack already has. OpenStack has so many it is causing operators to 
> reconsider adoption. I'm asking, if existing dependencies can be made to 
> solve the problem without adding more?
>
> Stateful dependencies are much harder to deal with then stateless ones, as 
> they take much more operator care/attention. Consul is stateful as is etcd, 
> and etcd is already a dependency.
>
> Can etcd be used instead so as not to put more load on the operators?

While etcd is a strong KV store it lacks many features consul has. Using
consul for DNS based service discovery is very easy to implement without
making it a dependency.
So we will start with a "external" consul and see how to handle the
service registration without modifying the kolla containers or any
kolla-ansible code.

All the best,
Flo


>
> Thanks,
> Kevin
> 
> From: Florian Engelmann [florian.engelm...@everyware.ch]
> Sent: Wednesday, October 10, 2018 12:18 AM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, 
> fabio and FQDN endpoints
>
> by "another storage system" you mean the KV store of consul? That's just
> someting consul brings with it...
>
> consul is very strong in doing health checks
>
> Am 10/9/18 um 6:09 PM schrieb Fox, Kevin M:
>> etcd is an already approved openstack dependency. Could that be used instead 
>> of consul so as to not add yet another storage system? coredns with the 
>> https://coredns.io/plugins/etcd/ plugin would maybe do what you need?
>>
>> Thanks,
>> Kevin
>> 
>> From: Florian Engelmann [florian.engelm...@everyware.ch]
>> Sent: Monday, October 08, 2018 3:14 AM
>> To: openstack-dev@lists.openstack.org
>> Subject: [openstack-dev] [kolla] add service discovery, proxysql, vault, 
>> fabio and FQDN endpoints
>>
>> Hi,
>>
>> I would like to start a discussion about some changes and additions I
>> would like to see in in kolla and kolla-ansible.
>>
>> 1. Keepalived is a problem in layer3 spine leaf networks as any floating
>> IP can only exist in one leaf (and VRRP is a problem in layer3). I would
>> like to use consul and registrar to get rid of the "internal" floating
>> IP and use consuls DNS service discovery to connect all services with
>> each other.
>>
>> 2. Using "ports" for external API (endpoint) access is a major headache
>> if a firewall is involved. I would like to configure the HAProxy (or
>> fabio) for the external access to use "Host:" like, eg. "Host:
>> keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with HTTPS.
>> Any customer would just need HTTPS access and not have to open all those
>> ports in his firewall. For some enterprise customers it is not possible
>> to request FW changes like that.
>>
>> 3. HAProxy is not capable to handle "read/write" split with Galera. I
>> would like to introduce ProxySQL to be able to scale Galera.
>>
>> 4. HAProxy is fine but fabio integrates well with consul, statsd and
>> could be connected to a vault cluster to manage secure certificate access.
>>
>> 5. I would like to add vault as Barbican backend.
>>
>> 6. I would like to add an option to enable tokenless authentication for
>> all services with each other to get rid of all the openstack service
>> passwords (security issue).
>>
>> What do you think about it?
>>
>> All the best,
>> Florian
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> -

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-19 Thread Florian Engelmann

currently we are testing what is needed to get consul + registrator and 
kolla/kolla-ansible play together nicely.


To get the services created in consul by registrator all kolla 
containers running relevant services (eg. keystone, nova, cinder, ... 
but also mariadb, memcached, es, ...) need to "--expose" their ports.

Registrator will use those "exposed" ports to add a service to consul.

I there any (existing) option to add those ports to the container 
bootstrap?

What about "docker_common_options"?

command should look like:

docker run -d --expose 5000/tcp --expose 35357/tcp --name=keystone ...



After testing registrator I recognized the project seems to be 
unmaintained. So we won't use registrator.


I just need to find another method to register a container (service) in 
consul after the contaier has started.


I would like to do so without changing any kolla container or 
kolla-ansible code.





Am 10/10/18 um 9:18 AM schrieb Florian Engelmann:
by "another storage system" you mean the KV store of consul? That's 
just someting consul brings with it...


consul is very strong in doing health checks

Am 10/9/18 um 6:09 PM schrieb Fox, Kevin M:
etcd is an already approved openstack dependency. Could that be used 
instead of consul so as to not add yet another storage system? 
coredns with the https://coredns.io/plugins/etcd/ plugin would maybe 
do what you need?


Thanks,
Kevin

From: Florian Engelmann [florian.engelm...@everyware.ch]
Sent: Monday, October 08, 2018 3:14 AM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [kolla] add service discovery, proxysql, 
vault, fabio and FQDN endpoints


Hi,

I would like to start a discussion about some changes and additions I
would like to see in in kolla and kolla-ansible.

1. Keepalived is a problem in layer3 spine leaf networks as any floating
IP can only exist in one leaf (and VRRP is a problem in layer3). I would
like to use consul and registrar to get rid of the "internal" floating
IP and use consuls DNS service discovery to connect all services with
each other.

2. Using "ports" for external API (endpoint) access is a major headache
if a firewall is involved. I would like to configure the HAProxy (or
fabio) for the external access to use "Host:" like, eg. "Host:
keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with HTTPS.
Any customer would just need HTTPS access and not have to open all those
ports in his firewall. For some enterprise customers it is not possible
to request FW changes like that.

3. HAProxy is not capable to handle "read/write" split with Galera. I
would like to introduce ProxySQL to be able to scale Galera.

4. HAProxy is fine but fabio integrates well with consul, statsd and
could be connected to a vault cluster to manage secure certificate 
access.


5. I would like to add vault as Barbican backend.

6. I would like to add an option to enable tokenless authentication for
all services with each other to get rid of all the openstack service
passwords (security issue).

What do you think about it?

All the best,
Florian

__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--

EveryWare AG
Florian Engelmann
Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: mailto:florian.engelm...@everyware.ch
web: http://www.everyware.ch


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-19 Thread Florian Engelmann



On 17.10.2018 15:45, Florian Engelmann wrote:

On 10.10.2018 09:06, Florian Engelmann wrote:
Now I get you. I would say all configuration templates need to be 
changed to allow, eg.


$ grep http /etc/kolla/cinder-volume/cinder.conf
glance_api_servers = http://10.10.10.5:9292
auth_url = http://internal.somedomain.tld:35357
www_authenticate_uri = http://internal.somedomain.tld:5000
auth_url = http://internal.somedomain.tld:35357
auth_endpoint = http://internal.somedomain.tld:5000

to look like:

glance_api_servers = http://glance.service.somedomain.consul:9292
auth_url = http://keystone.service.somedomain.consul:35357
www_authenticate_uri = http://keystone.service.somedomain.consul:5000
auth_url = http://keystone.service.somedomain.consul:35357
auth_endpoint = http://keystone.service.somedomain.consul:5000



The idea with Consul looks interesting.

But I don't get your issue with VIP address and spine-leaf network.

What we have:
- controller1 behind leaf1 A/B pair with MLAG
- controller2 behind leaf2 A/B pair with MLAG
- controller3 behind leaf3 A/B pair with MLAG

The VIP address is active on one controller server.
When the server fail then the VIP will move to another controller 
server.

Where do you see a SPOF in this configuration?



So leaf1 2 and 3 have to share the same L2 domain, right (in IPv4 
network)?



Yes, they share L2 domain but we have ARP and ND suppression enabled.

It is an EVPN network where there is a L3 with VxLANs between leafs and 
spines.


So we don't care where a server is connected. It can be connected to any 
leaf.


Ok that sounds very interesting. Is it possible to share some internals? 
Which switch vendor/model do you use? How does you IP address schema 
look like?
If VxLAN is used between spine and leafs are you using VxLAN networking 
for Openstack as well? Where is your VTEP?






But we wanna deploy a layer3 spine-leaf network were every leaf is 
it's own L2 domain and everything above is layer3.


eg:

leaf1 = 10.1.1.0/24
leaf2 = 10.1.2.0/24
leaf2 = 10.1.3.0/24

So a VIP like, eg. 10.1.1.10 could only exist in leaf1


In my opinion it's a very constrained environment, I don't like the idea.


Regards,

Piotr



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--

EveryWare AG
Florian Engelmann
Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: mailto:florian.engelm...@everyware.ch
web: http://www.everyware.ch


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-19 Thread Florian Engelmann




No, I mean, Consul would be an extra dependency in a big list of dependencies 
OpenStack already has. OpenStack has so many it is causing operators to 
reconsider adoption. I'm asking, if existing dependencies can be made to solve 
the problem without adding more?

Stateful dependencies are much harder to deal with then stateless ones, as they 
take much more operator care/attention. Consul is stateful as is etcd, and etcd 
is already a dependency.

Can etcd be used instead so as not to put more load on the operators?


While etcd is a strong KV store it lacks many features consul has. Using 
consul for DNS based service discovery is very easy to implement without 
making it a dependency.
So we will start with a "external" consul and see how to handle the 
service registration without modifying the kolla containers or any 
kolla-ansible code.


All the best,
Flo




Thanks,
Kevin

From: Florian Engelmann [florian.engelm...@everyware.ch]
Sent: Wednesday, October 10, 2018 12:18 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, 
fabio and FQDN endpoints

by "another storage system" you mean the KV store of consul? That's just
someting consul brings with it...

consul is very strong in doing health checks

Am 10/9/18 um 6:09 PM schrieb Fox, Kevin M:

etcd is an already approved openstack dependency. Could that be used instead of 
consul so as to not add yet another storage system? coredns with the 
https://coredns.io/plugins/etcd/ plugin would maybe do what you need?

Thanks,
Kevin

From: Florian Engelmann [florian.engelm...@everyware.ch]
Sent: Monday, October 08, 2018 3:14 AM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio 
and FQDN endpoints

Hi,

I would like to start a discussion about some changes and additions I
would like to see in in kolla and kolla-ansible.

1. Keepalived is a problem in layer3 spine leaf networks as any floating
IP can only exist in one leaf (and VRRP is a problem in layer3). I would
like to use consul and registrar to get rid of the "internal" floating
IP and use consuls DNS service discovery to connect all services with
each other.

2. Using "ports" for external API (endpoint) access is a major headache
if a firewall is involved. I would like to configure the HAProxy (or
fabio) for the external access to use "Host:" like, eg. "Host:
keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with HTTPS.
Any customer would just need HTTPS access and not have to open all those
ports in his firewall. For some enterprise customers it is not possible
to request FW changes like that.

3. HAProxy is not capable to handle "read/write" split with Galera. I
would like to introduce ProxySQL to be able to scale Galera.

4. HAProxy is fine but fabio integrates well with consul, statsd and
could be connected to a vault cluster to manage secure certificate access.

5. I would like to add vault as Barbican backend.

6. I would like to add an option to enable tokenless authentication for
all services with each other to get rid of all the openstack service
passwords (security issue).

What do you think about it?

All the best,
Florian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--

EveryWare AG
Florian Engelmann
Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: mailto:florian.engelm...@everyware.ch
web: http://www.everyware.ch

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--

EveryWare AG
Florian Engelmann
Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: mailto:florian.engelm...@everyware.ch
web: http://www.everyware.ch


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-17 Thread Piotr Misiak



On 17.10.2018 15:45, Florian Engelmann wrote:

On 10.10.2018 09:06, Florian Engelmann wrote:
Now I get you. I would say all configuration templates need to be 
changed to allow, eg.


$ grep http /etc/kolla/cinder-volume/cinder.conf
glance_api_servers = http://10.10.10.5:9292
auth_url = http://internal.somedomain.tld:35357
www_authenticate_uri = http://internal.somedomain.tld:5000
auth_url = http://internal.somedomain.tld:35357
auth_endpoint = http://internal.somedomain.tld:5000

to look like:

glance_api_servers = http://glance.service.somedomain.consul:9292
auth_url = http://keystone.service.somedomain.consul:35357
www_authenticate_uri = http://keystone.service.somedomain.consul:5000
auth_url = http://keystone.service.somedomain.consul:35357
auth_endpoint = http://keystone.service.somedomain.consul:5000



The idea with Consul looks interesting.

But I don't get your issue with VIP address and spine-leaf network.

What we have:
- controller1 behind leaf1 A/B pair with MLAG
- controller2 behind leaf2 A/B pair with MLAG
- controller3 behind leaf3 A/B pair with MLAG

The VIP address is active on one controller server.
When the server fail then the VIP will move to another controller 
server.

Where do you see a SPOF in this configuration?



So leaf1 2 and 3 have to share the same L2 domain, right (in IPv4 
network)?



Yes, they share L2 domain but we have ARP and ND suppression enabled.

It is an EVPN network where there is a L3 with VxLANs between leafs and 
spines.


So we don't care where a server is connected. It can be connected to any 
leaf.



But we wanna deploy a layer3 spine-leaf network were every leaf is 
it's own L2 domain and everything above is layer3.


eg:

leaf1 = 10.1.1.0/24
leaf2 = 10.1.2.0/24
leaf2 = 10.1.3.0/24

So a VIP like, eg. 10.1.1.10 could only exist in leaf1


In my opinion it's a very constrained environment, I don't like the idea.


Regards,

Piotr



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-17 Thread Fox, Kevin M

No, I mean, Consul would be an extra dependency in a big list of dependencies 
OpenStack already has. OpenStack has so many it is causing operators to 
reconsider adoption. I'm asking, if existing dependencies can be made to solve 
the problem without adding more?

Stateful dependencies are much harder to deal with then stateless ones, as they 
take much more operator care/attention. Consul is stateful as is etcd, and etcd 
is already a dependency.

Can etcd be used instead so as not to put more load on the operators?

Thanks,
Kevin

From: Florian Engelmann [florian.engelm...@everyware.ch]
Sent: Wednesday, October 10, 2018 12:18 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, 
fabio and FQDN endpoints

by "another storage system" you mean the KV store of consul? That's just
someting consul brings with it...

consul is very strong in doing health checks

Am 10/9/18 um 6:09 PM schrieb Fox, Kevin M:
> etcd is an already approved openstack dependency. Could that be used instead 
> of consul so as to not add yet another storage system? coredns with the 
> https://coredns.io/plugins/etcd/ plugin would maybe do what you need?
>
> Thanks,
> Kevin
> 
> From: Florian Engelmann [florian.engelm...@everyware.ch]
> Sent: Monday, October 08, 2018 3:14 AM
> To: openstack-dev@lists.openstack.org
> Subject: [openstack-dev] [kolla] add service discovery, proxysql, vault, 
> fabio and FQDN endpoints
>
> Hi,
>
> I would like to start a discussion about some changes and additions I
> would like to see in in kolla and kolla-ansible.
>
> 1. Keepalived is a problem in layer3 spine leaf networks as any floating
> IP can only exist in one leaf (and VRRP is a problem in layer3). I would
> like to use consul and registrar to get rid of the "internal" floating
> IP and use consuls DNS service discovery to connect all services with
> each other.
>
> 2. Using "ports" for external API (endpoint) access is a major headache
> if a firewall is involved. I would like to configure the HAProxy (or
> fabio) for the external access to use "Host:" like, eg. "Host:
> keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with HTTPS.
> Any customer would just need HTTPS access and not have to open all those
> ports in his firewall. For some enterprise customers it is not possible
> to request FW changes like that.
>
> 3. HAProxy is not capable to handle "read/write" split with Galera. I
> would like to introduce ProxySQL to be able to scale Galera.
>
> 4. HAProxy is fine but fabio integrates well with consul, statsd and
> could be connected to a vault cluster to manage secure certificate access.
>
> 5. I would like to add vault as Barbican backend.
>
> 6. I would like to add an option to enable tokenless authentication for
> all services with each other to get rid of all the openstack service
> passwords (security issue).
>
> What do you think about it?
>
> All the best,
> Florian
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

--

EveryWare AG
Florian Engelmann
Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: mailto:florian.engelm...@everyware.ch
web: http://www.everyware.ch

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-17 Thread Florian Engelmann

currently we are testing what is needed to get consul + registrator and 
kolla/kolla-ansible play together nicely.


To get the services created in consul by registrator all kolla 
containers running relevant services (eg. keystone, nova, cinder, ... 
but also mariadb, memcached, es, ...) need to "--expose" their ports.

Registrator will use those "exposed" ports to add a service to consul.

I there any (existing) option to add those ports to the container bootstrap?
What about "docker_common_options"?

command should look like:

docker run -d --expose 5000/tcp --expose 35357/tcp --name=keystone ...


Am 10/10/18 um 9:18 AM schrieb Florian Engelmann:
by "another storage system" you mean the KV store of consul? That's just 
someting consul brings with it...


consul is very strong in doing health checks

Am 10/9/18 um 6:09 PM schrieb Fox, Kevin M:
etcd is an already approved openstack dependency. Could that be used 
instead of consul so as to not add yet another storage system? coredns 
with the https://coredns.io/plugins/etcd/ plugin would maybe do what 
you need?


Thanks,
Kevin

From: Florian Engelmann [florian.engelm...@everyware.ch]
Sent: Monday, October 08, 2018 3:14 AM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [kolla] add service discovery, proxysql, 
vault, fabio and FQDN endpoints


Hi,

I would like to start a discussion about some changes and additions I
would like to see in in kolla and kolla-ansible.

1. Keepalived is a problem in layer3 spine leaf networks as any floating
IP can only exist in one leaf (and VRRP is a problem in layer3). I would
like to use consul and registrar to get rid of the "internal" floating
IP and use consuls DNS service discovery to connect all services with
each other.

2. Using "ports" for external API (endpoint) access is a major headache
if a firewall is involved. I would like to configure the HAProxy (or
fabio) for the external access to use "Host:" like, eg. "Host:
keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with HTTPS.
Any customer would just need HTTPS access and not have to open all those
ports in his firewall. For some enterprise customers it is not possible
to request FW changes like that.

3. HAProxy is not capable to handle "read/write" split with Galera. I
would like to introduce ProxySQL to be able to scale Galera.

4. HAProxy is fine but fabio integrates well with consul, statsd and
could be connected to a vault cluster to manage secure certificate 
access.


5. I would like to add vault as Barbican backend.

6. I would like to add an option to enable tokenless authentication for
all services with each other to get rid of all the openstack service
passwords (security issue).

What do you think about it?

All the best,
Florian

__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--

EveryWare AG
Florian Engelmann
Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: mailto:florian.engelm...@everyware.ch
web: http://www.everyware.ch


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-17 Thread Florian Engelmann


On 10.10.2018 09:06, Florian Engelmann wrote:
Now I get you. I would say all configuration templates need to be 
changed to allow, eg.


$ grep http /etc/kolla/cinder-volume/cinder.conf
glance_api_servers = http://10.10.10.5:9292
auth_url = http://internal.somedomain.tld:35357
www_authenticate_uri = http://internal.somedomain.tld:5000
auth_url = http://internal.somedomain.tld:35357
auth_endpoint = http://internal.somedomain.tld:5000

to look like:

glance_api_servers = http://glance.service.somedomain.consul:9292
auth_url = http://keystone.service.somedomain.consul:35357
www_authenticate_uri = http://keystone.service.somedomain.consul:5000
auth_url = http://keystone.service.somedomain.consul:35357
auth_endpoint = http://keystone.service.somedomain.consul:5000



The idea with Consul looks interesting.

But I don't get your issue with VIP address and spine-leaf network.

What we have:
- controller1 behind leaf1 A/B pair with MLAG
- controller2 behind leaf2 A/B pair with MLAG
- controller3 behind leaf3 A/B pair with MLAG

The VIP address is active on one controller server.
When the server fail then the VIP will move to another controller server.
Where do you see a SPOF in this configuration?



So leaf1 2 and 3 have to share the same L2 domain, right (in IPv4 network)?

But we wanna deploy a layer3 spine-leaf network were every leaf is it's 
own L2 domain and everything above is layer3.


eg:

leaf1 = 10.1.1.0/24
leaf2 = 10.1.2.0/24
leaf2 = 10.1.3.0/24

So a VIP like, eg. 10.1.1.10 could only exist in leaf1



smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-10 Thread Piotr Misiak


On 10.10.2018 09:06, Florian Engelmann wrote:
Now I get you. I would say all configuration templates need to be 
changed to allow, eg.


$ grep http /etc/kolla/cinder-volume/cinder.conf
glance_api_servers = http://10.10.10.5:9292
auth_url = http://internal.somedomain.tld:35357
www_authenticate_uri = http://internal.somedomain.tld:5000
auth_url = http://internal.somedomain.tld:35357
auth_endpoint = http://internal.somedomain.tld:5000

to look like:

glance_api_servers = http://glance.service.somedomain.consul:9292
auth_url = http://keystone.service.somedomain.consul:35357
www_authenticate_uri = http://keystone.service.somedomain.consul:5000
auth_url = http://keystone.service.somedomain.consul:35357
auth_endpoint = http://keystone.service.somedomain.consul:5000



The idea with Consul looks interesting.

But I don't get your issue with VIP address and spine-leaf network.

What we have:
- controller1 behind leaf1 A/B pair with MLAG
- controller2 behind leaf2 A/B pair with MLAG
- controller3 behind leaf3 A/B pair with MLAG

The VIP address is active on one controller server.
When the server fail then the VIP will move to another controller server.
Where do you see a SPOF in this configuration?

Thanks


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-10 Thread Fox, Kevin M

Sorry. Couldn't quite think of the name. I was meaning, openstack project tags.

Thanks,
Kevin

From: Jay Pipes [jaypi...@gmail.com]
Sent: Tuesday, October 09, 2018 12:22 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, 
fabio and FQDN endpoints

On 10/09/2018 03:10 PM, Fox, Kevin M wrote:
> Oh, this does raise an interesting question... Should such information be 
> reported by the projects up to users through labels? Something like, 
> "percona_multimaster=safe" Its really difficult for folks to know which 
> projects can and can not be used that way currently.

Are you referring to k8s labels/selectors? or are you referring to
project tags (you know, part of that whole Big Tent thing...)?

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-10 Thread Mark Goddard

On Wed, 10 Oct 2018 at 08:08, Florian Engelmann <
florian.engelm...@everyware.ch> wrote:

> Am 10/9/18 um 1:47 PM schrieb Mark Goddard:
> >
> >
> > On Tue, 9 Oct 2018 at 12:03, Florian Engelmann
> > mailto:florian.engelm...@everyware.ch>>
>
> > wrote:
> >
> > Am 10/9/18 um 11:04 AM schrieb Mark Goddard:
> >  > Thanks for these suggestions Florian, there are some interesting
> > ideas
> >  > in here. I'm a little concerned about the maintenance overhead of
> > adding
> >  > support for all of these things, and wonder if some of them could
> be
> >  > done without explicit support in kolla and kolla-ansible. The
> kolla
> >  > projects have been able to move quickly by providing a flexible
> >  > configuration mechanism that avoids the need to maintain support
> for
> >  > every OpenStack feature. Other thoughts inline.
> >  >
> >
> > I do understand your apprehensions Mark. For some of the suggested
> > changes/additions I agree. But adding those components without
> > kolla/kolla-ansible integration feels not right.
> >
> >
> > I'm not entirely against adding some of these things, if enough people
> > in the community want them. I'd just like to make sure that if they can
> > be done in a sane way without changes, then we do that and document how
> > to do it instead.
> >
>
> Yes I agree and that is a very important role/job.
>
> >
> >  >
> >  > On Mon, 8 Oct 2018 at 11:15, Florian Engelmann
> >  >  > 
> >  > >>
> >  > wrote:
> >  >
> >  > Hi,
> >  >
> >  > I would like to start a discussion about some changes and
> > additions I
> >  > would like to see in in kolla and kolla-ansible.
> >  >
> >  > 1. Keepalived is a problem in layer3 spine leaf networks as
> any
> >  > floating
> >  > IP can only exist in one leaf (and VRRP is a problem in
> > layer3). I
> >  > would
> >  > like to use consul and registrar to get rid of the "internal"
> > floating
> >  > IP and use consuls DNS service discovery to connect all
> > services with
> >  > each other.
> >  >
> >  >
> >  > Without reading up, I'm not sure exactly how this fits together.
> If
> >  > kolla-ansible made the API host configurable for each service
> rather
> >  > than globally, would that be a step in the right direction?
> >
> > No that would not help. The problem is HA. Right now there is a
> > "central" floating IP (kolla_internal_vip_address) that is used for
> all
> > services to connect to (each other). Keepalived is failing that IP
> over
> > if the "active" host fails. In a layer3 (CLOS/Spine-Leaf) network
> this
> > IP is only available in one leaf/rack. So that rack is a "SPOF".
> > Using service discovery fits perfect in a CLOS network and scales
> much
> > better as a HA solution.
> >
> > Right, but what I'm saying as a thought experiment is, if we gave you
> > the required variables in kolla-ansible (e.g. nova_internal_fqdn) to
> > make this possible with an externally managed consul service, could that
> > work?
>
> Now I get you. I would say all configuration templates need to be
> changed to allow, eg.
>
> $ grep http /etc/kolla/cinder-volume/cinder.conf
> glance_api_servers = http://10.10.10.5:9292
> auth_url = http://internal.somedomain.tld:35357
> www_authenticate_uri = http://internal.somedomain.tld:5000
> auth_url = http://internal.somedomain.tld:35357
> auth_endpoint = http://internal.somedomain.tld:5000
>
> to look like:
>
> glance_api_servers = http://glance.service.somedomain.consul:9292
> auth_url = http://keystone.service.somedomain.consul:35357
> www_authenticate_uri = http://keystone.service.somedomain.consul:5000
> auth_url = http://keystone.service.somedomain.consul:35357
> auth_endpoint = http://keystone.service.somedomain.consul:5000


Those values are generally formed using the internal FQDN
(kolla_internal_fqdn), that is supposed to resolve to the internal VIP.

If we had something like this in group_vars/all.yml, we could make the host
configurable on a per-service basis, while still retaining the default.
This is a common pattern in kolla-ansible. For example:

nova_external_fqdn: "{{ kolla_external_fqdn }}"
nova_internal_fqdn: "{{ kolla_internal_fqdn }}"


> >
> >  >
> >  >
> >  > 2. Using "ports" for external API (endpoint) access is a
> > major headache
> >  > if a firewall is involved. I would like to configure the
> > HAProxy (or
> >  > fabio) for the external access to use "Host:" like, eg. "Host:
> >  > keystone.somedomain.tld", "Host: nova.somedomain.tld", ...
> > with HTTPS.
> >  > Any customer would just need HTTPS access and not have to
> > open all
> >  > those
> >  >

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-10 Thread Florian Engelmann

by "another storage system" you mean the KV store of consul? That's just 
someting consul brings with it...


consul is very strong in doing health checks

Am 10/9/18 um 6:09 PM schrieb Fox, Kevin M:

etcd is an already approved openstack dependency. Could that be used instead of 
consul so as to not add yet another storage system? coredns with the 
https://coredns.io/plugins/etcd/ plugin would maybe do what you need?

Thanks,
Kevin

From: Florian Engelmann [florian.engelm...@everyware.ch]
Sent: Monday, October 08, 2018 3:14 AM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio 
and FQDN endpoints

Hi,

I would like to start a discussion about some changes and additions I
would like to see in in kolla and kolla-ansible.

1. Keepalived is a problem in layer3 spine leaf networks as any floating
IP can only exist in one leaf (and VRRP is a problem in layer3). I would
like to use consul and registrar to get rid of the "internal" floating
IP and use consuls DNS service discovery to connect all services with
each other.

2. Using "ports" for external API (endpoint) access is a major headache
if a firewall is involved. I would like to configure the HAProxy (or
fabio) for the external access to use "Host:" like, eg. "Host:
keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with HTTPS.
Any customer would just need HTTPS access and not have to open all those
ports in his firewall. For some enterprise customers it is not possible
to request FW changes like that.

3. HAProxy is not capable to handle "read/write" split with Galera. I
would like to introduce ProxySQL to be able to scale Galera.

4. HAProxy is fine but fabio integrates well with consul, statsd and
could be connected to a vault cluster to manage secure certificate access.

5. I would like to add vault as Barbican backend.

6. I would like to add an option to enable tokenless authentication for
all services with each other to get rid of all the openstack service
passwords (security issue).

What do you think about it?

All the best,
Florian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--

EveryWare AG
Florian Engelmann
Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: mailto:florian.engelm...@everyware.ch
web: http://www.everyware.ch


smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-10 Thread Florian Engelmann

Am 10/9/18 um 1:47 PM schrieb Mark Goddard:

On Tue, 9 Oct 2018 at 12:03, Florian Engelmann 
mailto:florian.engelm...@everyware.ch>> 
wrote:

Am 10/9/18 um 11:04 AM schrieb Mark Goddard:
 > Thanks for these suggestions Florian, there are some interesting
ideas
 > in here. I'm a little concerned about the maintenance overhead of
adding
 > support for all of these things, and wonder if some of them could be
 > done without explicit support in kolla and kolla-ansible. The kolla
 > projects have been able to move quickly by providing a flexible
 > configuration mechanism that avoids the need to maintain support for
 > every OpenStack feature. Other thoughts inline.
 >

I do understand your apprehensions Mark. For some of the suggested
changes/additions I agree. But adding those components without
kolla/kolla-ansible integration feels not right.

I'm not entirely against adding some of these things, if enough people 
in the community want them. I'd just like to make sure that if they can 
be done in a sane way without changes, then we do that and document how 
to do it instead.

Yes I agree and that is a very important role/job.

 >
 > On Mon, 8 Oct 2018 at 11:15, Florian Engelmann
 > mailto:florian.engelm...@everyware.ch>
>>
 > wrote:
 >
 >     Hi,
 >
 >     I would like to start a discussion about some changes and
additions I
 >     would like to see in in kolla and kolla-ansible.
 >
 >     1. Keepalived is a problem in layer3 spine leaf networks as any
 >     floating
 >     IP can only exist in one leaf (and VRRP is a problem in
layer3). I
 >     would
 >     like to use consul and registrar to get rid of the "internal"
floating
 >     IP and use consuls DNS service discovery to connect all
services with
 >     each other.
 >
 >
 > Without reading up, I'm not sure exactly how this fits together. If
 > kolla-ansible made the API host configurable for each service rather
 > than globally, would that be a step in the right direction?

No that would not help. The problem is HA. Right now there is a
"central" floating IP (kolla_internal_vip_address) that is used for all
services to connect to (each other). Keepalived is failing that IP over
if the "active" host fails. In a layer3 (CLOS/Spine-Leaf) network this
IP is only available in one leaf/rack. So that rack is a "SPOF".
Using service discovery fits perfect in a CLOS network and scales much
better as a HA solution.

Right, but what I'm saying as a thought experiment is, if we gave you 
the required variables in kolla-ansible (e.g. nova_internal_fqdn) to 
make this possible with an externally managed consul service, could that 
work?

Now I get you. I would say all configuration templates need to be 
changed to allow, eg.

$ grep http /etc/kolla/cinder-volume/cinder.conf
glance_api_servers = http://10.10.10.5:9292
auth_url = http://internal.somedomain.tld:35357
www_authenticate_uri = http://internal.somedomain.tld:5000
auth_url = http://internal.somedomain.tld:35357
auth_endpoint = http://internal.somedomain.tld:5000

to look like:

glance_api_servers = http://glance.service.somedomain.consul:9292
auth_url = http://keystone.service.somedomain.consul:35357
www_authenticate_uri = http://keystone.service.somedomain.consul:5000
auth_url = http://keystone.service.somedomain.consul:35357
auth_endpoint = http://keystone.service.somedomain.consul:5000

 >
 >
 >     2. Using "ports" for external API (endpoint) access is a
major headache
 >     if a firewall is involved. I would like to configure the
HAProxy (or
 >     fabio) for the external access to use "Host:" like, eg. "Host:
 >     keystone.somedomain.tld", "Host: nova.somedomain.tld", ...
with HTTPS.
 >     Any customer would just need HTTPS access and not have to
open all
 >     those
 >     ports in his firewall. For some enterprise customers it is
not possible
 >     to request FW changes like that.
 >
 >     3. HAProxy is not capable to handle "read/write" split with
Galera. I
 >     would like to introduce ProxySQL to be able to scale Galera.
 >
 >
 > It's now possible to use an external database server with
kolla-ansible,
 > instead of deploying a mariadb/galera cluster. This could be
implemented
 > how you like, see
 >

https://docs.openstack.org/kolla-ansible/latest/reference/external-mariadb-guide.html.

Yes I agree. And this is what we will do in our first production
deployment. But I would love to see ProxySQL in Kolla as well.
As a side note: Kolla-ansible does use:

option mysql-check user haproxy post-41

to check Galera, but that check does not fail if the node is out of
sync

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-09 Thread Jay Pipes


On 10/09/2018 03:10 PM, Fox, Kevin M wrote:

Oh, this does raise an interesting question... Should such information be reported by the 
projects up to users through labels? Something like, "percona_multimaster=safe" 
Its really difficult for folks to know which projects can and can not be used that way 
currently.


Are you referring to k8s labels/selectors? or are you referring to 
project tags (you know, part of that whole Big Tent thing...)?


-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-09 Thread Fox, Kevin M

Oh, this does raise an interesting question... Should such information be 
reported by the projects up to users through labels? Something like, 
"percona_multimaster=safe" Its really difficult for folks to know which 
projects can and can not be used that way currently.

Is this a TC question?

Thanks,
Kevin

From: melanie witt [melwi...@gmail.com]
Sent: Tuesday, October 09, 2018 10:35 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, 
fabio and FQDN endpoints

On Tue, 9 Oct 2018 07:23:03 -0400, Jay Pipes wrote:
> That explains where the source of the problem comes from (it's the use
> of SELECT FOR UPDATE, which has been removed from Nova's quota-handling
> code in the Rocky release).

Small correction, the SELECT FOR UPDATE was removed from Nova's
quota-handling code in the Pike release.

-melanie

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-09 Thread melanie witt


On Tue, 9 Oct 2018 10:35:23 -0700, Melanie Witt wrote:

On Tue, 9 Oct 2018 07:23:03 -0400, Jay Pipes wrote:

That explains where the source of the problem comes from (it's the use
of SELECT FOR UPDATE, which has been removed from Nova's quota-handling
code in the Rocky release).

Small correction, the SELECT FOR UPDATE was removed from Nova's
quota-handling code in the Pike release.


Elaboration: the calls to quota reserve/commit/rollback were removed in 
the Pike release, so with_lockmode('update') is not called for quota 
operations, even though the reserve/commit/rollback methods are still 
there for use by old (Ocata) computes during an Ocata => Pike upgrade. 
Then, the reserve/commit/rollback methods were removed in Queens once no 
old computes could be calling them.


-melanie




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-09 Thread melanie witt


On Tue, 9 Oct 2018 07:23:03 -0400, Jay Pipes wrote:

That explains where the source of the problem comes from (it's the use
of SELECT FOR UPDATE, which has been removed from Nova's quota-handling
code in the Rocky release).


Small correction, the SELECT FOR UPDATE was removed from Nova's 
quota-handling code in the Pike release.


-melanie




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-09 Thread Fox, Kevin M

etcd is an already approved openstack dependency. Could that be used instead of 
consul so as to not add yet another storage system? coredns with the 
https://coredns.io/plugins/etcd/ plugin would maybe do what you need?

Thanks,
Kevin

From: Florian Engelmann [florian.engelm...@everyware.ch]
Sent: Monday, October 08, 2018 3:14 AM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio 
and FQDN endpoints

Hi,

I would like to start a discussion about some changes and additions I
would like to see in in kolla and kolla-ansible.

1. Keepalived is a problem in layer3 spine leaf networks as any floating
IP can only exist in one leaf (and VRRP is a problem in layer3). I would
like to use consul and registrar to get rid of the "internal" floating
IP and use consuls DNS service discovery to connect all services with
each other.

2. Using "ports" for external API (endpoint) access is a major headache
if a firewall is involved. I would like to configure the HAProxy (or
fabio) for the external access to use "Host:" like, eg. "Host:
keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with HTTPS.
Any customer would just need HTTPS access and not have to open all those
ports in his firewall. For some enterprise customers it is not possible
to request FW changes like that.

3. HAProxy is not capable to handle "read/write" split with Galera. I
would like to introduce ProxySQL to be able to scale Galera.

4. HAProxy is fine but fabio integrates well with consul, statsd and
could be connected to a vault cluster to manage secure certificate access.

5. I would like to add vault as Barbican backend.

6. I would like to add an option to enable tokenless authentication for
all services with each other to get rid of all the openstack service
passwords (security issue).

What do you think about it?

All the best,
Florian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-09 Thread Fox, Kevin M

There are specific cases where it expects the client to retry and not all code 
tests for that case. Its safe funneling all traffic to one server. It can be 
unsafe to do so otherwise.

Thanks,
Kevin

From: Jay Pipes [jaypi...@gmail.com]
Sent: Monday, October 08, 2018 10:48 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, 
fabio and FQDN endpoints

On 10/08/2018 06:14 AM, Florian Engelmann wrote:
> 3. HAProxy is not capable to handle "read/write" split with Galera. I
> would like to introduce ProxySQL to be able to scale Galera.

Why not send all read and all write traffic to a single haproxy endpoint
and just have haproxy spread all traffic across each Galera node?

Galera, after all, is multi-master synchronous replication... so it
shouldn't matter which node in the Galera cluster you send traffic to.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-09 Thread Florian Engelmann

Am 10/9/18 um 1:23 PM schrieb Jay Pipes:

On 10/09/2018 06:34 AM, Florian Engelmann wrote:

Am 10/9/18 um 11:41 AM schrieb Jay Pipes:

On 10/09/2018 04:34 AM, Christian Berendt wrote:

On 8. Oct 2018, at 19:48, Jay Pipes wrote:

Why not send all read and all write traffic to a single haproxy
endpoint and just have haproxy spread all traffic across each
Galera node?

Galera, after all, is multi-master synchronous replication... so it
shouldn't matter which node in the Galera cluster you send traffic to.

Probably because of MySQL deadlocks in Galera:

—snip—
Galera cluster has known limitations, one of them is that it uses
cluster-wide optimistic locking. This may cause some transactions to
rollback. With an increasing number of writeable masters, the
transaction rollback rate may increase, especially if there is write
contention on the same dataset. It is of course possible to retry
the transaction and perhaps it will COMMIT in the retries, but this
will add to the transaction latency. However, some designs are
deadlock prone, e.g sequence tables.

—snap—

Source:
https://severalnines.com/resources/tutorials/mysql-load-balancing-haproxy-tutorial

Have you seen the above in production?

Yes of course. Just depends on the application and how high the
workload gets.

Please read about deadloks and nova in the following report by Intel:

http://galeracluster.com/wp-content/uploads/2017/06/performance_analysis_and_tuning_in_china_mobiles_openstack_production_cloud_2.pdf

I have read the above. It's a synthetic workload analysis, which is why
I asked if you'd seen this in production.

For the record, we addressed much of the contention/races mentioned in
the above around scheduler resource consumption in the Ocata and Pike
releases of Nova.

I'm aware that the report above identifies the quota handling code in
Nova as the primary culprit of the deadlock issues but again, it's a
synthetic workload that is designed to find breaking points. It doesn't
represent a realistic production workload.

You can read about the deadlock issue in depth on my blog here:

http://www.joinfu.com/2015/01/understanding-reservations-concurrency-locking-in-nova/

That explains where the source of the problem comes from (it's the use
of SELECT FOR UPDATE, which has been removed from Nova's quota-handling
code in the Rocky release).

Thank you very much for your link. Great article!!! Took my a while to
read it and understand everything but helped a lot!!!

If just Nova is affected we could also create an additional HAProxy
listener using all Galera nodes with round-robin for all other services?

I fail to see the point of using Galera with a single writer. At that
point, why bother with Galera at all? Just use a single database node
with a single slave for backup purposes.

From my point of view Galera is easy to manage and great for HA. Having
to handle a manual failover in production with mysql master/slave never
was fun...
Indeed writing to a single node and not using the other nodes (even for
read, like it is done in kolla-ansible) is not the best solution. Galera
is slower than a standalone MySQL.
Using ProxySQL would enable us to use caching and read/write split to
speed up database queries while HA and management are still good.

Anyway - proxySQL would be a great extension.

I don't disagree that proxySQL is a good extension. However, it adds yet
another services to the mesh that needs to be deployed, configured and
maintained.

True. I guess we will start with an external MySQL installation to
collect some experience.

smime.p7s
Description: S/MIME cryptographic signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-09 Thread Mark Goddard

On Tue, 9 Oct 2018 at 12:03, Florian Engelmann <
florian.engelm...@everyware.ch> wrote:

> Am 10/9/18 um 11:04 AM schrieb Mark Goddard:
> > Thanks for these suggestions Florian, there are some interesting ideas
> > in here. I'm a little concerned about the maintenance overhead of adding
> > support for all of these things, and wonder if some of them could be
> > done without explicit support in kolla and kolla-ansible. The kolla
> > projects have been able to move quickly by providing a flexible
> > configuration mechanism that avoids the need to maintain support for
> > every OpenStack feature. Other thoughts inline.
> >
>
> I do understand your apprehensions Mark. For some of the suggested
> changes/additions I agree. But adding those components without
> kolla/kolla-ansible integration feels not right.
>

I'm not entirely against adding some of these things, if enough people in
the community want them. I'd just like to make sure that if they can be
done in a sane way without changes, then we do that and document how to do
it instead.

>
> >
> > On Mon, 8 Oct 2018 at 11:15, Florian Engelmann
> > mailto:florian.engelm...@everyware.ch>>
>
> > wrote:
> >
> > Hi,
> >
> > I would like to start a discussion about some changes and additions I
> > would like to see in in kolla and kolla-ansible.
> >
> > 1. Keepalived is a problem in layer3 spine leaf networks as any
> > floating
> > IP can only exist in one leaf (and VRRP is a problem in layer3). I
> > would
> > like to use consul and registrar to get rid of the "internal"
> floating
> > IP and use consuls DNS service discovery to connect all services with
> > each other.
> >
> >
> > Without reading up, I'm not sure exactly how this fits together. If
> > kolla-ansible made the API host configurable for each service rather
> > than globally, would that be a step in the right direction?
>
> No that would not help. The problem is HA. Right now there is a
> "central" floating IP (kolla_internal_vip_address) that is used for all
> services to connect to (each other). Keepalived is failing that IP over
> if the "active" host fails. In a layer3 (CLOS/Spine-Leaf) network this
> IP is only available in one leaf/rack. So that rack is a "SPOF".
> Using service discovery fits perfect in a CLOS network and scales much
> better as a HA solution.
>
> Right, but what I'm saying as a thought experiment is, if we gave you the
required variables in kolla-ansible (e.g. nova_internal_fqdn) to make this
possible with an externally managed consul service, could that work?

>
> >
> > 2. Using "ports" for external API (endpoint) access is a major
> headache
> > if a firewall is involved. I would like to configure the HAProxy (or
> > fabio) for the external access to use "Host:" like, eg. "Host:
> > keystone.somedomain.tld", "Host: nova.somedomain.tld", ... with
> HTTPS.
> > Any customer would just need HTTPS access and not have to open all
> > those
> > ports in his firewall. For some enterprise customers it is not
> possible
> > to request FW changes like that.
> >
> > 3. HAProxy is not capable to handle "read/write" split with Galera. I
> > would like to introduce ProxySQL to be able to scale Galera.
> >
> >
> > It's now possible to use an external database server with kolla-ansible,
> > instead of deploying a mariadb/galera cluster. This could be implemented
> > how you like, see
> >
> https://docs.openstack.org/kolla-ansible/latest/reference/external-mariadb-guide.html
> .
>
> Yes I agree. And this is what we will do in our first production
> deployment. But I would love to see ProxySQL in Kolla as well.
> As a side note: Kolla-ansible does use:
>
> option mysql-check user haproxy post-41
>
> to check Galera, but that check does not fail if the node is out of sync
> with the other nodes!
>
> http://galeracluster.com/documentation-webpages/monitoringthecluster.html
>
> That's good to know. Could you raise a bug in kolla-ansible on launchpad,
and offer advice on how to improve this check if you have any?

>
> >
> > 4. HAProxy is fine but fabio integrates well with consul, statsd and
> > could be connected to a vault cluster to manage secure certificate
> > access.
> >
> > As above.
> >
> > 5. I would like to add vault as Barbican backend.
> >
> > Does this need explicit support in kolla and kolla-ansible, or could it
> > be done through configuration of barbican.conf? Are there additional
> > packages required in the barbican container? If so, see
> >
> https://docs.openstack.org/kolla/latest/admin/image-building.html#package-customisation
> .
>
> True but the vault (and consul) containers could be deployed and managed
> by kolla-ansible.
>
> I'd like to see if anyone else is interested in this. Kolla ansible
already deploys a large number of services, which is great. As with many
other projects I'm seeing the resources of core contributors fall off a
little, and I think we need to

Re: [openstack-dev] [kolla] add service discovery, proxysql, vault, fabio and FQDN endpoints

2018-10-09 Thread Jay Pipes

On 10/09/2018 06:34 AM, Florian Engelmann wrote:

Am 10/9/18 um 11:41 AM schrieb Jay Pipes:

On 10/09/2018 04:34 AM, Christian Berendt wrote:

On 8. Oct 2018, at 19:48, Jay Pipes wrote:

Why not send all read and all write traffic to a single haproxy
endpoint and just have haproxy spread all traffic across each Galera
node?

Galera, after all, is multi-master synchronous replication... so it
shouldn't matter which node in the Galera cluster you send traffic to.

Probably because of MySQL deadlocks in Galera:

—snip—
Galera cluster has known limitations, one of them is that it uses
cluster-wide optimistic locking. This may cause some transactions to
rollback. With an increasing number of writeable masters, the
transaction rollback rate may increase, especially if there is write
contention on the same dataset. It is of course possible to retry the
transaction and perhaps it will COMMIT in the retries, but this will
add to the transaction latency. However, some designs are deadlock
prone, e.g sequence tables.

—snap—

Source:
https://severalnines.com/resources/tutorials/mysql-load-balancing-haproxy-tutorial

Have you seen the above in production?

Yes of course. Just depends on the application and how high the workload
gets.