Re: [openstack-dev] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

2015-12-09 Thread Thomas Goirand 
On 12/08/2015 04:09 AM, Dolph Mathews wrote:
> In Debian, many services/daemons are run, then their API is used by the
> package. In the case of Keystone, for example, it is possible to ask,
> via Debconf, that Keystone registers itself in the service catalog. If
> we get Keystone within Apache, it becomes at least harder to do so.
> 
> 
> You start the next paragraph with "the other issue," but I'm not clear
> on what the issue is here? This sounds like a bunch of complexity that
> is working as you expect it to.

Ok, let me try again. If the only way to get Keystone up and running is
through a web server like Apache, then starting Keystone and then using
its API is not as easy as if it was a daemon on its own. For example,
there may be some other types of configuration in use in the web server
which the package doesn't control.

> The other issue is that if all services are sharing the same web server,
> restarting the web server restarts all services. Or, said otherwise: if
> I need to change a configuration value of any of the services served by
> Apache, I will need to restart them all, which is very annoying: I very
> much prefer to just restart *ONE* service if I need.
> 
> As a deployer, I'd solve this by running one API service per server. As
> a packager, I don't expect you to solve this outside of AIO
> architectures, in which case, uptime is obviously not a concern.

Let's say there's a security issue in Keystone. One would expect that a
simple "apt-get dist-upgrade" will do all. If Keystone is installed in a
web server, should the package aggressively attempts to restart it? If
not, what is the proposed solution to have Keystone restarted in this case?

> Also, something which we learned the hard way at Mirantis: it is *very*
> annoying that Apache restarts every Sunday morning by default in
> distributions like Ubuntu and Debian (I'm not sure for the other
> distros). No, the default config of logrotate and Apache can't be
> changed in distros just to satisfy OpenStack users: there's other users
> of Apache in these distros.
> 
> Yikes! Is the debian Apache package intended to be useful in production?
> That doesn't sound like an OpenStack-specific problem at all. How is
> logrotate involved? Link?

It is logrotate which restarts apache. From /etc/logrotate.d/apache2:

/var/log/apache2/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 640 root adm
sharedscripts
postrotate
if /etc/init.d/apache2 status > /dev/null ; then \
/etc/init.d/apache2 reload > /dev/null; \
fi;
endscript
prerotate
if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
run-parts /etc/logrotate.d/httpd-prerotate; \
fi; \
endscript
}

This is to be considered quite harmful. Yes, I am fully aware that this
file can be tweaked. Though this is the default, and it is always best
to provide a default which works for our users. And in this case, no
OpenStack package maintainer controls it.

> Then, yes, uWSGI becomes a nice option. I used it for the Barbican
> package, and it worked well. Though the uwsgi package in Debian isn't
> very well maintained, and multiple times, Barbican could have been
> removed from Debian testing because of RC bugs against uWSGI.
> 
> uWSGI is a nice option, but no one should be tied to that either-- in
> the debian world or otherwise.

For Debian users, I intend to provide useful configuration so that
everything works by default. OpenStack is complex enough. It is my role,
as a package maintainer, to make it easier to use.

One of the options I have is to create new packages like this:

python-keystone -> Already exists, holds the Python code
keystone -> Currently contains the Eventlet daemon

I could transform the later into a meta package depending on any of the
below options:

keystone-apache -> Default configuration for Apache
keystone-uwsgi -> An already configured startup script using uwsgi

Though I'm not sure the FTP masters will love me if I create so many new
packages just to create automated configurations... Also, maybe it's
just best to provide *one* implementation which we all consider the
best. I'm just not sure yet which one it is. Right now, I'm leaning
toward uwsgi.

> So, all together, I'm a bit reluctant to see the Eventlet based servers
> going away. If it's done, then yes, I'll work around it. Though I'd
> prefer if it didn't.
> 
> Think of it this way: keystone is moving towards using Python
> where Python excels, and is punting up the stack where Python is
> handicapped. Don't think of it as a work around, think of it as having
> the freedom to architect your own deployment.

I'm ok with that, but as per the above, I'd like to provide something
which just works for

Re: [openstack-dev] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

2015-12-09 Thread Dolph Mathews 
On Wed, Dec 9, 2015 at 2:25 AM, Thomas Goirand  wrote:

> On 12/08/2015 04:09 AM, Dolph Mathews wrote:
> > In Debian, many services/daemons are run, then their API is used by
> the
> > package. In the case of Keystone, for example, it is possible to ask,
> > via Debconf, that Keystone registers itself in the service catalog.
> If
> > we get Keystone within Apache, it becomes at least harder to do so.
> >
> >
> > You start the next paragraph with "the other issue," but I'm not clear
> > on what the issue is here? This sounds like a bunch of complexity that
> > is working as you expect it to.
>
> Ok, let me try again. If the only way to get Keystone up and running is
> through a web server like Apache, then starting Keystone and then using
> its API is not as easy as if it was a daemon on its own. For example,
> there may be some other types of configuration in use in the web server
> which the package doesn't control.
>

I would make the assumption that the node is solely intended for use by
keystone. Either it is solely for use by keystone and I care about uptime,
or the node is not solely for use by keystone and thus I don't care about
uptime.


>
> > The other issue is that if all services are sharing the same web
> server,
> > restarting the web server restarts all services. Or, said otherwise:
> if
> > I need to change a configuration value of any of the services served
> by
> > Apache, I will need to restart them all, which is very annoying: I
> very
> > much prefer to just restart *ONE* service if I need.
> >
> > As a deployer, I'd solve this by running one API service per server. As
> > a packager, I don't expect you to solve this outside of AIO
> > architectures, in which case, uptime is obviously not a concern.
>
> Let's say there's a security issue in Keystone. One would expect that a
> simple "apt-get dist-upgrade" will do all. If Keystone is installed in a
> web server, should the package aggressively attempts to restart it? If
> not, what is the proposed solution to have Keystone restarted in this case?
>

Yes, restart the web server aggressively, because I'm likely doing the
upgrade with the intent of receiving the immediate benefit of updates.


>
> > Also, something which we learned the hard way at Mirantis: it is
> *very*
> > annoying that Apache restarts every Sunday morning by default in
> > distributions like Ubuntu and Debian (I'm not sure for the other
> > distros). No, the default config of logrotate and Apache can't be
> > changed in distros just to satisfy OpenStack users: there's other
> users
> > of Apache in these distros.
> >
> > Yikes! Is the debian Apache package intended to be useful in production?
> > That doesn't sound like an OpenStack-specific problem at all. How is
> > logrotate involved? Link?
>
> It is logrotate which restarts apache. From /etc/logrotate.d/apache2:
>
> /var/log/apache2/*.log {
> daily
> missingok
> rotate 14
> compress
> delaycompress
> notifempty
> create 640 root adm
> sharedscripts
> postrotate
> if /etc/init.d/apache2 status > /dev/null ; then \
> /etc/init.d/apache2 reload > /dev/null; \
> fi;
> endscript
> prerotate
> if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
> run-parts /etc/logrotate.d/httpd-prerotate; \
> fi; \
> endscript
> }
>
> This is to be considered quite harmful. Yes, I am fully aware that this
> file can be tweaked. Though this is the default, and it is always best
> to provide a default which works for our users. And in this case, no
> OpenStack package maintainer controls it.
>
> > Then, yes, uWSGI becomes a nice option. I used it for the Barbican
> > package, and it worked well. Though the uwsgi package in Debian isn't
> > very well maintained, and multiple times, Barbican could have been
> > removed from Debian testing because of RC bugs against uWSGI.
> >
> > uWSGI is a nice option, but no one should be tied to that either-- in
> > the debian world or otherwise.
>
> For Debian users, I intend to provide useful configuration so that
> everything works by default. OpenStack is complex enough. It is my role,
> as a package maintainer, to make it easier to use.
>
> One of the options I have is to create new packages like this:
>
> python-keystone -> Already exists, holds the Python code
> keystone -> Currently contains the Eventlet daemon
>
> I could transform the later into a meta package depending on any of the
> below options:
>
> keystone-apache -> Default configuration for Apache
> keystone-uwsgi -> An already configured startup script using uwsgi
>
> Though I'm not sure the FTP masters will love me if I create so many new
> packages just to create automated configurations... Also, maybe it's
> just best to provide *one*

Re: [openstack-dev] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

2015-12-09 Thread Dolph Mathews 
On Wednesday, December 9, 2015, Ginwala, Aliasgar  wrote:

> Hi Dolph/team:
>
> As requested I have outlined most of the files and configs to give more
> clear picture @ https://gist.github.com/noah8713/7d5554d78b60cd9a4999:
>

The number of threads per API your config files show range from 20 to
10,000 (processes
* threads). On one end, your server might be idling during the benchmark.
On the other end, you're probably exhausting the server's memory during the
benchmark run. So, absolutely nothing about this benchmark appears to be
fair.

What is the maximum number of concurrent users you're spinning up to in
locust? (not the spawn rate)


>
> * keystone.conf   —uploaded
> * distro used for testing, in case they override the project's defaults
>  —mentioned
> * all nginx config files —uploaded nginx-keystone.conf
> * all uwsgi config files —keystone-main.ini, keystone-admin.ini and
> upstart file
> * apache config, including virtual hosts and mods —apache-keystone.conf
> and python files (common for nginx)
> * your test client and it's configuration —mentioned
> * server & client architecture, and at least some idea of what lies in
> between (networking, etc)  —briefly outlined
> * whatever else I'm forgetting —feel free to add in the comments
>
>
>
> Regards,
> Ali
>
> From: Dolph Mathews  >
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org
> >
> Date: Wednesday, December 9, 2015 at 5:42 AM
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org
> >
> Subject: Re: [openstack-dev] [Openstack-operators] [keystone] Removing
> functionality that was deprecated in Kilo and upcoming deprecated
> functionality in Mitaka
>
> Benchmarks always appreciated!
>
> But, these types of benchmarks are *entirely* useless unless you can
> provide the exact configuration you used for each scenario so that others
> can scrutinize the test method and reproduce your results. So, off the top
> of my head, I'm looking for:
>
> * keystone.conf
> * distro used for testing, in case they override the project's defaults
> * all nginx config files
> * all uwsgi config files
> * apache config, including virtual hosts and mods
> * your test client and it's configuration
> * server & client architecture, and at least some idea of what lies in
> between (networking, etc)
> * whatever else I'm forgetting
>
> A mailing list is probably not the best method to provide anything other
> than a summary; so I'd suggest publishing the details in a gist, blog post,
> or both.
>
> And to comment on the results themselves: you shouldn't be seeing that big
> of a performance gap between httpd and everything else; something is
> fundamentally different about that configuration. These are just web
> servers, after all. Choosing between them should not be a matter of
> performance, but it should be a choice of documentation, licensing,
> community, operability, supportability, reliability, etc. Performance
> should be relatively similar, and thus a much lower priority in making your
> selection.
>
> On Tue, Dec 8, 2015 at 10:09 PM, Ginwala, Aliasgar  > wrote:
>
>>
>>
>> Hi All:
>>
>> Just to inform Steve and all the folks who brought up this talk ;We did
>> some benchmarking using wsgi, apache and nginx for keystone with mysql as
>> token backend and we got following results on Juno version. Hence I am just
>> giving you brief highlight about the results we got.
>>
>> spawning 100 users per sec for create token, below are the results:
>>
>> *Using nginx with uwsgi: *
>> rps *32*—(reqests/sec)
>> median time ~ 3.3 sec
>> no of processes 20
>>
>> *using apache *
>> rps *75*
>> median time ~ 1.3 sec
>> avg time - 1.5 sec
>> no of processes 20
>>
>> *using wsgi *
>> rps *28*
>> median time ~ 3.4
>> avg 3.5
>> no of processes 20
>>
>>
>> We are planning to switch to apache since we are not seeing good results
>> using nginx with uwsgi. May be some more added support is required for
>> nginx performance.We did not encounter this auto restart issue in test
>> yet and hence are open to more inputs.
>>
>> Any other suggestions are welcome too. Let us know in case of queries.
>>
>> Regards,
>> Aliasgar
>>
>> On 8 December 2015 at 07:53, Thomas Goirand > > wrote:
>>
>>> On 12/01/2015 07:57 AM, Steve Martinelli wrote:
>>> > Trying to summarize here...
>>> >
>>> > - There isn't much interest in keeping eventlet around.
>>> > - Folks are OK with running keystone in a WSGI server, but feel they
>>> are
>>> > constrained by Apache.
>>> > - uWSGI could help to support multiple web

Re: [openstack-dev] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

2015-12-07 Thread Dolph Mathews 
On Monday, December 7, 2015, Thomas Goirand  wrote:

> On 12/01/2015 07:57 AM, Steve Martinelli wrote:
> > Trying to summarize here...
> >
> > - There isn't much interest in keeping eventlet around.
> > - Folks are OK with running keystone in a WSGI server, but feel they are
> > constrained by Apache.
> > - uWSGI could help to support multiple web servers.
> >
> > My opinion:
> >
> > - Adding support for uWSGI definitely sounds like it's worth
> > investigating, but not achievable in this release (unless someone
> > already has something cooked up).
> > - I'm tempted to let eventlet stick around another release, since it's
> > causing pain on some of our operators.
> > - Other folks have managed to run keystone in a web server (and
> > hopefully not feel pain when doing so!), so it might be worth getting
> > technical details on just how it was accomplished. If we get an OK from
> > the operator community later on in mitaka, I'd still be OK with removing
> > eventlet, but I don't want to break folks.
> >
> > stevemar
> >
> > From: John Dewey >
> > 100% agree.
> >
> > We should look at uwsgi as the reference architecture. Nginx/Apache/etc
> > should be interchangeable, and up to the operator which they choose to
> > use. Hell, with tcp load balancing now in opensource Nginx, I could get
> > rid of Apache and HAProxy by utilizing uwsgi.
> >
> > John
>
> The main problem I see with running Keystone (or any other service) in a
> web server, is that *I* (as a package maintainer) will loose the control
> over when the service is started. Let me explain why that is important
> for me.


You can still control *a* web server, it's just that keystone no
longer attempts to bundle *the* web server.


>
> In Debian, many services/daemons are run, then their API is used by the
> package. In the case of Keystone, for example, it is possible to ask,
> via Debconf, that Keystone registers itself in the service catalog. If
> we get Keystone within Apache, it becomes at least harder to do so.


You start the next paragraph with "the other issue," but I'm not clear on
what the issue is here? This sounds like a bunch of complexity that is
working as you expect it to.


>
> The other issue is that if all services are sharing the same web server,
> restarting the web server restarts all services. Or, said otherwise: if
> I need to change a configuration value of any of the services served by
> Apache, I will need to restart them all, which is very annoying: I very
> much prefer to just restart *ONE* service if I need.


As a deployer, I'd solve this by running one API service per server. As a
packager, I don't expect you to solve this outside of AIO architectures, in
which case, uptime is obviously not a concern.


>
> Also, something which we learned the hard way at Mirantis: it is *very*
> annoying that Apache restarts every Sunday morning by default in
> distributions like Ubuntu and Debian (I'm not sure for the other
> distros). No, the default config of logrotate and Apache can't be
> changed in distros just to satisfy OpenStack users: there's other users
> of Apache in these distros.


Yikes! Is the debian Apache package intended to be useful in production?
That doesn't sound like an OpenStack-specific problem at all. How is
logrotate involved? Link?


>
> Then, yes, uWSGI becomes a nice option. I used it for the Barbican
> package, and it worked well. Though the uwsgi package in Debian isn't
> very well maintained, and multiple times, Barbican could have been
> removed from Debian testing because of RC bugs against uWSGI.


uWSGI is a nice option, but no one should be tied to that either-- in the
debian world or otherwise.


>
> So, all together, I'm a bit reluctant to see the Eventlet based servers
> going away. If it's done, then yes, I'll work around it. Though I'd
> prefer if it didn't.


Think of it this way: keystone is moving towards using Python where Python
excels, and is punting up the stack where Python is handicapped. Don't
think of it as a work around, think of it as having the freedom to
architect your own deployment.


>
> It is also my view that it's up to the deployers to decide how they want
> to implement things. For many small use cases, Eventlet performs well
> enough.


Unfortunately, "small" is not "all."


>
> Finally, one thing which I never understood: if Eventlet is bad as an
> HTTP server, can't we use anything else written in Python? Isn't it
> possible to write a decent HTTP server in Python? Why are we forced into
> just Eventlet for doing the job? I haven't searched around, but there
> must be loads of alternatives, no?


Yep! There are many. Eventlet is a bit unique, but ("Core") OpenStack
services have historically been tightly bound to Eventlet for it's
native-ish threading support. As Keystone has broken free, you are then
free to deploy our generic WSGI app/s using any generic WSGI server in any
process / threading architecture that suits

[openstack-dev] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

2015-11-30 Thread Steve Martinelli 

In the Mitaka release, the keystone team will be removing functionality
that was marked for deprecation in Kilo, and marking certain functions as
deprecated in Mitaka (that may be removed in at least 2 cycles).

removing deprecated functionality
=

This is not a full list, but these are by and large the most contentious
topics.

* Eventlet support: This was marked as deprecated back in Kilo and is
currently scheduled to be removed in Mitaka in favor of running keystone in
a WSGI server. This is currently how we test keystone in the gate, and
based on the feedback we received at the summit, a lot of folks have moved
to running keystone under Apache since we’ve announced this change.
OpenStack's CI is configured to mainly test using this deployment model.
See [0] for when we started to issue warnings.

* Using LDAP to store assignment data: Like eventlet support, this feature
was also deprecated in Kilo and scheduled to be removed in Mitaka. To store
assignment data (role assignments) we suggest using an SQL based backend
rather than LDAP. See [1] for when we started to issue warnings.

* Using LDAP to store project and domain data: The same as above, see [2]
for when we started to issue warnings.

* for a complete list:
https://blueprints.launchpad.net/keystone/+spec/removed-as-of-mitaka

functions deprecated as of mitaka
=

The following will adhere to the TC’s new standard on deprecating
functionality [3].

* LDAP write support for identity: We suggest simply not writing to LDAP
for users and groups, this effectively makes create, delete and update of
LDAP users and groups a no-op. It will be removed in the O release.

* PKI tokens: We suggest using UUID or fernet tokens instead. The PKI token
format has had issues with security and causes problems with both horizon
and swift when the token contains an excessively large service catalog. It
will be removed in the O release.

* v2.0 of our API: Lastly, the keystone team recommends using v3 of our
Identity API. We have had the intention of deprecating v2.0 for a while
(since Juno actually), and have finally decided to formally deprecate v2.0.
OpenStack’s CI runs successful v3 only jobs, there is complete feature
parity with v2.0, and we feel the CLI exposed via openstackclient is mature
enough to say with certainty that we can deprecate v2.0. It will be around
for at least FOUR releases, with the authentication routes
(POST /auth/tokens) potentially sticking around for longer.

* for a complete list:
https://blueprints.launchpad.net/keystone/+spec/deprecated-as-of-mitaka


If you have ANY concern about the following, please speak up now and let us
know!


Thanks!

Steve Martinelli
OpenStack Keystone Project Team Lead


[0]
https://github.com/openstack/keystone/blob/b475040636ccc954949e6372a60dd86845644611/keystone/server/eventlet.py#L77-L80
[1]
https://github.com/openstack/keystone/blob/28a30f53a6c0d4e84d60795e08f137e8194abbe9/keystone/assignment/backends/ldap.py#L34
[2]
https://github.com/openstack/keystone/blob/28a30f53a6c0d4e84d60795e08f137e8194abbe9/keystone/resource/backends/ldap.py#L36-L39

[3]
http://governance.openstack.org/reference/tags/assert_follows-standard-deprecation.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev