On 01/07/2016 06:55 PM, Mike Bayer wrote:
On 01/07/2016 11:02 AM, Sean Dague wrote:
On 01/07/2016 09:56 AM, Brant Knudson wrote:
On Thu, Jan 7, 2016 at 6:39 AM, Clayton O'Neill <clay...@oneill.net
<mailto:clay...@oneill.net>> wrote:
On Thu, Jan 7, 2016 at 2:49 AM, Roman Podoliaka
<rpodoly...@mirantis.com <mailto:rpodoly...@mirantis.com>> wrote:
>
> Linux gurus please correct me here, but my understanding is that Linux
> kernel queues up to $backlog number of connections *per socket*. In
> our case child processes inherited the FD of the socket, so they will
> accept() connections from the same queue in the kernel, i.e. the
> backlog value is for *all* child processes, not *per* process.
Yes, it will be shared across all children.
>
> In each child process eventlet WSGI server calls accept() in a loop to
> get a client socket from the kernel and then puts into a greenlet from
> a pool for processing:
It’s worse than that. What I’ve seen (via strace) is that eventlet
actually
converts socket into a non-blocking socket, then converts that
accept() into a
epoll()/accept() pair in every child. Then when a connection comes
in, every
child process wakes up out of poll and races to try to accept on the the
non-blocking socket, and all but one of them fails.
This means that every time there is a request, every child process
is woken
up, scheduled on CPU and then put back to sleep. This is one of the
reasons we’re (slowly) moving to uWSGI.
I just want to note that I've got a change proposed to devstack that
adds a config option to run keystone in uwsgi (rather than under
eventlet or in apache httpd mod_wsgi), see
https://review.openstack.org/#/c/257571/ . It's specific to keystone
since I didn't think other projects were moving away from eventlet, too.
I feel like this is a confused point that keeps being brought up.
The preferred long term direction of all API services is to be deployed
on a real web server platform. It's a natural fit for those services as
they are accepting HTTP requests and doing things with them.
Most OpenStack projects have worker services beyond just an HTTP server.
(Keystone is one of the very few exceptions here). Nova has nearly a
dozen of these worker services. These don't naturally fit as wsgi apps,
they are more traditional daemons, which accept requests over the
network, but also have periodic jobs internally and self initiate
actions. They are not just call / response. There is no long term
direction for these to move off of eventlet.
This is totally speaking as an outsider without taking into account all
the history of these decisions, but the notion of "Python + we're a
daemon" == "we must use eventlet" seems a little bit rigid. Also, the
notion of "we have background tasks" == "we cant run in a web server",
also not clear. If a service intends to serve HTTP requests, that
portion of that service should be deployed in a web server; if the
system has other "background tasks", ideally those are in a separate
daemon altogether, but also even if you're under something like
mod_wsgi, you can spawn a child process or worker thread regardless.
You always have a Python interpreter running and all the things it can do.
In the case of nova at least most (all?) of these separate worker services do
not process HTTP requests, but rather RPC requests.
It might make sense for nova-api to run under a web server even if
nova-compute/nova-conductor/nova-scheduler/etc don't.
Chris
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
