On 05/19/2017 04:23 AM, Mehdi Abaakouk wrote:
And some applications rely
on implicit internal contract/behavior/assumption.
IMO that's a bug for them. I'm inspired to see that Keystone, Nova
etc. are able to move between and eventlet backend and a mod_wsgi
backend. IMO eventlet is really not needed for those services that
present a REST interface. Although for a message queue with lots of
long-running connections that receive events, that's a place where I
*would* want to use a polling / non-blocking model. But I'd use it
explicitly, not with monkeypatching.
Since a new API is needed, why not writing a new lib. Anyways when you
get rid of eventlet you have so many thing to change to ensure your
performance will not drop.
While I don't know the specifics for your project(s), I don't buy that
in general because IMO eventlet is not giving us any performance boost
in the majority of cases. most of our IO is blocking on the database
and all the applications have DB connections throttled at about 50 per
process at the max, and that's only recently, it used to be just 15.
Changing from oslo.service to cotyledon is
really easy on the side.
I'd ask why not oslo.cotyledon but it seems there's a faction here that
is overall moving out of the Openstack umbrella in any case.
Docs state: "oslo.service being impossible to fix and bringing an
heavy dependency on eventlet, " is there a discussion thread on that?
Not really, I just put some comments on reviews and discus this on IRC.
Since nobody except Telemetry have expressed/try to get rid of eventlet.
Many (most?) of the web services can run under mod_wsgi with threads,
Keystone seems to be standard on this now and I get the impression Nova
is going in that direction too. (anyone correct me if I'm wrong on
any of that, I looked to ask around on IRC but it's too late in the day).
For the story we first get rid of eventlet in Telemetry, fixes couple of
performance issue due to using threading/process instead
greenlet/greenthread.
Then we fall into some weird issue due to oslo.service internal
implementation. Process not exiting properly, signals not received,
deadlock when signal are received, unkillable process,
tooz/oslo.messaging heartbeat not scheduled correctly, worker not
restarted when they are dead. All of what we expect from oslo.service
was not working correctly anymore because we remove the line
'eventlet.monkeypatch()'.
So, I've used gevent more than eventlet in my own upstream non-blocking
work, and while this opinion is like spilling water in the ocean, I
think applications should never use monkeypatching. They should call
into the eventlet/gevent greenlet API directly if that's what they want
to do.
Of course this means that database logic has to move out of greenlets
entirely since none of the DBAPIs use non-blocking IO. That's fine.
Database-oriented business logic should not be in greenlets. I've
written about this as well. If one is familiar enough with greenlets
and threads you can write an application that makes explicit use of
both. However, that's application level stuff. Web service apps like
Nova conductor / Neutron / Keystone should not be aware of any of that.
They should be coded to assume nothing about context switching. IMO
the threading model is "safer" to code towards since you have to handle
locking and concurrency contingencies explicitly without hardwiring that
to your assumptions about when context switching is to take place and
when it's not.
For example, when oslo.service receive a signal, it can arrive on any
thread, this thread is paused, the callback is run in this thread
context, but if the callback try to discus to your code in this thread,
the process lockup, because your code is paused. Python
offers tool to avoid that (signal.set_wakeup_fd), but oslo.service don't
use it. I have tried to run callbacks only on the main thread with
set_wakeup_fd, to avoid this kind of issue but I fail. The whole
oslo.service code is clearly not designed to be threadsafe/signalsafe.
Well, it works for eventlet because you have only one real thread.
And this is just one example on complicated thing I have tried to fix,
before starting cotyledon.
I've no doubt oslo.service has major eventlet problems baked in, I've
looked at it a little bit but didn't go too far with it. That still
doesn't mean there shouldn't be an "oslo.service2" that can effectively
produce a concurrency-agnostic platform. It of course would have the
goal in mind of moving projects off eventlet since as I mentioned,
eventlet monkeypatching should not be used which means our services
should do most of their "implicitly concurrent" work within threads.
Basically I think openstack should be getting off eventlet in a big way
so I guess my sentiment here is that the Gnocchi / Cotyledon /etc.
faction is just splitting off rather than serving as any kind of
direction for the rest of Openstack to start looking. But that's only
an impression, maybe projects will use Cotyledon anyway. If every
project goes off and uses something completely different though, then I
think we're losing. The point of oslo was to prevent that.
I'm finding it hard to believe that only a few years ago, everyone saw
the wisdom of not re-implementing everything in their own projects and
using a common layer like oslo, and already that whole situation is
becoming forgotten - not just for consistency, but also when a bug is
found, if fixed in oslo it gets fixed for everyone.
Because the internal of cotyledon and oslo.service are so different.
Having the code in oslo or not doesn't help for maintenance anymore.
Cotyledon is a lib, code and bugs :) can already be shared between
projects that doesn't want eventlet.
An increase in the scope of oslo is essential to dealing with the
issue of "complexity" in openstack.
Increasing the scope of oslo works only if libs have maintainers. But
most of them lack of people today. Most of oslo libs are in maintenance
mode. But that another subject.
The state of openstack as dozens of individual software projects each
with their own idiosyncratic quirks, CLIs, process and deployment
models, and everything else that is visible to operators is ground
zero for perceived operator complexity.
Cotyledon have been written to be Openstack agnostic. But I have also
write an optional module within the library to glue oslo.config and
cotyledon. Mainly to mimic the oslo.config options/reload of
oslo.service and make operators experience unchanged for Openstack
people.
OK, so that would be your oslo.cotyledon. That works.
--
Mehdi Abaakouk
mail: sil...@sileht.net
irc: sileht
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
