Yuriy Taraday <yorik....@gmail.com> wrote:
> On Fri Feb 20 2015 at 9:14:30 PM Joshua Harlow <harlo...@outlook.com> wrote: > > This feels like something we could do in the service manager base class, > > maybe by adding a "post fork" hook or something. > > +1 to that. > > I think it'd be nice to have the service __init__() maybe be something like: > > def __init__(self, threads=1000, prefork_callbacks=None, > postfork_callbacks=None): > self.postfork_callbacks = postfork_callbacks or [] > self.prefork_callbacks = prefork_callbacks or [] > # always ensure we are closing any left-open fds last... > self.prefork_callbacks.append(self._close_descriptors) > ... > > (you must've meant postfork_callbacks.append) > > Note that multiprocessing module already have > `multiprocessing.util.register_after_fork` method that allows to register > callback that will be called every time a Process object is run. If we remove > explicit use of `os.fork` in oslo.service (replace it with Process class) > we'll be able to specify any after-fork callbacks in libraries that they > need. > For example, EngineFacade could register `pool.dispose()` callback there (it > should have some proper finalization logic though). +1 to use Process and the callback system for required initialization steps and so forth, however I don’t know that an oslo lib should silently register global events on the assumption of how its constructs are to be used. I think whatever Oslo library is responsible for initiating the Process/fork should be where it ensures that resources from other Oslo libraries are set up correctly. So oslo.service might register its own event handler with oslo.db such that it gets notified of new database engines so that it can associate a disposal with it; it would do something similar for oslo.messaging and other systems that use file handles. The end result might be that it uses register_after_fork(), but the point is that oslo.db.sqlalchemy.create_engine doesn’t do this; it lets oslo.service apply a hook so that oslo.service can do it on behalf of oslo.db. So, instead of oslo.service cutting through and closing out the file descriptors from underneath other oslo libraries that opened them, we set up communication channels between oslo libs that maintain a consistent layer of abstraction, and instead of making all libraries responsible for the side effects that might be introduced from other oslo libraries, we make the side-effect-causing library the point at which those effects are ameliorated as a service to other oslo libraries. This allows us to keep the knowledge of what it means to use “multiprocessing” in one place, rather than spreading out its effects. __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev