Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack

[Robert Collins]

Ok so this is interesting. I think this new feature will have caused
bugs in anything using this idiom:


def foo(bar=CONF):

because that is only evaluated at import time - any later reevaluation
of the config settings won't propagate into code. (This is why we
recently avoided that idiom in Ironic, but part of that review was
looking at where else it was used).

So - something for reviewers to watch out for - default parameter
values from config settings in daemons is no longer 'safe' (it never
really was, but now it's clearly always unsafe).

-Rob

On 21 December 2013 09:09, Doug Hellmann  wrote:
>
>
>
> On Fri, Dec 20, 2013 at 11:22 AM, Sean Dague  wrote:
>>
>> On 12/20/2013 10:56 AM, Sean Dague wrote:
>> > On 12/20/2013 09:59 AM, Sean Dague wrote:
>> > 
>> >> So as Clint said, SIGHUP is only appropriate to do that *if* the
>> >> process
>> >> is daemonized. If it's in the foreground it's not.
>> >>
>> >> So that logic needs to be better.
>> >
>> > This is basically a blocker for adding any upgrade testing from
>> > something later than havana. Grenade upstream is still functioning
>> > because the service code wasn't merged into nova until after havana was
>> > cut.
>> >
>> > However there is a desire to do more interesting upgrade patterns, and
>> > without the ability to shutdown nova services on master in the gate,
>> > that's going to hit us pretty hard.
>> >
>> > So I'd like to get this fixed soon. As digging us out of this later is
>> > going to be way more expensive.
>>
>> Work around here for review - https://review.openstack.org/#/c/63444/
>
>
> That fix has landed in oslo, and I'm working on a patch to copy it into nova
> now.
>
> Doug
>
>
>>
>>
>>
>> -Sean
>>
>> --
>> Sean Dague
>> Samsung Research America
>> s...@dague.net / sean.da...@samsung.com
>> http://dague.net
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack

[Doug Hellmann]

On Fri, Dec 20, 2013 at 11:22 AM, Sean Dague  wrote:

> On 12/20/2013 10:56 AM, Sean Dague wrote:
> > On 12/20/2013 09:59 AM, Sean Dague wrote:
> > 
> >> So as Clint said, SIGHUP is only appropriate to do that *if* the process
> >> is daemonized. If it's in the foreground it's not.
> >>
> >> So that logic needs to be better.
> >
> > This is basically a blocker for adding any upgrade testing from
> > something later than havana. Grenade upstream is still functioning
> > because the service code wasn't merged into nova until after havana was
> cut.
> >
> > However there is a desire to do more interesting upgrade patterns, and
> > without the ability to shutdown nova services on master in the gate,
> > that's going to hit us pretty hard.
> >
> > So I'd like to get this fixed soon. As digging us out of this later is
> > going to be way more expensive.
>
> Work around here for review - https://review.openstack.org/#/c/63444/


That fix has landed in oslo, and I'm working on a patch to copy it into
nova now.

Doug



>
>
> -Sean
>
> --
> Sean Dague
> Samsung Research America
> s...@dague.net / sean.da...@samsung.com
> http://dague.net
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack

[Sean Dague]

On 12/20/2013 10:56 AM, Sean Dague wrote:
> On 12/20/2013 09:59 AM, Sean Dague wrote:
> 
>> So as Clint said, SIGHUP is only appropriate to do that *if* the process
>> is daemonized. If it's in the foreground it's not.
>>
>> So that logic needs to be better.
> 
> This is basically a blocker for adding any upgrade testing from
> something later than havana. Grenade upstream is still functioning
> because the service code wasn't merged into nova until after havana was cut.
> 
> However there is a desire to do more interesting upgrade patterns, and
> without the ability to shutdown nova services on master in the gate,
> that's going to hit us pretty hard.
> 
> So I'd like to get this fixed soon. As digging us out of this later is
> going to be way more expensive.

Work around here for review - https://review.openstack.org/#/c/63444/

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack

[Sean Dague]

On 12/20/2013 09:59 AM, Sean Dague wrote:

> So as Clint said, SIGHUP is only appropriate to do that *if* the process
> is daemonized. If it's in the foreground it's not.
> 
> So that logic needs to be better.

This is basically a blocker for adding any upgrade testing from
something later than havana. Grenade upstream is still functioning
because the service code wasn't merged into nova until after havana was cut.

However there is a desire to do more interesting upgrade patterns, and
without the ability to shutdown nova services on master in the gate,
that's going to hit us pretty hard.

So I'd like to get this fixed soon. As digging us out of this later is
going to be way more expensive.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack

[Sean Dague]

On 12/20/2013 09:55 AM, Doug Hellmann wrote:
> 
> 
> 
> On Fri, Dec 20, 2013 at 1:54 AM, Clint Byrum  > wrote:
> 
> Excerpts from Sean Dague's message of 2013-12-19 16:33:12 -0800:
> > So a few people had been reporting recently that unstack no longer
> stops
> > nova processes, which I only got around to looking at today. It turns
> > out the new common.service stack from oslo takes SIGHUP and treats
> it as
> > a restart. Which isn't wrong, but is new, and is incompatible with
> > screen (the way we use it). Because we use -X stuff, the resulting -X
> > quit sends SIGHUP to the child processes.
> >
> > So the question is, are we definitely in a state now where nova
> services
> > can and do want to support SIGHUP as restart?
> >
> > If so, is there interest in being able to disable that behavior at
> start
> > time, so we can continue with a screen based approach as well?
> >
> > If not, we'll need to figure out another way to approach the
> shutdown in
> > devstack. Which is fine, just work that wasn't expected.
> >
> 
> Perhaps if the process is running in the foreground, as it does in
> devstack, it should still terminate on SIGHUP rather than restart.
> 
> 
> It looks like the changes to ServiceLauncher.wait() that introduced this
> behavior are related to
> https://blueprints.launchpad.net/oslo/+spec/cfg-reload-config-files
> where we wanted a service to be able to re-read its configuration files
> on a signal. HUP seems appropriate for a production
> environment, but probably not for development.
> 
> I opened https://bugs.launchpad.net/oslo/+bug/1263122 to track the fix.

So as Clint said, SIGHUP is only appropriate to do that *if* the process
is daemonized. If it's in the foreground it's not.

So that logic needs to be better.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack

[Doug Hellmann]

On Fri, Dec 20, 2013 at 1:54 AM, Clint Byrum  wrote:

> Excerpts from Sean Dague's message of 2013-12-19 16:33:12 -0800:
> > So a few people had been reporting recently that unstack no longer stops
> > nova processes, which I only got around to looking at today. It turns
> > out the new common.service stack from oslo takes SIGHUP and treats it as
> > a restart. Which isn't wrong, but is new, and is incompatible with
> > screen (the way we use it). Because we use -X stuff, the resulting -X
> > quit sends SIGHUP to the child processes.
> >
> > So the question is, are we definitely in a state now where nova services
> > can and do want to support SIGHUP as restart?
> >
> > If so, is there interest in being able to disable that behavior at start
> > time, so we can continue with a screen based approach as well?
> >
> > If not, we'll need to figure out another way to approach the shutdown in
> > devstack. Which is fine, just work that wasn't expected.
> >
>
> Perhaps if the process is running in the foreground, as it does in
> devstack, it should still terminate on SIGHUP rather than restart.
>

It looks like the changes to ServiceLauncher.wait() that introduced this
behavior are related to
https://blueprints.launchpad.net/oslo/+spec/cfg-reload-config-files where
we wanted a service to be able to re-read its configuration files on a
signal. HUP seems appropriate for a production
 environment, but probably not for development.

I opened https://bugs.launchpad.net/oslo/+bug/1263122 to track the fix.

Doug
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack

[Oleg Gelbukh]

I'd +1 Clint on this. I believe that the only right way to handle SIGHUP
for process running in foreground is to terminate.

--
Best regards,
Oleg Gelbukh


On Fri, Dec 20, 2013 at 10:54 AM, Clint Byrum  wrote:

> Excerpts from Sean Dague's message of 2013-12-19 16:33:12 -0800:
> > So a few people had been reporting recently that unstack no longer stops
> > nova processes, which I only got around to looking at today. It turns
> > out the new common.service stack from oslo takes SIGHUP and treats it as
> > a restart. Which isn't wrong, but is new, and is incompatible with
> > screen (the way we use it). Because we use -X stuff, the resulting -X
> > quit sends SIGHUP to the child processes.
> >
> > So the question is, are we definitely in a state now where nova services
> > can and do want to support SIGHUP as restart?
> >
> > If so, is there interest in being able to disable that behavior at start
> > time, so we can continue with a screen based approach as well?
> >
> > If not, we'll need to figure out another way to approach the shutdown in
> > devstack. Which is fine, just work that wasn't expected.
> >
>
> Perhaps if the process is running in the foreground, as it does in
> devstack, it should still terminate on SIGHUP rather than restart.
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack

[Clint Byrum]

Excerpts from Sean Dague's message of 2013-12-19 16:33:12 -0800:
> So a few people had been reporting recently that unstack no longer stops
> nova processes, which I only got around to looking at today. It turns
> out the new common.service stack from oslo takes SIGHUP and treats it as
> a restart. Which isn't wrong, but is new, and is incompatible with
> screen (the way we use it). Because we use -X stuff, the resulting -X
> quit sends SIGHUP to the child processes.
> 
> So the question is, are we definitely in a state now where nova services
> can and do want to support SIGHUP as restart?
> 
> If so, is there interest in being able to disable that behavior at start
> time, so we can continue with a screen based approach as well?
> 
> If not, we'll need to figure out another way to approach the shutdown in
> devstack. Which is fine, just work that wasn't expected.
> 

Perhaps if the process is running in the foreground, as it does in
devstack, it should still terminate on SIGHUP rather than restart.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][nova] oslo common.service vs. screen and devstack

[Sean Dague]

So a few people had been reporting recently that unstack no longer stops
nova processes, which I only got around to looking at today. It turns
out the new common.service stack from oslo takes SIGHUP and treats it as
a restart. Which isn't wrong, but is new, and is incompatible with
screen (the way we use it). Because we use -X stuff, the resulting -X
quit sends SIGHUP to the child processes.

So the question is, are we definitely in a state now where nova services
can and do want to support SIGHUP as restart?

If so, is there interest in being able to disable that behavior at start
time, so we can continue with a screen based approach as well?

If not, we'll need to figure out another way to approach the shutdown in
devstack. Which is fine, just work that wasn't expected.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev