Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title Sean Griffin updated an issue Puppet / PUP-5387 AIX service provider returns before service operations are complete Change By: Sean Griffin QA Risk Assessment: Medium QA Highest Test Level Reason: unit QA Status: Reviewed Add Comment This message was sent by Atlassian JIRA (v6.4.13#64028-sha1:b7939e9) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title
William Hopper created an issue
Puppet / PUP-5387
AIX service provider returns before service operations are complete
Issue Type:
Bug
Assignee:
Unassigned
Created:
2015/10/15 11:01 AM
Environment:
AIX 5-7
Priority:
Normal
Reporter:
William Hopper
In AIX, when we stop the puppet service with puppet resource service puppet ensure=stopped, puppet returns before the service has actually finished stopping. AIX's service manager takes a while to stop the service, and for a brief period puppet doesn't understand what state the service is in:
-bash-3.2# /opt/puppetlabs/puppet/bin/puppet resource service puppet
service { 'puppet':
enable => 'false',
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title Josh Cooper updated an issue Puppet / PUP-5387 AIX service provider returns before service operations are complete Change By: Josh Cooper Sprint: Client 2015-11-11 Add Comment This message was sent by Atlassian JIRA (v6.4.11#64026-sha1:78f6ec4) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title Branan Riley commented on PUP-5387 Re: AIX service provider returns before service operations are complete Josh Cooper Do we have a fix version for this? Add Comment This message was sent by Atlassian JIRA (v6.4.11#64026-sha1:78f6ec4) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title Josh Cooper commented on PUP-5387 Re: AIX service provider returns before service operations are complete Branan Riley William Hopper was going to investigate whether this is an issue in the provider (when managing any service), or an issue in the puppet service itself, e.g. perhaps an issue in the ext script? Mostly likely this will be a PUP 4.3.0 issue. Add Comment This message was sent by Atlassian JIRA (v6.4.11#64026-sha1:78f6ec4) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title
William Hopper commented on PUP-5387
Re: AIX service provider returns before service operations are complete
This one's a bit odd, and it doesn't seem to happen all the time, It seems that the best way to reproduce is:
[0] [AIX] root@pe-aix-61-agent:~ # puppet apply -e "service { 'puppet': ensure => running }"
[0] [AIX] root@pe-aix-61-agent:~ # puppet apply refresh_manifest.pp --debug --verbose
...
Info: /Stage[main]/Main/Notify[refreshing]: Scheduling refresh of Service[puppet]
Debug: /Stage[main]/Main/Notify[refreshing]: The container Class[Main] will propagate my refresh event
Debug: Executing: '/usr/bin/lssrc -s puppet'
Debug: Service puppet is active
Debug: Executing: '/usr/bin/lssrc -s puppet'
Debug: Service puppet is active
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title William Hopper commented on PUP-5387 Re: AIX service provider returns before service operations are complete update: what's actually happening here is that when restarting a service, we stop it and send it into the stopping state, and then immediately try to start it. AIX will complain quietly at this, but still exit with 0 and ignore the start request. It seems that we need to wait a reasonable amount of time in the provider while restarting to start the service. Add Comment This message was sent by Atlassian JIRA (v6.4.11#64026-sha1:78f6ec4) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title William Hopper assigned an issue to William Hopper Puppet / PUP-5387 AIX service provider returns before service operations are complete Change By: William Hopper Assignee: William Hopper Add Comment This message was sent by Atlassian JIRA (v6.4.11#64026-sha1:78f6ec4) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title
William Hopper updated an issue
Puppet / PUP-5387
AIX service provider returns before service operations are complete
Change By:
William Hopper
Comment:
This one's a bit odd, and it doesn't seem to happen all the time. One way to somewhat reliably reproduce it is:{noformat}[0] [AIX] root@pe-aix-61-agent:~ # cat refresh_manifest.ppservice { 'puppet': }notify { 'refreshing': notify => Service['puppet'],}[0] [AIX] root@pe-aix-61-agent:~ # puppet apply -e "service { 'puppet': ensure => running }"[0] [AIX] root@pe-aix-61-agent:~ # puppet apply refresh_manifest.pp --debug --verbose...Info: /Stage[main]/Main/Notify[refreshing]: Scheduling refresh of Service[puppet]Debug: /Stage[main]/Main/Notify[refreshing]: The container Class[Main] will propagate my refresh eventDebug: Executing: '/usr/bin/lssrc -s puppet'Debug: Service puppet is activeDebug: Executing: '/usr/bin/lssrc -s puppet'Debug: Service puppet is activeDebug: Executing: '/usr/bin/lssrc -Ss puppet'Debug: Executing: '/usr/bin/stopsrc -s puppet'Debug: Executing: '/usr/bin/startsrc -s puppet'...[0] [AIX] root@pe-aix-61-agent:~ # lssrc -s puppetSubsystem GroupPID Status puppet inoperative[0] [AIX] root@pe-aix-61-agent:~ # puppet resource service puppetservice { 'puppet': ensure => 'stopped', enable => 'true',}{noformat}It seems that at least part of this issue is that sometimes {{startsrc -s puppet}} fails to actually start the service.While testing this out on another service, {{nfsd}}, I ran into a separate problem:{noformat}[0] [AIX] root@pe-aix-61-agent:~ # puppet resource service nfsdservice { 'nfsd': ensure => 'running', enable => 'false',}[0] [AIX] root@pe-aix-61-agent:~ # puppet apply ref_man.pp --debug --verboseInfo: /Stage[main]/Main/Notify[refreshing]: Scheduling refresh of Service[nfsd]Debug: /Stage[main]/Main/Notify[refreshing]: The container Class[Main] will propagate my refresh eventDebug: Executing: '/usr/bin/lssrc -s nfsd'Debug: Service nfsd is activeDebug: Executing: '/usr/bin/lssrc -s nfsd'Debug: Service nfsd is activeDebug: Executing: '/usr/bin/lssrc -Ss nfsd'Debug: Executing: '/usr/bin/stopsrc -s nfsd'Debug: Executing: '/usr/bin/startsrc -s nfsd'Notice: /Stage[main]/Main/Service[nfsd]: Triggered 'refresh' from 1 events...[0] [AIX] root@pe-aix-61-agent:~ # puppet resource service nfsdservice { 'nfsd': enable => 'false', # <=== note the lack of an 'ensure' attribute here}[0] [AIX] root@pe-aix-61-agent:~ # lssrc -s nfsdSubsystem GroupPID Status nfsd nfs 9568386 stopping # <== Puppet doesn't know about this state.[0] [AIX] root@pe-aix-61-agent:~ # lssrc -s nfsdSubsystem GroupPID Status nfsd nfs inoperative[0] [AIX] root@pe-aix-61-agent:~ # puppet resource service nfsdservice { 'nfsd': ensure => 'stopped', # <== we ended up being 'stopped' even though puppet ran 'startsrc' earlier. enable => 'false',}{noformat}A number of things went wrong here:1. Since the service started off running, this refresh should have resulted in {{nfsd}} being active, but instead it is {{inoperative}}. In the debug output we see that puppet did run {{startsrc -s puppet}}, but clearly the service was still in the process of stopping. I'm wo
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title William Hopper commented on PUP-5387 Re: AIX service provider returns before service operations are complete Investigating this further, I've realized that this issue is more complex than just the restart case, Stopping a service (at least on a slow AIX box, like our testing platforms) can cause a race condition in service status (see PUP-5406). Due to the asynchronous nature of service management in AIX, it seems like waiting for any service to stop (in the AIX service provider's stop method) might not be the right thing to do. What makes things worse is that the puppet service itself, which is what kicked off this whole chain of issues, can take anywhere from 0 to a 60 seconds to stop, depending on how slow the box is. Since this problem is entirely internal (it broke one of our tests, but no user has ever complained about it), I want to propose the following: 1) We fix the restart case for all services; that is, the AIX service restart method needs to be updated to not call start immediately after it called stop. A 30 second-or-so timeout there is probably sufficient on most machines. 2) As per a suggestion from Matthaus Owens, we could look into adding a shim service start / stop script for puppet which is a target of startsrc and stopsrc. This script would loop and wait on the service really being dead / alive like we do on other platforms. This would solve our internal acceptance problems and ensure the puppet service itself doesn't suffer from race conditions. Add Comment This message was sent by Atlassian JIRA (v6.4.11#64026-sha1:78f6ec4) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title William Hopper commented on PUP-5387 Re: AIX service provider returns before service operations are complete +1 (as Branan Riley) Add Comment This message was sent by Atlassian JIRA (v6.4.11#64026-sha1:78f6ec4) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title William Hopper updated an issue Puppet / PUP-5387 AIX service provider returns before service operations are complete Change By: William Hopper Comment: +1 (as [~branan]) Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title William Hopper commented on PUP-5387 Re: AIX service provider returns before service operations are complete And now (confusingly opposed to my previous proposition), I'm wondering if this is something that we should be fixing in our codebase at all. The asynchronous weirdness of AIX services is universal for all services - not just us. Perhaps what we're seeing is actually expected behavior, and it would be weird on our part to alter our startup to loop until finished? If this is the case, then maybe our best course of action would be to work on the tests to accommodate this, rather than somehow patching our own service to behave completely differently than any other service, or adding a timeout in the provider which doesn't guarantee the removal of all race conditions. All of this being said, my experience with AIX services is limited, so I will seek out more qualified opinions around this idea. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title William Hopper assigned an issue to Unassigned Puppet / PUP-5387 AIX service provider returns before service operations are complete Change By: William Hopper Assignee: William Hopper Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title William Hopper assigned an issue to William Hopper Puppet / PUP-5387 AIX service provider returns before service operations are complete Change By: William Hopper Assignee: William Hopper Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title William Hopper updated an issue Puppet / PUP-5387 AIX service provider returns before service operations are complete Change By: William Hopper Comment: And now (confusingly opposed to my previous proposition), I'm wondering if this is something that we should be fixing in our codebase at all. The asynchronous weirdness of AIX services is universal for all services - not just us. Perhaps what we're seeing is actually expected behavior, and it would be weird on our part to alter our startup to loop until finished?If this is the case, then maybe our best course of action would be to work on the tests to accommodate this, rather than somehow patching our own service to behave completely differently than any other service, *or* adding a timeout in the provider which doesn't guarantee the removal of all race conditions.All of this being said, my experience with AIX services is limited, so I will seek out more qualified opinions around this idea. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title Michael Smith assigned an issue to qa Puppet / PUP-5387 AIX service provider returns before service operations are complete Change By: Michael Smith Status: Ready for CI Test Assignee: qa Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title
John Duarte commented on PUP-5387
Re: AIX service provider returns before service operations are complete
Using pupet-agent at SHA 8be23474284a9f8e2bae551d266f5730af6681cd containing puppet at a068707, this passes manual review on AIX 5.3
[0] [AIX] root@pe-aix-53-agent:~ # /opt/puppetlabs/bin/puppet resource service puppet ensure=stopped
Notice: /Service[puppet]/ensure: ensure changed 'running' to 'stopped'
service { 'puppet':
ensure => 'stopped',
}
[0] [AIX] root@pe-aix-53-agent:~ # /opt/puppetlabs/bin/puppet resource service puppet
service { 'puppet':
ensure => 'stopped',
enable => 'true',
}
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title John Duarte assigned an issue to John Duarte Puppet / PUP-5387 AIX service provider returns before service operations are complete Change By: John Duarte Assignee: qa John Duarte Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5387) AIX service provider returns before service operations are complete
Title: Message Title William Hopper updated an issue Puppet / PUP-5387 AIX service provider returns before service operations are complete Change By: William Hopper Release Notes Summary: In AIX, service state transitions are not atomic. On slower systems, this could cause race conditions when starting, stopping or restarting services, as puppet did not wait for services to conclude their operations before continuing to apply different resources. The SRC service provider has been updated to wait up for up to 60 seconds when changing the state of a service. Release Notes: Bug Fix Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
