date:20140721

Jira (PUP-1965) Clients are hung when server has intermittent service

2014-07-21 Thread Mike Fitzgerald (JIRA)

Title: Message Title

Mike Fitzgerald commented on an issue

Re: Clients are hung when server has intermittent service

Thank you for the detailed reply.
I'm running ruby 1.8.7 (2011-06-30 patchlevel 352)
The behaviour is occurring on puppet runs that have no updates to the configuration/catalog - so there are no large files being transferred.
Here is an example of an agent process, and a stuck 'applying configuration' process (running since yesterday) root 2200 0.0 0.1 127444 49436 ? S Jul21 0:00 puppet agent: applying configuration root 30667 0.0 0.1 114712 45568 ? Ss Jul18 0:09 /usr/bin/ruby1.8 /usr/local/bin/puppet agent
Attaching strace to process 30667 (puppet agent): wait4(2200,
Attaching strace to process 2200 (sits here forever): select(5, [4], NULL, NULL
If I attach strace to the puppet agent process (following any child process), and kill the stuck 'applying configuration' process, it will spawn another which runs through the puppet configuration successfully. Because of this, I am unable to reproduce the problem with strace attached.

Add Comment

Puppet / PUP-1965

Clients are hung when server has intermittent service

Unrelated server in data center (same subnet) comes under DOS attack. Network service to other machines in subnet is slow, sparse and subject to failure during this time. All puppet clients that attempt to connect / update during this period are hung. Process list shows puppet agent service process and 'puppet agent: applying configuration' process. ...

Jira (PUP-2831) Puppet does not work with RGen 0.7.0

2014-07-21 Thread Henrik Lindberg (JIRA)

Title: Message Title










 

 Henrik Lindberg updated an issue











 






 Puppet /  PUP-2831



  Puppet does not work with RGen 0.7.0 










Change By:

 Henrik Lindberg




Assignee:

 Henrik Lindberg Charlie Sharpsteen












   

 Add Comment











 










 This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede)




 














-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.

Jira (PUP-2891) creating a resource override for a Class fails (future parser)

2014-07-21 Thread Andy Parker (JIRA)

Title: Message Title










 

 Andy Parker assigned an issue to Andy Parker











 






 Puppet /  PUP-2891



  creating a resource override for a Class fails (future parser) 










Change By:

 Andy Parker




Assignee:

 Andy Parker












   

 Add Comment











 










 This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede)




 














-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.

Jira (PUP-2886) future parser should validate class/resource names that are types

2014-07-21 Thread Andy Parker (JIRA)

Title: Message Title










 

 Andy Parker assigned an issue to Andy Parker











 






 Puppet /  PUP-2886



  future parser should validate class/resource names that are types 










Change By:

 Andy Parker




Assignee:

 Andy Parker












   

 Add Comment











 










 This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede)




 














-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.

Jira (PUP-1965) Clients are hung when server has intermittent service

2014-07-21 Thread Henrik Lindberg (JIRA)

Title: Message Title










 

 Henrik Lindberg commented on an issue











 






  Re: Clients are hung when server has intermittent service 










With a bit of digging, it turns out that it likely reads in chunks of 16k, and thus a 1Mb transfer can max take ~2hours. It may be hard to distinguish that from a hung agent. It would be valuable to try to figure out if the agent is receiving at all, or if it is indeed hung by tracing it at the system level.












   

 Add Comment











 













 Puppet /  PUP-1965



  Clients are hung when server has intermittent service 







 Unrelated server in data center (same subnet) comes under DOS attack. Network service to other machines in subnet is slow, sparse and subject to failure during this time. All puppet clients that attempt to connect / update during this period are hung.   Process list shows puppet agent service process and 'puppet agent: applying configuration' process.  ...















 This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede)




 














-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this grou

Jira (FACT-569) Run Facter acceptance against ruby 2.0 x64

2014-07-21 Thread Andy Parker (JIRA)

Title: Message Title










 

 Andy Parker assigned an issue to Andy Parker











 






 Facter /  FACT-569



  Run Facter acceptance against ruby 2.0 x64 










Change By:

 Andy Parker




Assignee:

 Andy Parker












   

 Add Comment











 










 This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede)




 














-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.

Jira (PUP-1965) Clients are hung when server has intermittent service

2014-07-21 Thread Henrik Lindberg (JIRA)

Title: Message Title

Henrik Lindberg commented on an issue

Re: Clients are hung when server has intermittent service

Hm, that suggests that timeouts are not working; if they did the agent should timeout a read after 2 minutes of no activity and then fail. Maybe it manages to get a read through every 2 minutes when transferring a larger file, and thus will take a very long time to complete a file transfer. Do you have any / many larger files being transferred?
I am not 100% sure how the read_timeout in Ruby Net::HTTP translates to packet reads, if we assume this is at the Ethernet MTU level (i.e. ~1500 bytes), the the max transfer time of 1 Mb (with a 2 min read timeout) would be something like 23 hours. It is unclear when reading the Ruby docs if the read_timeout applies to per packet (as HTTP Session suggests), or to an entire read (the Net::HTTP simply states that it is the read_timeout without further explanation). It is also unclear if the timeout is per the underlying media packet size, or the size requested from the socket (consisting of potentially many media packets over Ethernet). It is unclear if the Ruby implementation relies on the socket option SO_RCVTIMEO (which seems to not be supported on all platforms).
It may be that we run into this: http://stackoverflow.com/questions/9853516/set-socket-timeout-in-ruby-via-so-rcvtimeo-socket-option (again, I am not sure if what is stated there is correct "timeout does not work under Ruby 1.9", that seems pretty major, and is now perhaps fixed). It raises the questions though regarding which version of Ruby that is used (on agents), since we do see timeouts occurring in our test environments (we are using 1.9.3-p194 or later).
Unfortunately, until 3.7, the read_timeout was also applied to the overall transfer time of all plugins. That issue meant that plugins that took more than 2 minutes to complete in total failed. Thus if you try to set the timeout to a shorter value (to fail if the transfer is going to take a very long time) you may always fail in versions before 3-7 because there is not enough time to complete the plugin-sync.
I guess that was a long way of asking which Ruby version the hanging agents are using.
More information about what the hanging agent is doing can be obtained by running strace, try to look for the call to setsockopt to see what options have been set, and what the process is hanging on; a recvfrom or ppoll or something like that. (It is not meaningful to send an strace for the entire agent run as it will potentially be huge).

61 matches

Mail list logo