Issue #20561 has been updated by eric sorenson. Status changed from Needs Decision to Closed Priority changed from Urgent to Normal Keywords deleted ( customer)
Sorry, we don't have more releases scheduled for the 2.7 series, especially for changes like this which are pretty invasive and caused regressions against other parts of the code. If this is preventing your masters from compiling there is a very straightforward change you can make to get a true random splay with your current code version. Set your puppet daemons to `enable=>false` and instead use cron; there are many patterns/code described here: http://projects.puppetlabs.com/projects/1/wiki/Cron_Patterns ---------------------------------------- Bug #20561: 2nd puppet run after restart is ignoring runinterval, negating splay https://projects.puppetlabs.com/issues/20561#change-90610 * Author: Russell Harrison * Status: Closed * Priority: Normal * Assignee: eric sorenson * Category: agent * Target version: * Affected Puppet version: 2.7.21 * Keywords: * Branch: https://github.com/puppetlabs/puppet/pull/1130 ---------------------------------------- This bug is a MAJOR problem that needs to be fixed in the 2.7 line we do plan to go to 3.x but won't be able to for several months at a minimum. It preventing puppet from working in our environment at all. If I restart the puppet agents on my hosts the first run afterwards will succeed, however after that point everything syncs up. Now I've got 262 out of 354 nodes that failed to get a catalog at all! A 74% failure rate is absolutely cause for back porting this fix to 2.7. I don't know of any installs that can handle more than 100 clients requesting their catalog within a minute or two of each other. after upgrading from 2.7.13 to 2.7.14 I noticed that our puppets got "synchonized" and we have 10-20 clients running in one minute but 1-2 in other, which causes big spikes in load, especially if it hits few VMs on same physical machine It seems that second puppet run after splay ignores runinterval: May 13 05:00:03 blade308 puppet-agent[8820]: Sleeping for 346 seconds (splay is enabled) # start May 13 05:05:49 blade308 puppet-agent[8820]: Retrieving plugin # puppet run 5m after start May 13 05:10:02 blade308 puppet-agent[8820]: Finished catalog run in 106.23 seconds May 13 05:30:17 blade308 puppet-agent[8820]: Retrieving plugin # puppet run 25m after last one, 30m after start May 13 05:37:28 blade308 puppet-agent[8820]: Finished catalog run in 107.28 seconds May 14 05:00:04 blade308 puppet-agent[17967]: Sleeping for 1019 seconds (splay is enabled) May 14 05:17:03 blade308 puppet-agent[17967]: Retrieving plugin # puppet run 17 min after start May 14 05:21:03 blade308 puppet-agent[17967]: Finished catalog run in 99.93 seconds May 14 05:30:16 blade308 puppet-agent[17967]: Retrieving plugin # puoer run 13 min after last one 30m after start May 14 05:37:21 blade308 puppet-agent[17967]: Finished catalog run in 103.47 seconds It seems like 2nd run is counting time from puppet start, not from last run For comparision, before update: Apr 29 04:03:09 blade307 puppet-agent[31595]: Sleeping for 723 seconds (splay is enabled) Apr 29 04:15:12 blade307 puppet-agent[31595]: Retrieving plugin # 12m after start Apr 29 04:19:54 blade307 puppet-agent[31595]: Finished catalog run in 127.66 seconds Apr 29 04:50:21 blade307 puppet-agent[31595]: Retrieving plugin # 35m after last run, 47 after start. Apr 29 04:54:27 blade307 puppet-agent[31595]: Finished catalog run in 95.67 seconds I noticed that now it counts run interval from start of puppet, not from end of last run. While it makes puppet run in interval closer to defined run interval, it causes problems when for some case puppet will be ran at same time on many nodes. Before every "peak" in load would cause puppet run to take longer so next run would be delayed, causing 'peak time' to smooth out after few runs. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-bugs?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
