[Puppet Users] 1.8.7 => 1.9.3 migration, compilation times raised twice!?
i've made a migration on our puppetmasters yesterday, hoping it will run faster and/or make less load on machines with newer ruby. but actual result is just opposite, our compilation times went up a lot, ie: Nov 27 14:30:12 punch puppet-master[15293]: Compiled catalog for lb1 in environment production in 183.42 seconds Nov 27 15:38:42 punch puppet-master[24399]: Compiled catalog for lb1 in environment production in 158.49 seconds Nov 27 15:51:19 punch puppet-master[11963]: Compiled catalog for lb1 in environment production in 175.00 seconds Nov 27 16:04:15 punch puppet-master[15264]: Compiled catalog for lb1 in environment production in 120.36 seconds Nov 27 19:25:52 punch puppet-master[552]: Compiled catalog for lb1 in environment production in 281.54 seconds Nov 27 20:20:19 punch puppet-master[2273]: Compiled catalog for lb1 in environment production in 239.05 seconds Nov 27 22:26:03 punch puppet-master[4919]: Compiled catalog for lb1 in environment production in 275.38 seconds Nov 28 00:55:24 punch puppet-master[12293]: Compiled catalog for lb1 in environment production in 267.91 seconds guess when migration has occured. this machine (lb1) has ~2000 resources. i'm a bit worried about scalability... we can go back to 1.8.7 now, but it won't be supported in 7 months (ie. no security updates). -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/NV-H40aVfrcJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] 1.8.7 => 1.9.3 migration, compilation times raised twice!?
On Wed, Nov 28, 2012 at 10:34 AM, asq wrote: > i've made a migration on our puppetmasters yesterday, hoping it will run > faster and/or make less load on machines with newer ruby. but actual result > is just opposite, our compilation times went up a lot, ie: > > Nov 27 14:30:12 punch puppet-master[15293]: Compiled catalog for lb1 in > environment production in 183.42 seconds > Nov 27 15:38:42 punch puppet-master[24399]: Compiled catalog for lb1 in > environment production in 158.49 seconds > Nov 27 15:51:19 punch puppet-master[11963]: Compiled catalog for lb1 in > environment production in 175.00 seconds > Nov 27 16:04:15 punch puppet-master[15264]: Compiled catalog for lb1 in > environment production in 120.36 seconds > Nov 27 19:25:52 punch puppet-master[552]: Compiled catalog for lb1 in > environment production in 281.54 seconds > Nov 27 20:20:19 punch puppet-master[2273]: Compiled catalog for lb1 in > environment production in 239.05 seconds > Nov 27 22:26:03 punch puppet-master[4919]: Compiled catalog for lb1 in > environment production in 275.38 seconds > Nov 28 00:55:24 punch puppet-master[12293]: Compiled catalog for lb1 in > environment production in 267.91 seconds > > guess when migration has occured. this machine (lb1) has ~2000 resources. > > i'm a bit worried about scalability... we can go back to 1.8.7 now, but it > won't be supported in 7 months (ie. no security updates). > > What version of Puppet are you on? Where did you get your 1.9.3? > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/puppet-users/-/NV-H40aVfrcJ. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] 1.8.7 => 1.9.3 migration, compilation times raised twice!?
W dniu środa, 28 listopada 2012 22:57:40 UTC+1 użytkownik Michael Stanhke napisał: > > What version of Puppet are you on? Where did you get your 1.9.3? > 3.0.1 and 1.9.3p327 from rvm. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/t2QS0K1gkK0J. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] 1.8.7 => 1.9.3 migration, compilation times raised twice!?
On Thu, Nov 29, 2012 at 12:24 AM, asq wrote: > W dniu środa, 28 listopada 2012 22:57:40 UTC+1 użytkownik Michael Stanhke > napisał: >> >> What version of Puppet are you on? Where did you get your 1.9.3? > > 3.0.1 and 1.9.3p327 from rvm. Ouch, do you happen to have a large number of ruby Gem's installed? -Jeff -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] 1.8.7 => 1.9.3 migration, compilation times raised twice!?
W dniu czwartek, 29 listopada 2012 17:42:48 UTC+1 użytkownik Jeff McCune napisał: > > On Thu, Nov 29, 2012 at 12:24 AM, asq > > wrote: > > W dniu środa, 28 listopada 2012 22:57:40 UTC+1 użytkownik Michael > Stanhke > > napisał: > >> > >> What version of Puppet are you on? Where did you get your 1.9.3? > > > > 3.0.1 and 1.9.3p327 from rvm. > > Ouch, do you happen to have a large number of ruby Gem's installed? > > -Jeff > yes, a bunch of gems for Foreman... <5>root@punch(8)~$ gem stale | wc -l 91 but it's the same pack that we've used for 1.8.7. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/JJhz2RiMjBIJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] 1.8.7 => 1.9.3 migration, compilation times raised twice!?
On Thu, Nov 29, 2012 at 8:56 AM, asq wrote: > W dniu czwartek, 29 listopada 2012 17:42:48 UTC+1 użytkownik Jeff McCune > napisał: > >> On Thu, Nov 29, 2012 at 12:24 AM, asq wrote: >> > W dniu środa, 28 listopada 2012 22:57:40 UTC+1 użytkownik Michael >> Stanhke >> > napisał: >> >> >> >> What version of Puppet are you on? Where did you get your 1.9.3? >> > >> > 3.0.1 and 1.9.3p327 from rvm. >> >> Ouch, do you happen to have a large number of ruby Gem's installed? >> >> -Jeff >> > > yes, a bunch of gems for Foreman... > It might not be the cause, but one related change we made in Puppet 3 is that we scan Ruby gem's and will load extensions contained within them. This is #7788 [1] This change in behavior could be causing the Puppet autoloader to perform a _lot_ of additional stat() system calls. Could you run strace, truss, or dtrace and compare the number of system calls between Puppet 2.7 and 3.0 in your environment? [1] http://projects.puppetlabs.com/issues/7788 -Jeff -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] 1.8.7 => 1.9.3 migration, compilation times raised twice!?
W dniu czwartek, 29 listopada 2012 18:23:23 UTC+1 użytkownik Jeff McCune napisał: > > It might not be the cause, but one related change we made in Puppet 3 is > that we scan Ruby gem's and will load extensions contained within them. > This is #7788 [1] This change in behavior could be causing the Puppet > autoloader to perform a _lot_ of additional stat() system calls. Could you > run strace, truss, or dtrace and compare the number of system calls between > Puppet 2.7 and 3.0 in your environment? not sure if it's about 2.7 => 3.0 migration, as we did it month ago with no aparent degradation (but also no such load decrease as on some slides from PC). now we just switched to newer ruby and passenger (1.8.7 => 1.9.3p327 and 3.0.2 => 3.0.17). i'll try to do some tests tomorrow but i need to get both instances up at the same server. another thing that is involved (and suspected) is hiera, as compilation time deltas differ on different groups of nodes, and it might be related to how much of modules with hiera lookups are included on mostly affected nodes (ie. 120s => 240s on our load balancers). another thing is puppetdb, but nodes that collect resources, which should impose highest load on puppetdb are not the ones which are much affected (ie. 320s => 360s increase on our nagios server). is it possible to do evaltrace on puppetmaster? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/ShLDKg4XRUkJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
