Hi, I'm proud to release a set of unofficial and certainly unstable (in various ways) patches against both 2.7.x and 3.0.x to improve the performance of the puppet compiler (and only the compiler).
Warning: I never tried to use those patches in production. Also those patches are almost not covered by tests (except those already in place). Those patches are not ready for general consumption and upstream merging (that's why I didn't publish a pull request), but I'd appreciate any feedback and tests on other sets of manifests to see the potential gains. My idea was to help PL dev team to produce a more performant puppet compiler in the future, and I hope those patches could be a good start :) If a pull request is a better format for commenting, please let me know. Those patches can be found here: https://github.com/masterzen/puppet/tree/performance-fixes-2.7 and https://github.com/masterzen/puppet/tree/performance-fixes-3.0 The methodology I used was using the perftools ruby profiler as Daniel suggested in a previous post, with the set of manifests (back from 0.24 era) I use in production. Those manifests have the following particularity: they are old and ancient and use almost no new Puppet feature, like paramterized class, but abuse templates, definitions. Those manifests contains around 100 modules. I also have some manually imported definitions that are heavily used (ie for concatenation for instance). I also have tons of classes not following the autoloader path scheme. So, for sure my optimizations are heavily tailored to my use case and might not give any improvement to any other cases (although I'd be surprised if compilation time could get worst with those patches). The biggest saver for me was to prevent touching the disk during the compilation (ie not try to autoload constantly or not try to autoload something that can't be autoloaded). For my tests, I had storeconfigs off (even though the manifests are collecting exported resources) as this was more convenient for me. Here are the results on my manifests (note, the first run includes manifest parsing which is something I didn't yet profile/optimize, the other run are without manifests modifications): 3.0 --- before: Compiled catalog for corp1 in environment production in 5.38 seconds Compiled catalog for corp1 in environment production in 3.01 seconds Compiled catalog for corp1 in environment production in 2.96 seconds Compiled catalog for corp1 in environment production in 2.93 seconds Compiled catalog for corp1 in environment production in 2.89 seconds after: Compiled catalog for corp1 in environment production in 4.12 seconds Compiled catalog for corp1 in environment production in 1.54 seconds Compiled catalog for corp1 in environment production in 1.51 seconds Compiled catalog for corp1 in environment production in 1.47 seconds Compiled catalog for corp1 in environment production in 1.46 seconds 2.7 --- before: Compiled catalog for corp1 in environment production in 5.10 seconds Compiled catalog for corp1 in environment production in 2.72 seconds Compiled catalog for corp1 in environment production in 2.71 seconds Compiled catalog for corp1 in environment production in 2.69 seconds Compiled catalog for corp1 in environment production in 2.68 seconds after: Compiled catalog for corp1 in environment production in 4.62 seconds Compiled catalog for corp1 in environment production in 1.46 seconds Compiled catalog for corp1 in environment production in 1.49 seconds Compiled catalog for corp1 in environment production in 1.42 seconds Compiled catalog for corp1 in environment production in 1.41 seconds As you can see, the speed gain is about 50%, which is not (that) bad. On a side note: I had a lot of test failure on 3.0.x with spec/unit/provider_spec.rb that looks like this: 223) Puppet::Provider features supports_parameter? should support two_features with provider two Failure/Error: type.feature :numeric, '', :methods => [:one, :two] NoMethodError: undefined method `feature' for nil:NilClass # ./spec/unit/provider_spec.rb:518 Running the provider_spec by itself doesn't exhibit those failures. Thanks for listening :) Please comment :) -- Brice Figureau My Blog: http://www.masterzen.fr/ -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to puppet-dev@googlegroups.com. To unsubscribe from this group, send email to puppet-dev+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
