This was using ruby 1.8.7 btw as that is what we currently use in
production.
We restart our puppetmasters though after every compile
(PassengerMaxRequests 1 on catalog requests) as that is the only way to get
around issues #17210 and #12173. So until they are solved we cannot cache
stuff like this from one compile to the next.
With the old parser it was a bit faster though, but not fast enough really:
user system total real
require 0.440000 0.110000 0.550000 ( 0.739670)
init 0.020000 0.000000 0.020000 ( 0.014703)
import 12.110000 0.330000 12.440000 ( 13.083297)
Stats
Known resource classes: 1034
Known resource defines: 113
Known resource nodes: 0
Manifests loaded: 1141
On 27 August 2013 20:28, Andy Parker <a...@puppetlabs.com> wrote:
> On Tue, Aug 27, 2013 at 1:02 PM, Erik Dalén
> <erik.gustav.da...@gmail.com>wrote:
>
>> ruby ./load_the_world.rb --parser future
>> user system total real
>> require 0.400000 0.060000 0.460000 ( 0.486126)
>> init 0.010000 0.000000 0.010000 ( 0.010955)
>> import 38.580000 0.280000 38.860000 ( 38.970678)
>> Stats
>> Known resource classes: 1034
>> Known resource defines: 113
>> Known resource nodes: 0
>> Manifests loaded: 1141
>>
>> If this is going to add 39s to every compile it won't be very good for
>> us. (this is not exactly the same hardware as most puppetmasters we have
>> though, but on the other hand they do many simultaneous compiles).
>>
>>
> Yeah, that wouldn't be good. I ran --parser future locally on my little
> set of testing data and found that --parser future is 20-30x slower than
> the current parser.
>
> Dalen, could you try the same thing, but without --parser future? I'd be
> interested if you see the same difference.
>
> Also, I don't think this would add 39s to every compile on the
> puppetmaster. It would only do that if it needs to reload. Based on earlier
> discussion, I think this would be best done from a signal. Even then there
> are techniques we could use to offload that (pre-parse and load serialized
> AST, for instance). This time would, currently, likely show up in puppet
> apply runs. I don't think it would be a good idea to cause puppet apply for
> testing purposes to jump from 1 or 2 seconds to 39+.
>
>
>> On the other hand we are working on a system to create dynamic per node
>> environments that only include the modules that node needs, so then the
>> difference might not be that big compared to the current method.
>>
>>
>>
>> On 27 August 2013 17:48, Andy Parker <a...@puppetlabs.com> wrote:
>>
>>> On Mon, Aug 26, 2013 at 4:13 PM, Joshua Hoblitt <jhobl...@cpan.org>wrote:
>>>
>>>> On 08/26/2013 11:41 AM, Andy Parker wrote:
>>>> > How would you handle reloading in this scenario?
>>>> >
>>>> >
>>>> > I can think of two ways: track all of the files loaded and reload when
>>>> > something changes or is added, reload only when told. I lean towards
>>>> the
>>>> > "reload only when told" for a production master, since then the
>>>> runtime
>>>> > behavior of it is much more predictable in time (no more stating) and
>>>> in
>>>> > catalogs produced (no more ending up with half of one manifest version
>>>> > and half of another).
>>>>
>>>> I think the "load everything" and "only reload when told to" approach is
>>>> the correct one for the master. I suspect it will solve the random
>>>> classes disappearing while pushing an update problem I've seen in the
>>>> wild (but can't reproduce on demand) and definitely the fall out I see
>>>> when trying to change the git branch for an entire environment. This is
>>>> a very logical workflow and one many folks are already accustomed to
>>>> from HUPing bind, etc.
>>>>
>>>> I'd advocate caution about making "greedy loading" the default for local
>>>> apply. Syntastic can already be painful slow. :(
>>>>
>>>>
>>> Yeah, I think if it turns out that greedy loading is prohibitively
>>> expensive our best bet is to a) make the parser faster, b) expose parts of
>>> the code better so that specific use cases can be done more efficiently,
>>> and only after that c) put laziness back in.
>>>
>>> In order to answer the question of how long import everything would
>>> take, I've put together a simple script that I think anyone can run to get
>>> some information. It would be great if some people with large sets of
>>> manifests could try running this and see what kinds of numbers we get. For
>>> example this is what I get
>>>
>>> [10:46:17][Ruby(ruby-1.9.3-p392)][Git(master)] andy:puppet
>>> > be ./load_the_world.rb --parser future
>>> user system total real
>>> require 0.530000 0.100000 0.630000 ( 0.634259)
>>> init 0.000000 0.000000 0.000000 ( 0.005957)
>>> import 0.310000 0.020000 0.330000 ( 0.326274)
>>> Stats
>>> Known resource classes: 4
>>> Known resource defines: 1
>>> Known resource nodes: 0
>>> Manifests loaded: 6
>>> Manifests:
>>> /Users/andy/.puppet/manifests/site.pp
>>> /Users/andy/.puppet/modules/manifest1/manifests/foo.pp
>>> /Users/andy/.puppet/modules/manifest1/manifests/fool.pp
>>> /Users/andy/.puppet/modules/manifest1/manifests/init.pp
>>> /Users/andy/.puppet/modules/manifest2/manifests/init.pp
>>> /Users/andy/.puppet/modules/myfile/manifests/init.pp
>>>
>>> You can also pass it any parameters (such as environment or --parser
>>> future)
>>>
>>> Please run it, do any scrubbing of sensitive data you want and send back
>>> the results.
>>>
>>>
>>>> -Josh
>>>>
>>>> --
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Puppet Developers" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to puppet-dev+unsubscr...@googlegroups.com.
>>>> To post to this group, send email to puppet-dev@googlegroups.com.
>>>> Visit this group at http://groups.google.com/group/puppet-dev.
>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>
>>>
>>>
>>>
>>> --
>>> Andrew Parker
>>> a...@puppetlabs.com
>>> Freenode: zaphod42
>>> Twitter: @aparker42
>>> Software Developer
>>>
>>> *Join us at PuppetConf 2014, September 23-24 in San Francisco*
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Puppet Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to puppet-dev+unsubscr...@googlegroups.com.
>>> To post to this group, send email to puppet-dev@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/puppet-dev.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>
>>
>> --
>> Erik Dalén
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Puppet Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to puppet-dev+unsubscr...@googlegroups.com.
>> To post to this group, send email to puppet-dev@googlegroups.com.
>> Visit this group at http://groups.google.com/group/puppet-dev.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
>
> --
> Andrew Parker
> a...@puppetlabs.com
> Freenode: zaphod42
> Twitter: @aparker42
> Software Developer
>
> *Join us at PuppetConf 2014, September 23-24 in San Francisco*
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-dev+unsubscr...@googlegroups.com.
> To post to this group, send email to puppet-dev@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-dev.
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
Erik Dalén
--
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-dev+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-dev@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev.
For more options, visit https://groups.google.com/groups/opt_out.
