This was using ruby 1.8.7 btw as that is what we currently use in
production.
We restart our puppetmasters though after every compile
(PassengerMaxRequests 1 on catalog requests) as that is the only way to get
around issues #17210 and #12173. So until they are solved we cannot cache
stuff like this from one compile to the next.
With the old parser it was a bit faster though, but not fast enough really:
user system total real
require 0.440000 0.110000 0.550000 ( 0.739670)
init 0.020000 0.000000 0.020000 ( 0.014703)
import 12.110000 0.330000 12.440000 ( 13.083297)
Stats
Known resource classes: 1034
Known resource defines: 113
Known resource nodes: 0
Manifests loaded: 1141
On 27 August 2013 20:28, Andy Parker <[email protected]> wrote:
> On Tue, Aug 27, 2013 at 1:02 PM, Erik Dalén
> <[email protected]>wrote:
>
>> ruby ./load_the_world.rb --parser future
>> user system total real
>> require 0.400000 0.060000 0.460000 ( 0.486126)
>> init 0.010000 0.000000 0.010000 ( 0.010955)
>> import 38.580000 0.280000 38.860000 ( 38.970678)
>> Stats
>> Known resource classes: 1034
>> Known resource defines: 113
>> Known resource nodes: 0
>> Manifests loaded: 1141
>>
>> If this is going to add 39s to every compile it won't be very good for
>> us. (this is not exactly the same hardware as most puppetmasters we have
>> though, but on the other hand they do many simultaneous compiles).
>>
>>
> Yeah, that wouldn't be good. I ran --parser future locally on my little
> set of testing data and found that --parser future is 20-30x slower than
> the current parser.
>
> Dalen, could you try the same thing, but without --parser future? I'd be
> interested if you see the same difference.
>
> Also, I don't think this would add 39s to every compile on the
> puppetmaster. It would only do that if it needs to reload. Based on earlier
> discussion, I think this would be best done from a signal. Even then there
> are techniques we could use to offload that (pre-parse and load serialized
> AST, for instance). This time would, currently, likely show up in puppet
> apply runs. I don't think it would be a good idea to cause puppet apply for
> testing purposes to jump from 1 or 2 seconds to 39+.
>
>
>> On the other hand we are working on a system to create dynamic per node
>> environments that only include the modules that node needs, so then the
>> difference might not be that big compared to the current method.
>>
>>
>>
>> On 27 August 2013 17:48, Andy Parker <[email protected]> wrote:
>>
>>> On Mon, Aug 26, 2013 at 4:13 PM, Joshua Hoblitt <[email protected]>wrote:
>>>
>>>> On 08/26/2013 11:41 AM, Andy Parker wrote:
>>>> > How would you handle reloading in this scenario?
>>>> >
>>>> >
>>>> > I can think of two ways: track all of the files loaded and reload when
>>>> > something changes or is added, reload only when told. I lean towards
>>>> the
>>>> > "reload only when told" for a production master, since then the
>>>> runtime
>>>> > behavior of it is much more predictable in time (no more stating) and
>>>> in
>>>> > catalogs produced (no more ending up with half of one manifest version
>>>> > and half of another).
>>>>
>>>> I think the "load everything" and "only reload when told to" approach is
>>>> the correct one for the master. I suspect it will solve the random
>>>> classes disappearing while pushing an update problem I've seen in the
>>>> wild (but can't reproduce on demand) and definitely the fall out I see
>>>> when trying to change the git branch for an entire environment. This is
>>>> a very logical workflow and one many folks are already accustomed to
>>>> from HUPing bind, etc.
>>>>
>>>> I'd advocate caution about making "greedy loading" the default for local
>>>> apply. Syntastic can already be painful slow. :(
>>>>
>>>>
>>> Yeah, I think if it turns out that greedy loading is prohibitively
>>> expensive our best bet is to a) make the parser faster, b) expose parts of
>>> the code better so that specific use cases can be done more efficiently,
>>> and only after that c) put laziness back in.
>>>
>>> In order to answer the question of how long import everything would
>>> take, I've put together a simple script that I think anyone can run to get
>>> some information. It would be great if some people with large sets of
>>> manifests could try running this and see what kinds of numbers we get. For
>>> example this is what I get
>>>
>>> [10:46:17][Ruby(ruby-1.9.3-p392)][Git(master)] andy:puppet
>>> > be ./load_the_world.rb --parser future
>>> user system total real
>>> require 0.530000 0.100000 0.630000 ( 0.634259)
>>> init 0.000000 0.000000 0.000000 ( 0.005957)
>>> import 0.310000 0.020000 0.330000 ( 0.326274)
>>> Stats
>>> Known resource classes: 4
>>> Known resource defines: 1
>>> Known resource nodes: 0
>>> Manifests loaded: 6
>>> Manifests:
>>> /Users/andy/.puppet/manifests/site.pp
>>> /Users/andy/.puppet/modules/manifest1/manifests/foo.pp
>>> /Users/andy/.puppet/modules/manifest1/manifests/fool.pp
>>> /Users/andy/.puppet/modules/manifest1/manifests/init.pp
>>> /Users/andy/.puppet/modules/manifest2/manifests/init.pp
>>> /Users/andy/.puppet/modules/myfile/manifests/init.pp
>>>
>>> You can also pass it any parameters (such as environment or --parser
>>> future)
>>>
>>> Please run it, do any scrubbing of sensitive data you want and send back
>>> the results.
>>>
>>>
>>>> -Josh
>>>>
>>>> --
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Puppet Developers" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>> Visit this group at http://groups.google.com/group/puppet-dev.
>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>
>>>
>>>
>>>
>>> --
>>> Andrew Parker
>>> [email protected]
>>> Freenode: zaphod42
>>> Twitter: @aparker42
>>> Software Developer
>>>
>>> *Join us at PuppetConf 2014, September 23-24 in San Francisco*
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Puppet Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To post to this group, send email to [email protected].
>>> Visit this group at http://groups.google.com/group/puppet-dev.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>
>>
>> --
>> Erik Dalén
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Puppet Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> Visit this group at http://groups.google.com/group/puppet-dev.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
>
> --
> Andrew Parker
> [email protected]
> Freenode: zaphod42
> Twitter: @aparker42
> Software Developer
>
> *Join us at PuppetConf 2014, September 23-24 in San Francisco*
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/puppet-dev.
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
Erik Dalén
--
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/puppet-dev.
For more options, visit https://groups.google.com/groups/opt_out.
