Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

> I can't stress how strongly I (we?) discourage the use of Ruby 1.8.6. =) Yeah, this is a backported version from one of the fedoras for EL5... > We (Puppet Labs) really don't do any testing; quality, performance, or > otherwise, against this version of Ruby. > > I'm not saying you'll see a hu

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Wed, May 23, 2012 at 6:00 AM, Peter Meier wrote: > + beginning_time = Time.now Puppet.info "Caching #{self.name} for #{request.key}" cache.save request(:save, result, *args) + Puppet.debug "Caching catalog time: #{(Time.now - beginning_time)}" >>>

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

+      beginning_time = Time.now      Puppet.info "Caching #{self.name} for #{request.key}"      cache.save request(:save, result, *args) +      Puppet.debug "Caching catalog time: #{(Time.now - beginning_time)}" Here're my results for a catalog of ~2000 resources, some of which are tidys of big

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

>> Which version of Ruby and Puppet ? >> >> 'Config retrieval time' include the 'caching catalog time' (write >> catalog on >> disk in YAML format). Your catalog is pretty big, so caching could be >> very >> slow. >> You can check this by add theses lines in >> lib/puppet/indirector/indirection.rb

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Tue, May 15, 2012 at 1:25 AM, James Turnbull wrote: > Brice Figureau wrote: >> On Tue, 2012-05-15 at 00:43 -0700, James Turnbull wrote: > Just getting rid of that yaml serialization -- or switching to JSON -- > would make a huge difference. > I'll sound like a broken record, bu

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Le 15/05/2012 10:25, James Turnbull a écrit : Brice Figureau wrote: On Tue, 2012-05-15 at 00:43 -0700, James Turnbull wrote: Just getting rid of that yaml serialization -- or switching to JSON -- would make a huge difference. I'll sound like a broken record, but that'd be awesome if we could

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

We're already always sending over JSON - we're just storing in YAML on the client. This is my stupidity in the beginning; I just couldn't get it done with the current model of how we do caching. On May 15, 2012, at 10:09 AM, Trevor Vaughan wrote: > Just out of curiosity, since all clients repo

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Exactly - the problem is that we're using YAML for a potentially very large file, and definitely for a very complex data structure with lots of internal references. That problem results in lots of symptoms, including huge memory consumption and painful slowness. On May 15, 2012, at 10:34 AM, A

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

That ticket (#2892) is about the amount of memory being used during conversion. Is that still the problem? Or is the problem really now centered around the amount of time that the conversion is taking? Either way using JSON all the way through would probably fix both problems. On May 15, 2012, at

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Just out of curiosity, since all clients report back to the server what version of Puppet and Ruby version they are running, couldn't you dynamically select a serialization format based on optimization factors? Marshal -> PSON -> YAML Just curious. Thanks, Trevor On Tue, May 15, 2012 at 4:25 A

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Brice Figureau wrote: > On Tue, 2012-05-15 at 00:43 -0700, James Turnbull wrote: Just getting rid of that yaml serialization -- or switching to JSON -- would make a huge difference. >>> I'll sound like a broken record, but that'd be awesome if we could >>> resurrect #2892 from the gr

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Tue, 2012-05-15 at 00:43 -0700, James Turnbull wrote: > >> Just getting rid of that yaml serialization -- or switching to JSON -- > >> would make a huge difference. > >> > > > > I'll sound like a broken record, but that'd be awesome if we could > > resurrect #2892 from the grave: > > https://pr

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

>> Just getting rid of that yaml serialization -- or switching to JSON -- >> would make a huge difference. >> > > I'll sound like a broken record, but that'd be awesome if we could > resurrect #2892 from the grave: > https://projects.puppetlabs.com/issues/2892 > Didn't that get supplanted by: h

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Mon, 2012-05-14 at 15:39 -0700, Luke Kanies wrote: > On May 14, 2012, at 2:22 PM, Nick Lewis wrote: > > > On Monday, May 14, 2012 at 2:19 PM, Trevor Vaughan wrote: > > > Any suggestions on where to benchmark here? I tried to figure out > > > where to hit it but didn't really get anywhere with i

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On May 14, 2012, at 2:22 PM, Nick Lewis wrote: > On Monday, May 14, 2012 at 2:19 PM, Trevor Vaughan wrote: >> Any suggestions on where to benchmark here? I tried to figure out >> where to hit it but didn't really get anywhere with it. >> >> I did try using pure ruby to load the catalog from disk

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Monday, May 14, 2012 at 2:29 PM, Trevor Vaughan wrote: > Yes, this does make sense. > > Mystery solved I suppose. > > But, in that case, couldn't you skip that last phase if you just pass > YAML the whole way and cut off a third of the load time? > Yes, or more ideally, we would store the cat

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Yes, this does make sense. Mystery solved I suppose. But, in that case, couldn't you skip that last phase if you just pass YAML the whole way and cut off a third of the load time? Trevor On Mon, May 14, 2012 at 5:22 PM, Nick Lewis wrote: > On Monday, May 14, 2012 at 2:19 PM, Trevor Vaughan wro

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Monday, May 14, 2012 at 2:19 PM, Trevor Vaughan wrote: > Any suggestions on where to benchmark here? I tried to figure out > where to hit it but didn't really get anywhere with it. > > I did try using pure ruby to load the catalog from disk using the > Puppet methods and it took 7 seconds in a

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Any suggestions on where to benchmark here? I tried to figure out where to hit it but didn't really get anywhere with it. I did try using pure ruby to load the catalog from disk using the Puppet methods and it took 7 seconds in a case where Config retrieval is reporting around 40 seconds. Thanks,

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Mon, May 14, 2012 at 10:21 AM, Luke Kanies wrote: > On May 14, 2012, at 9:53 AM, Daniel Pittman wrote: >> On Mon, May 14, 2012 at 9:49 AM, Luke Kanies wrote: >>> On May 14, 2012, at 9:46 AM, Daniel Pittman wrote: On Mon, May 14, 2012 at 2:11 AM, Emile MOREL wrote: > Daniel Pittman a

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On May 14, 2012, at 9:53 AM, Daniel Pittman wrote: > On Mon, May 14, 2012 at 9:49 AM, Luke Kanies wrote: >> On May 14, 2012, at 9:46 AM, Daniel Pittman wrote: >>> On Mon, May 14, 2012 at 2:11 AM, Emile MOREL wrote: Daniel Pittman a écrit : > It would be great to improve that, from

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

- Original Message - > From: "Daniel Pittman" > To: puppet-dev@googlegroups.com > Sent: Monday, May 14, 2012 5:53:29 PM > Subject: Re: [Puppet-dev] Trying to isolate performance issues with config > retrieval. > > On Mon, May 14, 2012 at 9:49 AM, Luke

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Mon, May 14, 2012 at 9:49 AM, Luke Kanies wrote: > On May 14, 2012, at 9:46 AM, Daniel Pittman wrote: >> On Mon, May 14, 2012 at 2:11 AM, Emile MOREL wrote: >>> Daniel Pittman a écrit : >>> It would be great to improve that, from our point of view, but I wonder a bit: when is this a

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On May 14, 2012, at 9:46 AM, Daniel Pittman wrote: > On Mon, May 14, 2012 at 2:11 AM, Emile MOREL wrote: >> Daniel Pittman a écrit : >> >>> It would be great to improve that, from our point of view, but I >>> wonder a bit: when is this a killer problem for everyone? >>> >>> Also, someone commen

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Mon, May 14, 2012 at 2:11 AM, Emile MOREL wrote: > Daniel Pittman a écrit : > >> It would be great to improve that, from our point of view, but I >> wonder a bit: when is this a killer problem for everyone? >> >> Also, someone commented about using marshall to speed this up - sadly, >> that isn

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Fri, 2012-05-11 at 20:16 -0400, Trevor Vaughan wrote: > For another point of data, I tried a handful of different ciphers > between the server and client with no noticeable difference in time. I think (but might be totally wrong) that since you're not seeing any real improvement with Nick's pat

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Jeff Weiss a écrit : First, Émile, your performance plots are awesome! Thank you so much for taking the time to put these together. They will help us tremendously as we investigate and kill off these issues. You're welcome, share my results with the community is quite natural ;) Second,

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Daniel Pittman a écrit : It would be great to improve that, from our point of view, but I wonder a bit: when is this a killer problem for everyone? Also, someone commented about using marshall to speed this up - sadly, that isn't an acceptable solution. We have had problems with Marshall data

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

For another point of data, I tried a handful of different ciphers between the server and client with no noticeable difference in time. Thanks, Trevor -- Trevor Vaughan Vice President, Onyx Point, Inc (410) 541-6699 tvaug...@onyxpoint.com -- This account not approved for unencrypted proprietary

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Friday, May 11, 2012 at 1:25 PM, Peter Meier wrote: > > On that note: After spending a few hours hunting yesterday, we > > identified the cause of a significant performance regression in > > compilation when using modules with metadata. I've attached a patch > > for anyone who would like to test

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

> On that note: After spending a few hours hunting yesterday, we > identified the cause of a significant performance regression in > compilation when using modules with metadata. I've attached a patch > for anyone who would like to test it out. We're seeing approximately > 5-10x faster compilation.

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Based on this change, caching catalog time appears to be about 1/10 of the total Catalog retrieval number fairly consistently. Thanks, Trevor On Wed, May 9, 2012 at 3:28 AM, Emile MOREL wrote: > Hi, > > Trevor Vaughan a écrit : > >> All, >> >> I've noticed an understandable correlation between

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Apologies for the double post but my Ruby version is 1.8.7. Thanks, Trevor On Thu, May 10, 2012 at 6:12 PM, Trevor Vaughan wrote: > Nick, > > I just tried this patch with 2.7.13 and I didn't notice a large > difference in timing overall but I also didn't notice any adverse > effects. > > And, y

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Nick, I just tried this patch with 2.7.13 and I didn't notice a large difference in timing overall but I also didn't notice any adverse effects. And, yes, I'm more than happy to try out new patches for optimization. Thanks, Trevor On Thu, May 10, 2012 at 2:44 PM, Nick Lewis wrote: > On Thursd

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

You guys are the best, quite the quick turn around! I'll be applying this to our largest puppetmaster tomorrow and running a series of tests so I can give you some feedback. On Thu, May 10, 2012 at 2:44 PM, Nick Lewis wrote: > On Thursday, May 10, 2012 at 10:30 AM, Ashley Penney wrote: > > On T

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Excellent work, Nick - thank you. -- Luke Kanies | +1-615-594-8199 | http://about.me/lak On May 10, 2012, at 11:44 AM, Nick Lewis wrote: On Thursday, May 10, 2012 at 10:30 AM, Ashley Penney wrote: On Thu, May 10, 2012 at 1:42 AM, Jeff Weiss wrote: Fourth, Peter, Émile, and Trevor (or anyone

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Nick Lewis wrote: > On Thursday, May 10, 2012 at 10:30 AM, Ashley Penney wrote: >> On Thu, May 10, 2012 at 1:42 AM, Jeff Weiss > > wrote: >>> >>> Fourth, Peter, Émile, and Trevor (or anyone else experiencing the >>> problem), would you be willing to be pre-release

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Thursday, May 10, 2012 at 10:30 AM, Ashley Penney wrote: > On Thu, May 10, 2012 at 1:42 AM, Jeff Weiss (mailto:jeff.we...@puppetlabs.com)> wrote: > > > > Fourth, Peter, Émile, and Trevor (or anyone else experiencing the problem), > > would you be willing to be pre-release testers of improvem

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Thu, May 10, 2012 at 1:42 AM, Jeff Weiss wrote: > > Fourth, Peter, Émile, and Trevor (or anyone else experiencing the > problem), would you be willing to be pre-release testers of improvements? > Our ops team is seeing the problem too, but that's only a single real-world > data point. We need

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Daniel Pittman a écrit : It would be great to improve that, from our point of view, but I wonder a bit: when is this a killer problem for everyone? We are using puppet only interactively, on command line. When writing your manifests, you can try to apply your freshly written module a coupl

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Fourth, Peter, Émile, and Trevor (or anyone else experiencing the problem), would you be willing to be pre-release testers of improvements? Our ops team is seeing the problem too, but that's only a single real-world data point. We need to make sure we don't self-optimize. We need your hel

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

But anyway I remember that caching the catalog (read serializing it as yaml to disk) has been very slow for years and also known. It usually makes the agent look like it's hanging. This especially happens on huge catalogs - I have catalogs with up to 10k resources. It would be great to improve t

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Wed, 2012-05-09 at 11:15 +0200, Peter Meier wrote: > But anyway I remember that caching the catalog (read serializing it as > yaml to disk) has been very slow for years and also known. It usually > makes the agent look like it's hanging. This especially happens on > huge catalogs - I have

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

First, Émile, your performance plots are awesome! Thank you so much for taking the time to put these together. They will help us tremendously as we investigate and kill off these issues. Second, we recognized the performance drop prior to this thread and Max and Carl had already been doing out

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Thu, May 3, 2012 at 2:25 PM, Trevor Vaughan wrote: > All, > > I've noticed an understandable correlation between catalog size and > config retrieval time. However, I'm a bit lost as to why it seems so > slow. > > For an example 1.5M catalog: > > 24 seconds compile time > Almost instantaneous tr

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On Wed, May 9, 2012 at 2:15 AM, Peter Meier wrote: >> Otherwise i work on Puppet optimizations and i have seen some strange >> things too. I work on big catalog (more than 1000 resources) and i want to >> reduce the application time (around 20 sec). >> I have seen two kind of things: >> - First,

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

It's interesting this problem has come up because all of a sudden I'm plagued by incredibly slow config retrieval: Changes: Total: 2 Events: Success: 2 Total: 2 Resources: Changed: 2 Out of sync: 2 Total: 390 Skipped: 6 Time:

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On 09.05.2012 11:15, Peter Meier wrote: But anyway I remember that caching the catalog (read serializing it as yaml to disk) has been very slow for years and also known. It usually makes the agent look like it's hanging. This especially happens on huge catalogs - I have catalogs with up to 10k re

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Otherwise i work on Puppet optimizations and i have seen some strange things too. I work on big catalog (more than 1000 resources) and i want to reduce the application time (around 20 sec). I have seen two kind of things: - First, recent Puppet versions are slowest (see PuppetVersions.png in

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Hi, Trevor Vaughan a écrit : All, I've noticed an understandable correlation between catalog size and config retrieval time. However, I'm a bit lost as to why it seems so slow. For an example 1.5M catalog: 24 seconds compile time Almost instantaneous transfer time (localhost) 40 seconds regis

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

Hi, Trevor Vaughan a écrit : All, I've noticed an understandable correlation between catalog size and config retrieval time. However, I'm a bit lost as to why it seems so slow. For an example 1.5M catalog: 24 seconds compile time Almost instantaneous transfer time (localhost) 40 seconds regis

Re: [Puppet-dev] Trying to isolate performance issues with config retrieval.

On May 3, 2012, at 1:25 PM, Trevor Vaughan wrote: > All, > > I've noticed an understandable correlation between catalog size and > config retrieval time. However, I'm a bit lost as to why it seems so > slow. > > For an example 1.5M catalog: > > 24 seconds compile time > Almost instantaneous tra

[Puppet-dev] Trying to isolate performance issues with config retrieval.

All, I've noticed an understandable correlation between catalog size and config retrieval time. However, I'm a bit lost as to why it seems so slow. For an example 1.5M catalog: 24 seconds compile time Almost instantaneous transfer time (localhost) 40 seconds registered by the puppet client as th