On Thu, 2010-03-18 at 11:11 +0100, Peter Meier wrote: > >> OK, I identified the issue (patch coming soon): > >> > >> Puppet::SimpleGraph#matching_edge is simply completely inefficient in > >> front of lots of events for a given resource (which is our case). > >> The problem is that matching_edge returns the list of edges for which we > >> should trigger an event. In the aforementioned manifest, since there is > >> no subscribe/notify, the list is empty. > >> > >> Unfortunately with the current code, it takes about 22s to find this > >> result for about 1100 events. With the (soon to be published patch), I > >> was able to reduce this time to 4s. > > > > Note: 22s doesn't seem that many, but I also tested with about 5000 > > events and got about 2min (new code), and stopped it after more than > > 15min (old code). > > > > It seems the previous code was at least quadratic (damn ruby allocator, > > not the code itself). > > > > Peter, if you have some spare time, I suggest you try this new patch on > > top of the previous one (although they're not related). > > Brice, thanks for that follow up. I tested your patch on a directory > containing ~8k of files or directories. Puppet finished chmod and > chown the files as before in about 1-2 minutes. Still it hangs > afterwards burning cpu. However it does not anymore grow an enormous > amount of memory! The amount of memory remains stable now, which is > great. > However after those 1-2 minutes it hangs for about 60 minutes burning > cpu as before and finishes after 62 minutes. However as discussed in > IRC this is imho more a matter of traveling the graph.
60 minutes is still too high for managing 8k change events that won't ever match (or be used in the end) :-( It's certainly better (in time) to what it would be without the patch (ie do you have this number, assuming the server is not swapping)? I think we need a smarter way of matching those events, something involving pruning the graph of some sort. I'll try to think about this. > To be clear: this is really only a problem within the first run when > puppet touches _every_ file. If it has to touch only a few files it > finishes much faster. For example in about 7 minutes to fix 1k of > files. Which means imho that the main problem remains in passing the > events up. > > Still regarding the features and advantages you get by chown/chmoding > such a structure with puppet resources, it would be great if that > could be passed in a reasonable amount of time. The problem is not only for chown/chmoding. It just means we are inefficient at managing many events (granted there might not be too many real situation generating that many events). -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
