Hi John, thank you for the explanation, these are very useful insights! Ok, let me follow this through and see where i get it and update you all here.
On Tuesday, March 8, 2016 at 11:51:31 PM UTC+9, jcbollinger wrote: > > > > On Monday, March 7, 2016 at 7:57:33 PM UTC-6, SG Madurai wrote: >> >> Hi John, Thank you for the update. >> >> Pardon me if i am asking about things that have been clarified/ settled >> already. >> >> From what i understand, agent run times are primarily determined by >> - catalog compilation time at master >> - the time for agent to apply catalog on its node >> >> > > Both of those are contributors. The former is rarely a major one. There > is also time spent by the agent computing facts, which is usually even > less, but can be costly if costly custom facts are installed. > > Also, catalog application often is not an agent-only activity, as it > commonly involves the agent obtaining files from the master's file server. > This can be very expensive for both the agent and the master. > > > >> So was basically wondering if there is an option to separate these 2 >> functions and manage these 2 independent of each other (at times convenient >> for each of these activities) >> >> > > Nodes have as much control as they want to exercise of when and how often > they perform catalog runs. If they run the agent in daemon mode then they > can configure the run interval, but they also have the option of running it > at the times they choose via a scheduler, such as cron, or on-demand > either manually or via a remote-control system such as MCollective. > > The master does perform some caching to speed catalog building, but as I > already said, it is impractical for it to cache whole catalogs for direct > service to clients. The problem here lies in determining accurately and > efficiently when cached catalogs are stale. > > > >> If these concerns shouldn't arise with running multiple puppet masters w/ >> puppet db (or by imply upgrading...we are on v3.8 btw), then will explore >> that option first. >> > > > If your master(s) do not adequately serve the catalog request load, then > the quickest solution is often to empower them by running more puppetmatser > threads, adding CPU, adding RAM, increasing network bandwidth, and/or > shutting down other services. "Shutting down other services" might include > moving PuppetDB to a separate machine. Do also attend to the possibility > of uneven load: some kinds of site configurations lend themselves to highly > uneven load on the master, such that it sometimes gets transiently > overloaded even though it has sufficient capacity for its average load. > > If individual catalog compilations are taking a long time, then it is > probably worthwhile investigating why that is. It may well be the case > that you can realize substantial improvements by modifying your manifest > set. If the master is bogged down at the file server then you are probably > managing either large numbers of files or very large files, or both, in an > inefficient way; this is an area where it is relatively easy to shoot > yourself in the foot. > > If none of those alternatives yield the catalog service bandwidth you > need, then the next logical step is multiple masters. > > >> >> I couldn't be sure if these configuration options (multiple puppet >> masters w/ puppet db) by itself can take care of the issues we are facing >> with agent runs in our environment >> (timeouts, slowness..) >> >> We have one puppet master (v3.8) managing 150-200 nodes in an environment. >> > > > That's a fairly substantial load for a single master, but whether it's at > or beyond the capacity you should expect depends greatly on your manifests, > data, and nodes. > > In any event, you started off in the wrong direction by asking about agent > run times. If agents' catalog requests are being serviced slowly, and > especially if they sometimes time out, then your problem is likely to be an > overtaxed master. Long catalog-building times can contribute to that, but > so can the overall number of requests, uneven load, competing jobs, and > other factors. > > > >> >> BEFORE actually implementing this setup (multiple puppetmasters w/ puppet >> db) in our environment, i wanted to see if this is all there is to do - to >> fix these timeouts/ delays we see in our agent runs. >> >> > > As with any optimization problem, you are best off proceeding in a manner > that is informed by *data* about what parts of the process are slowest. > To that end, you could consider enabling the built-in profiler on the > master <https://puppetlabs.com/blog/tune-puppet-performance-profiler>. > You should also look at the overall load on the machine -- are you maxing > out your available CPU? your network bandwidth? your physical RAM? > > Also, multiple masters and PuppetDB are separate considerations. If you > are not using PuppetDB then you probably should be using it, especially if > you rely on external resources, regardless of whether you have multiple > masters. Also, if you are still running the m,aster as a Rack application > -- for example, under Apache / Passenger -- then you should consider > running it under Puppet Server > <http://docs.puppetlabs.com/puppetserver/1.1/> instead. > > > John > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/3655d5ef-6397-4e93-9639-ee6e24d4dd1b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
