On Tuesday, December 12, 2017 at 11:17:20 AM UTC-6, n.be...@gaijin.ru wrote: > > > We have a problem with very slow work puppetserver. > > > We have ~300 nodes, master runs on a server with 24 cores and 20 GB of > memory. > > Pappet agent log: > [...] > Time: > [...] > File: 4466.81 > Total: 4810.97 > Config retrieval: 75.19 > Service: 8.01 > Version: > Config: 1513029567 > Puppet: 4.4.1 > > Your log shows very long runtime (80+ minutes) for the Puppet *agent*. That's neither puppetserver nor the master.
The vast majority of the agent runtime is consumed in syncing File resources. That tells me that you are syncing an enormous volume of files, and possibly also a great number of them. You will find numerous previous discussions of such problems in this group; here are the usual recommendations: 1. Prefer to reserve File resources for smallish numbers of smallish files. Config files are the sweet spot for this resource type. 2. Prefer to package files and manage them via Package resources, as opposed to recursively syncing directories full of files, or otherwise syncing large numbers of related files. 3. Avoid syncing temporary files. If you use a File resource to manage a file, then it should be one that will remain on the system, because if you remove it (or modify it) then Puppet will just sync it again on the next run. This can mean leaving a file in place that otherwise you would remove. 4. If you must manage a large file via a File resource then consider specifying a different `checksum` attribute for it. The default is md5, but 'md5lite' will be faster and still give an ok test for modification. Or you can even go with 'mtime', which is very fast, but is susceptible to both false positives and false negatives. Your log also shows a longish runtime (75 seconds) for catalog retrieval. The `stat()` calls in your strace suggest that you're using the 'eyaml' Hiera back end, and this does add overhead. If you're storing all your Hiera data in the eyaml back end, then it may add a lot of overhead. I suggest using eyaml only for those data that actually need to be kept in confidence, and among them, only those for which the general access controls provided by the master's system are insufficient. Configure the standard back end too, and use it for the rest of your data. There may be other issues, too, such as the system load from all those File transfers, but I see no details that point me to specific server-side problems. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/4f0e88f7-2f2a-4c86-9a22-e1e072879c95%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
