On Friday, January 25, 2013 9:29:00 AM UTC-6, asq wrote: > > W dniu piątek, 25 stycznia 2013 16:07:43 UTC+1 użytkownik jcbollinger > napisał: >> >> >> To be clear, it looks like you are talking about counts of defined type * >> instances*, not counts of distinct defined types. Is that right? >> > > yes. it's test::rule() and test::rule_inline() in linked manifest. > > >> I am inclined to suspect that the problem is not defined types in >> general, but rather the specific defined types you are using. Indeed, your >> data seem to support that inasmuch as inlining definitions doesn't cut your >> compilation times very much. >> >> > we're talking about 5m33s and 4m13s difference in compile. this *is* a lot. >
The version using more defined-type instances has more total resources. The time per resource is very nearly the same. Your compile times certainly will scale with the total number of resources declared. Do not confuse defined types with macros. They are bona fide resource types, functioning like native types in most ways. > > >> I can think of several things might cause long compilation times: >> >> - evaluating many templates, especially complex ones >> - making many hiera lookups if your hierarchy definition uses >> interpolated non-global variables >> - repetition of anything requiring network access, such as syncing >> files with a version-control repository, accessing files on a remote file >> system, or performing name resolutions. >> - calling generate() a lot >> >> That's not an exhaustive list. >> > > please check define rule_inline() in linked pp file - it's nothing fancy. > Fanciness is not the question. Even very simple manifests can make declarations that are costly to compile. > > - 3 static files > - 1 erb (35 lines with 5 variables substitution and 1 if/else clause) > The erb evaluation is probably more costly than everything else about that particular File declaration, combined. Especially so in that it requires (in that particular case) up to four separate probes of the file system just to find the template file. > - 1 hiera call > I believe hiera caches data to the extent that it can do, but depending on how your hierarchy is defined and how your data are arranged, it could still be that each one of those hiera() calls involves filesystem probes and possibly reading and parsing YAML. That's not particularly cheap even if your data are simple. > - 1 exported resource > I must have overlooked that, but it's not especially cheap, either, inasmuch as it involves connecting to and querying a DB. It's practically expensive if the underlying DB is not local, as that then would involve network activity. > - 4 calls to custom type (firewall) > Which are probably cheap(ish), but that's not certain. > > no generates or other puppetmaster functions, no network stuff, no > dns-related stuff or anything i haven't already hashed-out to test if it > will affect compile times significantly. > > Anyway, I think filing a bug now would be premature. It may be that there >> is a bona fide performance problem, but if so then I don't think it has >> been pinpointed yet. >> >> > i filed a bug #18880 anyway, as i sense it might be some design error in > puppet - i can't really see why number of define() calls should relate to > compile times. as they are not "real" resources, there should be no reason > for them to be included in catalog, which also might relate to apply times. > > Definition instances certainly are real resources in most ways that matter. In any event, your compilation times even for the "inline" version of your manifests seem unreasonably high, which is why I suggested that you look at other possible performance drags. In addition to the manifest-specific issues I discussed above, you should also look at how your master is provisioned and loaded. Puppet is a bit of a memory hog, so if the available physical RAM is low then Puppet may perform a lot of expensive swaps between physical and virtual RAM. If the master is sharing hardware with other services (even other masters in, for example, a passenger setup), then contention for the disk subsystem could slow everything, especially if you have SATA (or worse) disks instead of SCSI. There are many other such possibilities. On the other hand, I don't mean to suggest that declaration of defined-type instances is free or cheap. As I said, defined types are not macros, but rather bona fide resource types. Declaring a defined-type instance involves more work than simply making all the declarations that appear in the definition body. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
