On Tuesday, October 18, 2016 at 9:49:23 AM UTC-5, jcbollinger wrote: > > > > On Monday, October 17, 2016 at 3:15:51 PM UTC-5, re-g...@wiu.edu wrote: >> >> >> >> On Friday, October 14, 2016 at 5:08:31 PM UTC-5, re-g...@wiu.edu wrote: >>> >>> >>> >>> On Thursday, October 13, 2016 at 9:15:58 AM UTC-5, Rob Nelson wrote: >>>> >>>> I've worked with saz in the past and believe he would be very receptive >>>> to PRs that address this issue, as well, if RG or anyone else wanted to >>>> contribute them. That would be one of the better solutions to the problem. >>>> >>>> Rob Nelson >>>> rnel...@gmail.com >>>> >>>> >>>> I think I've given you a pretty good handle on the nature of the >>>>> problem, and I'm inclined to think that a solution that addresses the >>>>> problem at its root will take care of the whole suite of issues. >>>>> >>>> >>> Lots and lots of thanks to jcbollinger. >>> Here is what I have done as a result of this thread: >>> >>> Submitted an issue with saz-rsyslog: >>> Duplicate Declaration Class[Rsyslog] with The Foreman as ENC during >>> catalog compilation #237 >>> https://github.com/saz/puppet-rsyslog/issues/237 >>> >>> Posted a follow-up question to a related thread on The Foreman mail list >>> to determine if The Foreman also may be exhibiting a bug, or if the issue >>> may be my configuration: >>> Duplicate declaration error. >>> https://groups.google.com/d/topic/foreman-users/k0JgABtszdU/discussion >>> (my post is waiting for approval as of 22:00 GMT 10/14/2016) >>> >>> My current thought at this time is two points: >>> 1. saz-rsyslog is written in such a way that declaring a subclass before >>> the parent class causes duplicate declaration errors - but I am told the >>> module inheritance should be addressed, and would resolve this issue. >>> 2. The Foreman, under undetermined circumstances (maybe my >>> misconfiguration), will randomly declare parent classes and subclass out of >>> order - but I am told this is not an issue with puppet modules that use >>> inheritance correctly - thus there might be an underlying bug with The >>> Foreman because this error will not appear under modules' expected >>> inheritance. >>> >>> >> This is what was posted on The Foreman mail list group: >> https://groups.google.com/d/topic/foreman-users/k0JgABtszdU/discussion >> >>> This is an issue with the format of the ENC YAML used between Foreman >>> and Puppet, and is best fixed in the module. >>> The list of classes is given as a hash/dictionary and so has no >>> particular order defined - it's down to the Puppet master/server to >>> iterate over it, and it probably does so in no particular order. It >>> isn't under Foreman's control. >> >> >> I now believe this is truly the root of my problem. >> >> The evidence is looking at the ENC output and the Puppet-generated >> Catalog file on each of the master puppet servers. >> As you can see below, the puppet-generated catalog on the >> secondary-master puppet server has put rsyslog::client before rsyslog - >> this causes the failure. >> However, The Foreman ENC output is the same on both primary and secondary >> master puppet servers. >> >> > > You have taken a step backward there. I remind you that duplicate > declaration errors occur *during catalog building*, and that catalog > building fails if one occurs. It is therefore impossible to compare a > catalog from the failure case to a catalog from the success case, because > no catalog is ever created in the failure case. > > Then I was mistaken in thinking that the node file (/var/lib/puppet/yaml/node/server1.mydomain.example.com.yaml) was the data (catalog pre-data) written to a file before the catalog compilation procedure. I assumed this because both the foreman and node files are created new every time server1 performs a puppet agent update, even when the result is a 400 error for the catalog compilation. If I am mistaken in this, then I need someone to tell me how to get the data of the catalog that is failing to be compiled, so that I can compare it with other sources.
Here is the procedure I have been assuming (I am not an expert on puppet internals): 1. (server1)puppet-agent -> puppet-master 2. puppet-master -> (exec)puppet[external_nodes] server1.mydomain.example.com -> foreman ENC 3. foreman ENC -> /var/lib/puppet/yaml/foreman/server1.mydomain.example.com.yaml -> puppet-master 4. puppet-master -> /var/lib/puppet/yaml/node/server1.mydomain.example.com.yaml -> puppet-master(compile-catalog) 5. puppet-master(compile-catalog) -> Error -> 400 Server Error -> (server1)puppet-agent I was assuming the node file is the data of the catalog to be compiled. I had noticed it is updated in every puppet agent run, even when the run is in error. And it was the differing piece between primary and secondary puppet masters. Please correct me. Please let me know how to get the data of the catalog the puppet master is failing to compile for the puppet agent. > It is useful to confirm that Foreman's ENC produces consistent output -- > if indeed it does. I'm not surprised that you obtained identical ENC > output from two different Foreman servers, but it's unclear to me whether > you have compared the failure-case ENC output with the success-case ENC > output. I am inclined to guess that these do differ, because clearly > *something* changes. > > The /var/lib/puppet/yaml/foreman/server1.mydomain.example.com.yaml file is the same on both The (exec)puppet[external_nodes] server1.mydomain.example.com (/etc/puppet/node.rb server1.mydomain.example.com) output is the same on both - I assume this is the only source and final data from The Foreman ENC. > Unless .... > > Do check the the 'ordering' parameter in the [master] section of your > master's puppet configuration. If it is specified, and the specified value > is anything other than 'manifest', then it is possible that setting its > value to 'manifest' (the default) will help. I emphasize, however, that > this is a workaround for sloppy manifest code, not a bona fide solution. > > No, this parameter is not set on either the primary or secondary puppet masters. The only differing parameters are 'ca' and 'certname'. Between a this secondary-master and another secondary-master (not having issues), only the `certname` parameter is different. > You've made a lot of hay about saz-rsyslog being approved / known-good / > high-quality, with no particular justification. Certainly the module being > available from the Forge should not be interpreted to support such a > conclusion. To be clear: having looked at the code of the module in > question, I think it is generally of good quality. As I have already said, > however, and as Dominic Cleal told you over on the Foreman group, the > fundamental problem here is in the manifests -- *the module's* > manifests. The module is designed and its manifests are written in a > manner that is especially conducive to the kind of problem you've > encountered. > > The saz-rsyslog module is officially "approved", because it is labeled as "approved" on the forge site. I used that term as an identifying piece of information. I myself do not give it that label. If you search for saz-rsyslog on forge, you will see it as the first result with a 45 deg banner in the top right that reads "APPROVED". I do not remember mentioning it is known-good or high-quality. I apologize if I gave that perception. However, reading this source: https://forge.puppet.com/approved, the "APPROVED" label from Puppet indicates the module does "Adhere to Puppet's standards for style and design". I do remember saying it was either widely used or popular. But I said this only as another "labeling" or identifying factor, as the same search results entry say it has been downloaded 36,000+ times. Version 4.0.3 - Oct 4, 2016 - 36,632 downloads Again, I apologize for unintended misconceptions I have given. > > >> Is this correct, that the yaml file in /var/lib/puppet/yaml/node/ is >> exclusively generated by Puppet based on the ENC yaml? >> If this is correct, then this is the root of the problem. >> > > > I'm not sure what significance you're attributing to those particular > files, but it's probably the wrong significance. Yes, to the best of my > knowledge and understanding, Puppet generates those files based on the > node's certificate and facts, and the associated ENC output (if any). But > no, there is no reason to think that the root of your problem lies here. > > As I said earlier, I now assume I misunderstood this as being the file holding the data to be compiled as the catalog. I apologize for the misunderstanding, but that I intended it as the "root" of the catalog-compilation "problem". You are right that it is not the root of the entire underlying problem. Again, I need help to find the catalog data trying to be compiled, but which fails, if this /.../node/.. file is not that data. However, the failure point is the catalog compilation. Nothing fails leading up to this, so it has been very hard to track down the causes leading up to this revealed error. You are right, the solvable problem is fixing the inheritance issue of the puppet module. > I have already explained the root of the problem several times, but I'll > try just once more: the saz-rsyslog module is designed and implemented in a > way that does not support declaring class ::rsyslog via a resource-like > declaration (whether Puppet DSL's version of that or its ENC analog) while > also declaring any of several other public classes from the same module. > If there is any flaw to be attributed to Puppet here, it is that catalog > building sometimes works for you anyway, not that it sometimes fails in the > face this kind of misuse. > > Yes, I agreed with that. I apologize for not confirming this conclusion. Instead I responded by pointing to the bug I filed with saz-rsyslog. And in that saz-rsyslog bug report I provided your analysis of the issues, and pointed back at this thread. Thank you for that. That was what I needed to be able to file a bug. > > >> >> The next question is, why does Puppet assemble a catalog with classes out >> of order, when the ENC output has them in the correct order? >> > > > The order in which classes are listed in the ENC output is not > significant, nor indeed is the order of entries in any YAML hash logically > significant. That's basically the answer. > > Depending on several underlying factors, it might be that under some > circumstances you do reliably see classes specified by the ENC being > evaluated in an order consistent with the lexical order of the ENC output. > As I remarked in my previous message, the version of Ruby on which your > master is running is one of my first guesses as to a source differences in > that respect. > > I checked the versions of ruby installed. They are both running: Package: ruby193-1.1-9.sc1.el6.x86_64 Version: ruby 1.9.3p484 (2013-11-22) [x86_64-linux] To top it off, we are mirroring the SCLo repositories with Pulp, and so both puppet master servers are pulling packages from the same static repository. > > >> I have seen two answers to this question of classes getting out of order >> in the puppet catalog: >> > > > Classes are not getting out of order in the puppet catalog, nor indeed are > the files that you earlier asked about (serialized) Puppet catalogs. > > As I eluded to earlier, before I was assuming the node file (/var/lib/puppet/yaml/node/server1.mydomain.example.com.yaml) was the the data that was to be compiled as the catalog. I need help to know how to get the data that is compiled for the catalog. If I can compare this data on both servers, perhaps that is a lead. I am grasping at straws. > > >> 1. one class 'cA' depends on 'cB', so puppet puts 'cB' first in the >> catalog. - but this is not the reason in my case. >> 2. "... If you do not see the error on every run then it is modulated by >> something that varies between runs. That could be almost anything: >> manifests, data, results of function calls, node facts, or ENC output. ..." >> - which should be unlikely since it works on one secondary master and not >> the next secondary master, both configured exactly the same. >> >> > You're now diverting to a different topic: why the observed behavior > changes *over time*, as opposed to why it differs between masters. I > have no reason to doubt your assertion that the manifests involved are the > same everywhere. The data I refer to would be Puppet external data, which > it seems you are not using, so that is trivially the same from run to run. > But that still leaves many things that could vary over time; the list you > quoted covers other likely candidates, but it was never intended to be > exhaustive. If I had to guess, though, I'd place my bet on the ENC output > varying. Do note that it doesn't have to be the part of the ENC output > that designates classes rsyslog and rsyslog::client that changes; a > variety of other changes might result in an inversion of the order in which > those two are evaluated, too. > > Yes, I was considering this before. However, in my previous post, and as mentioned earlier in this reply, I had come to assume the foreman file (/var/lib/puppet/yaml/foreman/server1.mydomain.example.com.yaml) was the ENC output that Puppet used in the catalog. And these files are the same on both the primary and secondary master servers. And as mentioned earlier in this reply, I also checked the output of `(exec)puppet[external_nodes] server1.mydomain.example.com` (/etc/puppet/node.rb server1.mydomain.example.com) and they also match between primary and secondary puppet master servers. And that output put in a file matches exactly the content of the foreman file and the file size (/var/lib/puppet/yaml/foreman/server1.mydomain.example.com.yaml). > > >> Is there a configuration issue causing this issue? Where would I begin >> looking for something that may be causing this puppet-catalog problem? >> > > > There is an unsupported-use-of-module issue ultimately causing all your > problems, as I have been saying since the beginning. It would be best to > solve *that* problem, either by choosing a different module or by > restricting yourself to supported uses of the module you have chosen (e.g. > by using automated data binding to assign values to class parameters). > > "restricting yourself to supported uses of the module" So using The Foreman as ENC has not been an official tested use of the module and thus is not supported? This is true for all modules on forge I have reviewed, except for the ones published by The Foreman group. To be supported, should I instead restrict my use of either declaring my classes in a site.pp or use hiera? "using automated data binding to assign values to class parameters" What is automated data binding? Is that what I get when using hiera? Would that mean I would forgo the use of The Forman to instead use hiera? I have been assuming hiera was relatively the same as using The Foreman as ENC? Reference: https://ask.puppet.com/question/527/are-hiera-and-foreman-parameters-mutually-exclusive/ > If you insist on continuing in your misuse and trying to work around the > problems that follow from that, then it seems reasonable to focus on > differences between the secondary master that fails and those that always > work in this regard, if in fact they really do. Compare software versions > (especially of Puppet and of the Ruby on which Puppet is running (and watch > out for multiple Ruby versions being co-installed). Compare (puppet) > config files. Heck, replace the troublesome secondary master with a clone > of one of the others if you can't figure out what differs. > > > >> Running `puppet agent -t` on the client against the primary master puppet >> server is successful >> Running `puppet agent -t` on the client against the secondary master >> puppet server fails with duplicate declaration >> >> > > I note in passing that you've now shifted back to the topic of behavior > differing between different masters. > > I am grasping at straws. I have been assuming what I can find different between the servers would reveal the issue. I do not deny that the saz-rsyslog puppet module issues should be addressed, and would resolve the issues at hand. But it seems that if I have the problem on one server, but not another server, there is something more that is wrong that goes beyond only the module's issues. Thus I point at two possible issues. One with the module. Another with either The Foreman or Puppet. And on top of that, the secondary-master puppet server is not consistent in always throwing the duplicate declaration error. I guess I need someone to tell me that if a puppet module is having the issue we describe in this thread, that due to the intended and expected nature of some function/process "X" in The Foreman or Puppet, I can expect to see the module's inheritance issue cause catalog-compilation errors at random times. Example: it was suggested that the hash keys are not always ordered the same. So due to python sometimes ordering the hash keys differently, a module with inheritance issues like saz-rsyslog will sometimes cause a catalog compilation error, and sometimes not be an issue. This means Puppet expects modules to not have an inheritance issue like we are describing in order to guarantee there will not be issues during deployment, like in my case. > > >> On the primary master puppet server >> >> --The Foreman ENC output-- >> >> primary-puppet:/var/lib/puppet/yaml/foreman/server1.mydomain.example.com.yaml >> ...snip... >> rsyslog: >> gnutls_package_name: false >> preserve_fqdn: true >> relp_package_name: false >> rsyslog_package_name: false >> "rsyslog::client": >> log_local: true >> remote_type: udp >> server: rsyslog.mydomain.example.com >> ...snip... >> --end-- >> >> --Puppet generated catalog-- >> primary-puppet:/var/lib/puppet/yaml/node/server1.mydomain.example.com.yaml >> rsyslog: >> gnutls_package_name: false >> preserve_fqdn: true >> relp_package_name: false >> rsyslog_package_name: false >> "rsyslog::client": >> log_local: true >> remote_type: udp >> server: rsyslog.mydomain.example.com >> --end-- >> >> >> On the secondary master puppet server >> >> --The Foreman ENC output-- >> >> secondary-puppet:/var/lib/puppet/yaml/foreman/server1.mydomain.example.com.yaml >> ...snip... >> rsyslog: >> gnutls_package_name: false >> preserve_fqdn: true >> relp_package_name: false >> rsyslog_package_name: false >> rsyslog::client: >> log_local: true >> remote_type: udp >> server: rsyslog.mydomain.example.com >> ...snip... >> --end-- >> >> --Puppet generated catalog-- >> >> secondary-puppet:/var/lib/puppet/yaml/node/server1.mydomain.example.com.yaml >> ...snip... >> "rsyslog::client": >> remote_type: udp >> log_local: true >> server: rsyslog.mydomain.example.com >> ntp: >> server_list: >> - "ntpserver.mydomain.example.com" >> rsyslog: >> gnutls_package_name: false >> rsyslog_package_name: false >> preserve_fqdn: true >> relp_package_name: false >> ...snip... >> --end-- >> > > > 1. Those are not catalogs you are looking at. > 2. You've presented excerpts of the ENC output and node dumps as if > those are the only parts that matter. The totality of the ENC output > matters (but I don't think the node dumps actually matter at all). > > > John > > So you are saying the node dump is not the catalog. Please tell me how to obtain the catalog data that it is failing to compile due to duplicate declaration. You have given me a lot of information. I greatly appreciate your help. -RG -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/e499facc-457d-47ee-95a5-99b654eea4de%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
