On Tue, Apr 15, 2014 at 06:46:21AM -0700, Alexander Fortin wrote: > On Tuesday, April 15, 2014 1:10:05 PM UTC+2, Felix.Frank wrote: > > > > On 04/15/2014 09:25 AM, Nikola Petrov wrote: > > > I haven't used catalog-diff but I > > > am suspecting that it might be a substitute for the rspec tests. Tell me > > > > No. rspec allows you to implement unit tests to catch certain regression > > bugs. catalog-diff is rather concerned with integration testing > > (although you can use rspec for that as well). > > > > I think with catalog-diff we're catching regression bugs too, meaning that > if the module interface is broken in the proposed commit, it will show up > as an error (unless no node is actually including the refactored module).
Hmm I am suspecting that sadly we are talking about sort of different things here. In my company we have reusable puppet modules between different teams - those are build by experts and require major testing because we don't want to break things in incremental versions. The thing is that those modules are not applied as is but are mostly reused in different project specific modules(i.e. site modules) I can see catalog-diff valuable for our site modules because it is actually testing the end result - what will be applied to a given host. In most cases the reusable modules are used in different ways with different parameters - let's say a tomcat module that is responsible for deploying war files and what not but cannot be used standalone(you have to wrap it in some way) > > > > > > does it provide the following: > > > > > > * Automatic runs through guard/rake/whatever on every commit to verify > > > that I am not breaking things like permissions on files > > > > You could do that, but compiling each of your nodes' catalog on every > > commit would be excessive. > > I don't see it as such a big problem. You can always put more resources on your CI server to make that possible which leads me to my next question - have you guys automated this in some way. I would be glad to catch regressions from catalog-diff on all my hosts but automatically - let's say through rspec or some other srcipt. Shameless plug: I have been doing a more mature approval port for java lately https://github.com/nikolavp/approval and although it is not ready, there is a good looking thing for ruby here https://github.com/kytrinyx/approvals. The idea is that you approve the output of master compile the first time the test is run and then(on further runs) if there are differences(detected through catalog-diff) you are prompted(let's say in your editor). This allows you to "approve" the differences if you know that they won't do any harm but in a more automatic way. > > > I agree it can be overkill for many environments, I guess depends on what > you actually want to test, or better, what kind of confidence you want to > have that your committed code is not actually breaking anything. > > To me seems that the catalog-diff approach is catching a broad variety of > problems, also some that (I think) can't be easily catch by expressions in > rspec. > > For example, somewhere in a random manifest I'm just changing declaration > style for a class, from 'include myclass' to "class {'myclass': }", leading > to possible declaration dependency order / duplicated declarations > problems: Node A (which I am currently working on) is including only one > declaration, but node B was evaluating that class declaration two times, > hence the duplicate declaration only on node B. > > Can rspec-puppet catch also this kind of problems? More in general, I'd > like to know if rspec-puppet can give me the confidence that, given a node > resource, the new version of the catalog will compile for that node. > Actually no - that's what we use staging for. You push your modules up there and see if they run properly on real machines. We also use the modules through vagrant on every developer machines - so they are constantly tested. Of course we want to start using beaker for those kind of tests so we don't push crap on the bear metal(although it is not metal most of the time) > -- > Alex > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/826f278e-1b69-4721-9e35-7028627c44fa%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20140417070617.GD24790%40nikolavp-desktop. For more options, visit https://groups.google.com/d/optout.
