I use Hudson. I've used Test::Unit to wrap validations of generated config files in the past. I want to know as early as possible if something isn't going to work.
Functionally testing the config from the outside in might be easy but slow. You can deploy to a VM and actually test it using the goodness of cucumber. I'm still looking for the equivalent of unit tests: how do you quickly test function from the inside to give enough confidence and a fast feedback loop? Maybe we need to build out Patrick's approach with a library of command-line tools to verify that everything works. J. 2009/11/7 Isaac Christoffersen <isaac.christoffer...@gmail.com>: > > I'd be interested in hearing more about this. I've been working on a > project in which we do a lot of automated server builds and > configuration using various OSS tools, including puppet. However, one > thing that I've been looking for is a way to write a set of test suites > to validate the server configuration. The combination of TDD and BDD > with server rollouts is very attractive. > > I'd also be curious to know what CI server people are using to help > coordinate these intermediate builds and tests. > > > > On 11/7/09 11:00 AM, Patrick Debois wrote: >> for what it's worth: >> >> for a while I was experimenting with cucumber to test scripts running >> on linux machines. And yes , IMHO it is the way to go . >> >> It worked well with the behavior testing just a you guys described. >> still just as in regular development, bdd can be complemented with >> tdd. >> The way i did the tdd, was by using the exit codes of the shell >> commands. I created an abstraction in ruby for execute (local, remote) >> and upload, download of files. >> >> If the command did not have exit 0 (the default) then i imagined the >> test being failed. >> Using snapshots of the virtual machines, I ran the scripts against a >> virtual machine (used virtualbox), with the ability to do a rollback. >> I know this is not the way puppet normally operates, but I used the >> (if i recall correctly) the puppetrun command to execute it and see >> what happened >> If all of these exit codes where ok (similar to unit tests), i ran the >> bdd tests to see if everything was correct. >> >> By integrating the tdd, and the bdd within a CI-system, I was able to >> have it run continuously and rebuild the system over and over again. >> >> Another thing I want to mention is that you can easily abstract >> commands in a custom dsl, but I really like to see what actual >> commands got executed. >> This is slightly off topic maybe, but by logging/running the actual >> commands and have the whole environment build like that (including >> making a floppy, booting the vms) >> it allowed me to create a kind of install document with the actual >> commands instead of pointing to a newly invented DSL. >> So I say yes to an abstraction layer, but no to hiding the actual >> commands. >> >> just my 2 c >> >> On Nov 6, 3:56 pm, Lindsay Holmwood<lind...@holmwood.id.au> wrote: >> >>> 2009/11/6 Martin Englund<martin.engl...@sun.com>: >>> >>> >>> >>> >>> >>> >>>> Folks, >>>> >>> >>>> I've been struggling a bit with how we're using puppet (at my job): >>>> how do you validate that puppet has done what it is supposed to, and >>>> even troublesome, how you validate that it has done what you intended >>>> it to do? >>>> >>> >>>> Since I'm the only one who is writing the puppet profiles and working >>>> with it on a daily basis, I'm the only one who can decipher the puppet >>>> logs. I often get the question: how do we know when the system is >>>> ready for production? >>>> >>> >>>> I've been playing around[1] with cucumber& webrat, and have pieced >>>> together a way to do behavior driven infrastructure testing. Puppet >>>> takes care of getting the system configured correctly, but there are >>>> often other pieces involved, like opening firewall ports, adding DNS >>>> entries, sendmail routing, etc. Which must be done outside of puppet >>>> to get the system ready for release. >>>> >>> >>>> When you write code, you always use unit testing& integration testing >>>> to verify that the application is working as expected, but why don't >>>> we use that when we install a system? >>>> >>> >>>> What are you using to verify that your system is correctly configured >>>> and behaves the way you want? >>>> >>> >>>> [1]<http://blogs.sun.com/martin/entry/behavior_driven_infrastructure> >>>> >>> [although i've commented on the excellent blog entry, i'm posting here] >>> >>> Hi Martin, >>> It looks like there's a bit of crossover here with a project i've been >>> working on the last few months called cucumber-nagios[0]. It takes the >>> result of a Cucumber run and outputs it in the Nagios plugin format. >>> Essentially you use it to express your intentions in plain language, >>> and verify your intentions periodically through your monitoring >>> system. Just like what you've posted about. :-) >>> >>> Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in >>> Belgium last weekend, and I got talking with people about expanding >>> the library of steps to cover things like logins over SSH, file >>> manipulation, and mail delivery. It would be cool if we could >>> centralise our efforts and focus on building an awesome library of >>> reusable steps to test our infrastructure. >>> >>> Your point about doing behaviour driven development when writing >>> software is right on the mark. From an infrastructure perspective, I >>> like to think of Cucumber as the testing tool, and Puppet as the >>> programming language. >>> >>> Anyhow, i'd be interested to hear what other people think about this idea! >>> >>> Cheers, >>> Lindsay >>> >>> [0]http://auxesis.github.com/cucumber-nagios >>> >>> --http://holmwood.id.au/~lindsay/(me) >>> >> > >> > > > > -- Julian Simpson Software Build and Deployment http://www.build-doctor.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
