Nick Lewis has a script that you can grab from here: https://gist.github.com/1217637
If you know the order that your tests ran it, this will help you narrow it down to the particular two that are interfering with one another (accomplished via a sort of binary search, running half of the number of tests from the last run over and over again until it has identified the guilty parties). In my experience it's usually a matter of stubs/expectations stomping on one another, or global/singleton state variables not being reset properly. I have a pull request here: https://github.com/puppetlabs/puppet/pull/676 That shows how you can potentially log the order that the test files were run in, which is a prereq of using Nick's script to debug. I need to test it a bit further but what I'm hoping is that after I have confidence in it we can use it on the Jenkins jobs so that we are always capturing their order. Then, we can enable rspec's random order feature so that all of our jenkins runs will be randomized, and if we ever trigger an order-dependent failure we'll know the order so that we can use Nick's script. Once I've narrowed it down to two specific test files, if nothing obvious jumps out at me I'll just comment out all of the code in the first test and selectively uncomment it until I determine which lines are causing the problems. hth chris On Tue, Apr 17, 2012 at 5:27 PM, Jeff McCune <j...@puppetlabs.com> wrote: > Hey everyone, > > I'm working on fixing up a module and I've written a test that triggers > the bug against Puppet 2.7.x. I'm running into a problem where I get 0 > failures when I run `rake test` or `rspec --format d spec/` but I get the > failure I'm expecting if I run the spec file directly with > rspec spec/unit/application/describe_spec.rb > > Here's the gist of what's going on: > > https://gist.github.com/2410088 > > The code in question is: > > [1] > https://github.com/jeffmccune/puppetlabs-mount-providers/tree/ticket/master/13070_fix_mounttab > > And Puppet 2.7.x > > What is a common case for this type of order dependent false positive, and > what are some strategies I can employ to be a bit more defensive about the > context my test is running in? > > Thanks, > -Jeff > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To post to this group, send email to puppet-dev@googlegroups.com. > To unsubscribe from this group, send email to > puppet-dev+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-dev?hl=en. > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to puppet-dev@googlegroups.com. To unsubscribe from this group, send email to puppet-dev+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
