Thanks for you replies. I agree that it does not need to be something jbehave handles I was just wondering if it did.
It is a difficult case and it looks like we are going to get requirements in the future with the same criteria. Taking everything into account the test may end up taking a long time to run which wouldn't fit in with our CI and the idea of fast feedback. We are going looking at taking the statistic out of the story and seeing if we can use our CI to handle such tests that allow a certain failure percentage. There just does not seem to be a silver bullet with this type of requirement. Thank you for all your suggestions. I will pass them on to the developers. If we come to a nice solution for this case I'll follow up if you are interested. On Tue, Dec 18, 2012 at 11:53 AM, Alexander Lehmann <[email protected]>wrote: > I think it would be possible to write a step similar to a composite step > to run the necessary steps repeatedly and evaluate the result, however this > has the disadvantage that the steps are hidden in the implementation of the > step and not in the story file (and I think the steps are not reported > individually), though maybe given stories may be a possibility. > > When running individual steps with a statistical outcome, it would be > necessary to keep the state somewhere to be able to evaluate the > probability in the end, possibly in the page object, but this is difficult > to do if there are different tests to be accounted when multi-threading. > > One last thing I'd like to point out is that the evaluation of such a test > is not deterministic unless you have a mockup service that actually returns > a failure every n counts. When running the test 20 times, it would be valid > to have 0 or 2+ failures sometimes, where it would be misleading to have > the test fail. Maybe it would be feasible to run the 20 tests more than > once in that case and calculate the average, but this is still a bit flaky > to have a test failing when the expected value is not reached (either you > need an estimator function and a confidence interval or you can just report > the test result as warning with the calculated value) > > > > > > ------------------------------**------------------------------**--------- > To unsubscribe from this list, please visit: > > > http://xircles.codehaus.org/**manage_email<http://xircles.codehaus.org/manage_email> > > >
