On 10 February 2011 12:19, Mirsad Vojnikovic <mirsad.vojniko...@linaro.org> wrote: <snip > That I wrote:
>> I'd like to add as user stories: >> Dave wants to rerun a test on a particular machine to see if a >> failure is machine specific. > > An initial idea we had was to run jobs based on machine type, i.e. > BeagleBoard, not on a particular machine, i.e. BeagleBoard_ID001. The > dispatcher would choose on which particular machine to run, depending on > availability. I understand your point when running on a particular machine > is desirable, but maybe this feature should be enabled for admins trying to > track a deviating hardware? Or maybe this is a user story for dashboard, to > have a feature comparing and presenting results from all machines of the > same type, or even in broader aspect for chosen/all machine types we > support? I'm talking here of the case where the user has run a set of tests and one is showing up as bad and they are trying to work out why; lets say they run the test again and it works on a different machine; they might reasonably want to see if the original machine fails. Then the second subcase is that we've identified that a particular machine always fails a particular test but no one can explain why; you've been given the job of debugging the test and figuring out why it always fails on that machine. This might not be a hardware/admin issue - it might be something really subtle. >> Dave wants to run the same test on a set of machines to compare the >> results. > > This is almost same as first. Maybe the better solution, as I wrote above, > is to go to dashboard and compare all the existing results there instead? > This assumes of course that there are results already reported for wanted > hardware, which I think would be a case if looking at weekly execution > intervals, but probably not daily. What do you think, is this reasonable > enough or am I missing something important? OK, there were a few cases I was thinking here: 1) A batch of new machines arrives in the data centre; they are apparently identical - you want to run a benchmark on them all and make sure the variance between them is within the expected range. 2) Some upgrade has happened to a set of machines (e.g. new kernel/new linaro release) rolled out to them all - do they still all behave as expected? 3) You've got a test, it's results seem to vary wildly from run to run - is it consistent across machines in the farm? Note these set of requirements come from using a similar testing farm. Dave _______________________________________________ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
