On 03/09/2016 04:11 PM, floccul...@gmx.co.uk wrote:
On 09/03/16 20:59, Nicholas Skaggs wrote:
On 03/09/2016 03:51 PM, floccul...@gmx.co.uk wrote:
On 09/03/16 20:43, Nicholas Skaggs wrote:
Sure. Each of the test runs are a 1 to 1 copy of a manual test. So,
in the same way you would add a result, we'll have a bot account
add a result with a Pass or Fail to the daily image. It should also
leave a comment linking to the run so you can learn more if you are
curious. Simon has actually agreed to hack on this, so I hope we'll
start to see some bot results (though they will be failures!) on
the tracker soon. We could still use some help with fixing the
actual tests however, so they can provide value!
Nicholas
I would really really not want to see anything on the tracker from a
bot - that's reporting a fail on the test not the reality :(
[snip]
Hmm, well any other opinions? You are correct at this point in that
it would be showing failures which are not true. However, we want the
tests to run and show proper pass/fails! So it should be a temporary
thing. That said, we could not post results until the tests are
working, but it's certainly possible to have a failed test in the
future that isn't a real failure.
Nicholas
The trouble with posting any fail (assuming they run properly) from
these tests to the tracker is there is absolutely no way of knowing
what failed - just gobbledygook.
So a flavour QA team would have to run the image to see if the fail is
real or not, and 'where' it failed - at that point what have we gained?
It won't be adding a bug will it - or I would assume not.
Personally given the option to grab rss feeds from Jenkins - then
people interested in whether an image has failed could do that. That's
what I had intended way back when.
Maybe a seperate area on the tracker for them?
I was trying to keep it simple, and integrating it was the simpliest way
I could think of. A post to the notice board? Would that work better?
Nicholas
--
Ubuntu-quality mailing list
Ubuntu-quality@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality
