Hello,
this is a reply to the QA recap/feedback thread. As the original thread
went off track, I decided to start a new one to discuss the original
question at hand.
PACKAGE TESTING
First of all, I think it was a good move to run the package testing in
groups and in cadence before we hit the beta milestones. Running all
those tests and gathering a (big) list of bugs was and is important,
especially now that we have entered the "bug fixes only" stage of the
release preparing. I am sure we would be able to fix a lot less bugs
that are annoying and affect numerous of people.
That being said, I think the amount of calls was just about perfect for
an LTS cycle. I personally think we should go through all the groups
during regular releases as well, but possibly group more groups into one
call, and relax on the amount of testing "required". Optional tests
could be literally that; run if comfortable, but if they are left
untested, that's fine as well.
As to what (else) to test, I think we should try to focus on new
features, as we did this cycle. This can and probably should be extended
to running tests on applications that have had a major update during the
cycle. All of this in a flexible manner; the more new things we have
about to test, the looser running the other tests should be. Except on
the LTS releases...
I've yet to decide if some of the testcases are a bit too thorough or if
they are just about right. I guess we can agree and assume that the
amount of bugs is somewhat correlating with how deep the tests are. As I
see it though, the deeper and specific the tests are, the more mechanic
running them is. Which leads us to exploratory testing...
I have a few doubtful thoughts on exploratory testing. How do we
motivate people to run exploratory testing with the development version,
while it is not ready for production, or day-to-day environments? If the
tests aren't run on/as your main system, how can the testing be natural
enough to be of exploratory nature? How do we specify a good balance
between feature and exploratory testing?
MILESTONE (ISO) TESTING
It is hard to evaluate how the milestone ISO testing succeeded because
we still have one beta to go, which is also the most important
milestone. That is something where we can improve though.
The alpha releases could have been focused more on specific issues. Now
we kind of just ran through them without clear focus. Of course this
means that developers need to have their stuff together earlier in the
cycle, but that is a desirable direction generally.
I would rethink the amount of alpha releases we want to participate in
especially with non-LTS releases. We can opt-in for as many as we did
now if we have set a clear point of focus for those. This looks
unrealistic for T+1 though, as this cycle has been really busy for
everybody and we have got a lot of stuff that was prepared in the last 2
years included.
For the beta releases, we should get more publicity. We still have the
beta 2 release to come, so let's try to fix at least some of that for
Trusty.
CONCLUSION
To end the feedback on a positive note (though there weren't so many
negative points in total anyway), I think we have been up to the highest
possible standard with QA considering the size of our team and the
amount of new things landing this cycle.
Finally, a big THANK YOU Elfy for running the QA team, doing all the
calls, reporting back to us, taking care of bugs being noticed, features
landing in time et cetera... Last but not least, thanks for putting up
with us all who have sometimes more or less neglected our duties in QA
and being unresponsive to questions and calls. It is very much
appreciated, and I totally think that 14.04 would be a lesser release
without your work and persistence!
Cheers,
Pasi
--
Pasi Lallinaho (knome) » http://open.knome.fi/
Leader of Shimmer Project and Xubuntu » http://shimmerproject.org/
Graphic artist, webdesigner, Ubuntu member » http://xubuntu.org/
--
xubuntu-devel mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/xubuntu-devel
