On 20/03/14 23:53, Elizabeth Krumbach Joseph wrote:
On Thu, Mar 20, 2014 at 4:38 PM, Pasi Lallinaho <[email protected]> wrote:
this is a reply to the QA recap/feedback thread. As the original thread went
off track, I decided to start a new one to discuss the original question at
hand.
Thanks, I do hope we can keep this one on track as it's an important discussion
PACKAGE TESTING
First of all, I think it was a good move to run the package testing in
groups and in cadence before we hit the beta milestones. Running all those
tests and gathering a (big) list of bugs was and is important, especially
now that we have entered the "bug fixes only" stage of the release
preparing. I am sure we would be able to fix a lot less bugs that are
annoying and affect numerous of people.
That being said, I think the amount of calls was just about perfect for an
LTS cycle. I personally think we should go through all the groups during
regular releases as well, but possibly group more groups into one call, and
relax on the amount of testing "required". Optional tests could be literally
that; run if comfortable, but if they are left untested, that's fine as
well.
As far as the calls for testing go, both for packages (even though I
didn't participate, sorry!) and ISOs (where I did!), I found the
direct emails via Launchpad super helpful (they go to my Inbox, not
filtered to -devel). It also really made me feel like I knew what was
going on with testing so when others asked me how they could jump in I
had a good starting point to know where the help was required.
This is useful to know - thanks :)
As to what (else) to test, I think we should try to focus on new features,
as we did this cycle. This can and probably should be extended to running
tests on applications that have had a major update during the cycle. All of
this in a flexible manner; the more new things we have about to test, the
looser running the other tests should be. Except on the LTS releases...
I've yet to decide if some of the testcases are a bit too thorough or if
they are just about right. I guess we can agree and assume that the amount
of bugs is somewhat correlating with how deep the tests are. As I see it
though, the deeper and specific the tests are, the more mechanic running
them is. Which leads us to exploratory testing...
I'd love to hear some feedback about the thoroughness of the
testcases. We don't want folks to be put off when they see the test
case being long, but at the same time it won't have much value if it's
short.
I too would love to get feedback on testcases - but I would much rather
that came as specific mails to the list, rather than lumped in with this.
I have a few doubtful thoughts on exploratory testing. How do we motivate
people to run exploratory testing with the development version, while it is
not ready for production, or day-to-day environments? If the tests aren't
run on/as your main system, how can the testing be natural enough to be of
exploratory nature? How do we specify a good balance between feature and
exploratory testing?
I think what we'd struggle with is not people being unwilling to do
the testing, as we know there are lots of people who do actually run
development versions since we're always hearing feedback about how
stable it is :) I think the issue is connecting them with bug
reporting and other mechanisms for reporting results. I think if we
even got feedback given via the mailing list it would be helpful. Not
sure how to make this easier for people.
I'm not sure that having a launchpad account and going to the tracker
could be much easier. While people's mails to the list do get read. It's
not so easy to follow up. A bug reported via the tracker ticks all the
boxes.
MILESTONE (ISO) TESTING
It is hard to evaluate how the milestone ISO testing succeeded because we
still have one beta to go, which is also the most important milestone. That
is something where we can improve though.
The alpha releases could have been focused more on specific issues. Now we
kind of just ran through them without clear focus. Of course this means that
developers need to have their stuff together earlier in the cycle, but that
is a desirable direction generally.
I would rethink the amount of alpha releases we want to participate in
especially with non-LTS releases. We can opt-in for as many as we did now if
we have set a clear point of focus for those. This looks unrealistic for T+1
though, as this cycle has been really busy for everybody and we have got a
lot of stuff that was prepared in the last 2 years included.
For the beta releases, we should get more publicity. We still have the beta
2 release to come, so let's try to fix at least some of that for Trusty.
I could have done a much better job at handling social media for this
cycle, I should discipline myself to send something out whenever a
call hits my Inbox... or add more admins on social media to help
handle this.
This could also be helped by -QA people pinging you in channel :)
CONCLUSION
To end the feedback on a positive note (though there weren't so many
negative points in total anyway), I think we have been up to the highest
possible standard with QA considering the size of our team and the amount of
new things landing this cycle.
Finally, a big THANK YOU Elfy for running the QA team, doing all the calls,
reporting back to us, taking care of bugs being noticed, features landing in
time et cetera... Last but not least, thanks for putting up with us all who
have sometimes more or less neglected our duties in QA and being
unresponsive to questions and calls. It is very much appreciated, and I
totally think that 14.04 would be a lesser release without your work and
persistence!
Absolutely, Elfy's really done an exceptional job staying on top of
all of this even with all his other commitments to Ubuntu and beyond.
Thank you for your work!
--
Ubuntu Forum Council Member
Xubuntu QA Lead
--
xubuntu-devel mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/xubuntu-devel
