Great email, John. May I summarize a potential proposal. Do people
feel we need a vote on this?
- Create a new alias, derby-test-results, that send *all* test results
- Send test *failures* to derby-dev
- Have a general guideline that test failures should be logged under the
Test component with a high priority (I am using Critical for
cross-platform issues and Major for single-platform issues).
- Ultimately committers are responsible for maintaining the quality of
the codeline and they may take further action where necessary (such as
vetoing any checkins except those related to test fixes).
Sound good?
David
John Embretsen wrote:
David W. Van Couvering wrote:
I would like to have an email list set up that sends out test results,
for those of us who would like a "push" model to see what's going on.
Personally I would want to see test failures sent to derby-dev. If
you don't like seeing them, you can filter them out, but I think it's
good for the developers to have awareness of what's going on.
I think that combined with logging JIRA issues would be good. We
could use Kathey's idea to get a report on open test bugs by filtering
on the right component.
If it is agreed upon that we need to change the way developers get
alerted about test failures, I basically support what what is quoted
above. I've seen many good suggestions the past 24 hours for how to
increase awareness and potentially trigger action when a test in
derbyall fails. However....
With regards to the _awareness_ part of the challenge, what we need
(IMHO) more than yet another wiki-page or other kind of web page that
people have to manually update and/or navigate to, is some kind of
"push" mechanism such as automatically sending e-mails to some mailing
list.
Adding JIRA-entries sounds good to me, since e-mails are sent to
derby-dev when issues are created/updated (although this strategy seemed
a bit chaotic to me yesterday), thus increasing awareness. But, it
requires a certain amount of *manual* work. Someone has to have the itch
to check the actual test results (whether they are provided by Sun or
IBM or someone else), pick a test failure (if any exist), check if it
has been reported in JIRA already, and, if not, create a new JIRA issue,
etc. Because of this, I do not think this strategy is sufficient as a
long-term solution (but it may be a part of it).
I suggest that we focus on deciding:
a) Do we want to get automated e-mails reporting test results?
b) If so, how often should it be reported (after every test run, after
every test run with failures, after every test run with a certain
number of failures, once a day, once a day if failures occured, ...)?
(I guess the number of sources producing public test reports regularly
may be a factor here in the long run, but today there is only one such
source)
c) Where should such e-mails be sent (derby-dev, derby-test-results,
derby-commits, ...)?
...and that we try to keep this separate from the discussions of adding
new web-pages with test results, wiki-pages with test results, new
JIRA-types, etc.
begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:[EMAIL PROTECTED]
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard
