Mike Matrigali wrote:
there are 2 issues, email and JIRA tracking.
I don't have much to say to the email discussion. I find that if
some email gets sent every day I am going to ignore it unless I run
my tests and have a problem. In that case I can either check the
email or check a web site, so either solution is fine. The most
useful daily email to me would be a 1 line subject that says N nightly
failures across all platforms, and maybe a pointer in the email to
find more info.
Are you amenable to a derby-test-results list and emails to derby-dev
when there is a failure? The idea would be that these failure emails
would be RARE, otherwise we have a problem :)
As to the JIRA issue, I still would prefer a new category just because
I want it very easy to query. I think even the below proposal will
not allow for querying of just nightly test failures as there are bugs
that could (though I agree probably unlikely), fall into the proposed
classification and not be nightly test errors. If jira supported
some sort of "sub-category" that would be perfect. Does anyone feel
strongly enough to vote -1 on a new component?
No, I actually think your idea makes sense, it would help separate out
bugs in the tests to actual test failures.
To the guidelines it would be good to encourage people to add comments
to the bug in the case of intermittent failures, especially if they
see the problem in a different environment than has already been
proposed. Sometimes these can be a pain to track down and any evidence
you can gather helps, especially if the developer can never make it
happen in the environments that he has access to. The comments can
provide a historical trace of how often it is happening and where.
Maybe we need a Wiki page that contain these guidelines. I guess since
I brought this up I can work on that.
Process wise what should be done about an issue like myrna's DERBY-804.
A "fix" has been made so that nightly tests won't fail, by not running
the test. This is a reasonable first step, and sometimes for old JVM's
it is likely we just won't ever fix the test. One option is that the
bug gets moved off of the derby nightly testing component and into the
test category with an appropriate severity level setting (in this case
likely low).
I think that your idea makes sense, move the bug off nightly tests
component to tests, and assign it an appropriate category.
David W. Van Couvering wrote:
Great email, John. May I summarize a potential proposal. Do people
feel we need a vote on this?
- Create a new alias, derby-test-results, that send *all* test results
- Send test *failures* to derby-dev
- Have a general guideline that test failures should be logged under
the Test component with a high priority (I am using Critical for
cross-platform issues and Major for single-platform issues).
- Ultimately committers are responsible for maintaining the quality of
the codeline and they may take further action where necessary (such as
vetoing any checkins except those related to test fixes).
Sound good?
David
John Embretsen wrote:
David W. Van Couvering wrote:
I would like to have an email list set up that sends out test
results, for those of us who would like a "push" model to see what's
going on. Personally I would want to see test failures sent to
derby-dev. If you don't like seeing them, you can filter them out,
but I think it's good for the developers to have awareness of what's
going on.
I think that combined with logging JIRA issues would be good. We
could use Kathey's idea to get a report on open test bugs by
filtering on the right component.
If it is agreed upon that we need to change the way developers get
alerted about test failures, I basically support what what is quoted
above. I've seen many good suggestions the past 24 hours for how to
increase awareness and potentially trigger action when a test in
derbyall fails. However....
With regards to the _awareness_ part of the challenge, what we need
(IMHO) more than yet another wiki-page or other kind of web page that
people have to manually update and/or navigate to, is some kind of
"push" mechanism such as automatically sending e-mails to some
mailing list.
Adding JIRA-entries sounds good to me, since e-mails are sent to
derby-dev when issues are created/updated (although this strategy seemed
a bit chaotic to me yesterday), thus increasing awareness. But, it
requires a certain amount of *manual* work. Someone has to have the itch
to check the actual test results (whether they are provided by Sun or
IBM or someone else), pick a test failure (if any exist), check if it
has been reported in JIRA already, and, if not, create a new JIRA issue,
etc. Because of this, I do not think this strategy is sufficient as a
long-term solution (but it may be a part of it).
I suggest that we focus on deciding:
a) Do we want to get automated e-mails reporting test results?
b) If so, how often should it be reported (after every test run, after
every test run with failures, after every test run with a certain
number of failures, once a day, once a day if failures occured, ...)?
(I guess the number of sources producing public test reports regularly
may be a factor here in the long run, but today there is only one such
source)
c) Where should such e-mails be sent (derby-dev, derby-test-results,
derby-commits, ...)?
...and that we try to keep this separate from the discussions of adding
new web-pages with test results, wiki-pages with test results, new
JIRA-types, etc.
begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:[EMAIL PROTECTED]
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard
