there are 2 issues, email and JIRA tracking.
I don't have much to say to the email discussion. I find that if
some email gets sent every day I am going to ignore it unless I run
my tests and have a problem. In that case I can either check the
email or check a web site, so either solution is fine. The most
useful daily email to me would be a 1 line subject that says N nightly
failures across all platforms, and maybe a pointer in the email to
find more info.
As to the JIRA issue, I still would prefer a new category just because
I want it very easy to query. I think even the below proposal will
not allow for querying of just nightly test failures as there are bugs
that could (though I agree probably unlikely), fall into the proposed
classification and not be nightly test errors. If jira supported
some sort of "sub-category" that would be perfect. Does anyone feel
strongly enough to vote -1 on a new component?
To the guidelines it would be good to encourage people to add comments
to the bug in the case of intermittent failures, especially if they
see the problem in a different environment than has already been
proposed. Sometimes these can be a pain to track down and any evidence
you can gather helps, especially if the developer can never make it
happen in the environments that he has access to. The comments can
provide a historical trace of how often it is happening and where.
Process wise what should be done about an issue like myrna's DERBY-804.
A "fix" has been made so that nightly tests won't fail, by not running
the test. This is a reasonable first step, and sometimes for old JVM's
it is likely we just won't ever fix the test. One option is that the
bug gets moved off of the derby nightly testing component and into the
test category with an appropriate severity level setting (in this case
likely low).
David W. Van Couvering wrote:
Great email, John. May I summarize a potential proposal. Do people
feel we need a vote on this?
- Create a new alias, derby-test-results, that send *all* test results
- Send test *failures* to derby-dev
- Have a general guideline that test failures should be logged under the
Test component with a high priority (I am using Critical for
cross-platform issues and Major for single-platform issues).
- Ultimately committers are responsible for maintaining the quality of
the codeline and they may take further action where necessary (such as
vetoing any checkins except those related to test fixes).
Sound good?
David
John Embretsen wrote:
David W. Van Couvering wrote:
I would like to have an email list set up that sends out test
results, for those of us who would like a "push" model to see what's
going on. Personally I would want to see test failures sent to
derby-dev. If you don't like seeing them, you can filter them out,
but I think it's good for the developers to have awareness of what's
going on.
I think that combined with logging JIRA issues would be good. We
could use Kathey's idea to get a report on open test bugs by
filtering on the right component.
If it is agreed upon that we need to change the way developers get
alerted about test failures, I basically support what what is quoted
above. I've seen many good suggestions the past 24 hours for how to
increase awareness and potentially trigger action when a test in
derbyall fails. However....
With regards to the _awareness_ part of the challenge, what we need
(IMHO) more than yet another wiki-page or other kind of web page that
people have to manually update and/or navigate to, is some kind of
"push" mechanism such as automatically sending e-mails to some mailing
list.
Adding JIRA-entries sounds good to me, since e-mails are sent to
derby-dev when issues are created/updated (although this strategy seemed
a bit chaotic to me yesterday), thus increasing awareness. But, it
requires a certain amount of *manual* work. Someone has to have the itch
to check the actual test results (whether they are provided by Sun or
IBM or someone else), pick a test failure (if any exist), check if it
has been reported in JIRA already, and, if not, create a new JIRA issue,
etc. Because of this, I do not think this strategy is sufficient as a
long-term solution (but it may be a part of it).
I suggest that we focus on deciding:
a) Do we want to get automated e-mails reporting test results?
b) If so, how often should it be reported (after every test run, after
every test run with failures, after every test run with a certain
number of failures, once a day, once a day if failures occured, ...)?
(I guess the number of sources producing public test reports regularly
may be a factor here in the long run, but today there is only one such
source)
c) Where should such e-mails be sent (derby-dev, derby-test-results,
derby-commits, ...)?
...and that we try to keep this separate from the discussions of adding
new web-pages with test results, wiki-pages with test results, new
JIRA-types, etc.
