On Mar 29, 2007, at 10:14 PM, Will Coleda wrote:
On Mar 27, 2007, at 4:45 PM, Allison Randal wrote:
<particle> but, we need better smoke tools
So lets document what we need. Right now 'make smoke' generates an
HTML report which is uploaded to the smoke server.
Talk has happened in the past about making this more DB like instead
of rendered output, but my concern is for the user visible features
we're lacking. Perhaps:
1) Notify mailing list on failure (threshold of 9x%?)
2) Notify mailing list on new platform/compiler?
3) Better GUI on the server, including:
a - sort by <foo>?
b - highlight platforms with failures on summary screen?
c - ability to which tests are failing across platforms. only on
single platform?
4) Not have to search through a whole page to find that one failing
test, or "unexpected succeeded" test, etc.
5) Easily compare different revisions of one platform, and two
platforms with near revisions, since unless we get cluster of very
heterogeneous computers, we must rely on user submissions. Updates to
languages/ don't really matter unless you're smoking languages for
instance, and sometimes a lot of the revision updates are to
languages/. Without a lot of work, I can't go to the smoke page and
find out the failing differences between revision, say 16000 and 17000.
Pugs has done some nifty stuff with their smokes since we originally
borrowed the concept. Perhaps we can rebase to their current code.
Once we've decided what we want/need, we can open a ticket (or tickets
on various components) and worry about implementation.
Feedback, please.
One other thing I've noticed is that todo tests sometimes become
forgotten tests. And since they're sometimes platform specific, they
don't get fixed for that platform because feature x doesn't have the
code support. Other than doing a grep of t/ or something similar,
there's no simple way for being aware of bugs that are todo'd.
--
Will "Coke" Coleda
[EMAIL PROTECTED]
