Re: Request for Comments: Package testing results analysis, result codes

2006-02-19 Thread Adam Kennedy
(Andreas J. Koenig) wrote: On Sun, 19 Feb 2006 22:22:20 +1100, Adam Kennedy <[EMAIL PROTECTED]> said: > 1. Broken or corrupt packaging. > A bad tarball, MANIFEST files missing. Make sure you verify that all files in the distro are readable. Reject if the permissions are bogus. Recentl

Re: Request for Comments: Package testing results analysis, result codes

2006-02-19 Thread Adam Kennedy
While an interesting idea, I forsee two challenges to doing this... Firstly is that it might turn an otherwise normal result into something else, with no clear rule. It makes a judgement call that some level of testing is good or bad, which isn't really the place of an installer to call. The

Re: Request for Comments: Package testing results analysis, result codes

2006-02-19 Thread Sébastien Aperghis-Tramoni
Tyler MacDonald wrote: Tests run, but >50% (or maybe >80%?) are skipped. From what I've seen, the most common cause of this is that the package is untestable with the current build configuration. Eg; you needed to specify a webserver or database or something to get these tests

Re: Request for Comments: Package testing results analysis, result codes

2006-02-19 Thread Michael Graham
Tyler MacDonald <[EMAIL PROTECTED]> wrote: > Adam, > I have one more edgey case I'd like to see on this list: > Tests run, but >50% (or maybe >80%?) are skipped. > > From what I've seen, the most common cause of this is that the > package is untestable with the current build

Re: Request for Comments: Package testing results analysis, result codes

2006-02-19 Thread Tyler MacDonald
Adam, I have one more edgey case I'd like to see on this list: > > 13. Tests exist, but fail to be executed. > There is tests, but the tests themselves aren't failing. > It's the build-process that is failing. > > 14. Tests run, and some/all tests fail. > The normal FAIL case

Re: Request for Comments: Package testing results analysis, result codes

2006-02-19 Thread Andreas J. Koenig
> On Sun, 19 Feb 2006 22:22:20 +1100, Adam Kennedy <[EMAIL PROTECTED]> said: > 1. Broken or corrupt packaging. > A bad tarball, MANIFEST files missing. Make sure you verify that all files in the distro are readable. Reject if the permissions are bogus. Recently we had an increasig nu

Request for Comments: Package testing results analysis, result codes

2006-02-19 Thread Adam Kennedy
I'm starting to get a bit closer (waiting on a test images and some last testing to be done) to finishing the initial PITA test cycle (and thus be able to do an initial release) and so I'm starting to do some prep work now for the next stage, which is to start to assemble some infrastructure ar