Re: Parallel test execution: new option `parallel-tests': [1/4]
Hi Akim, again, thanks for your and Jim's valuable feedback! * Akim Demaille wrote on Thu, Oct 16, 2008 at 04:58:48PM CEST: The logic to create this kind of message is really complex, and in retrospect, I don't think that the message delivered is much easier to read than something more regular. So currently check.mk does as follows: Hmm. I'm still thinking about which type of summary I prefer. (Anyway your current summary would need singular/plurals fixed.) We introduced TFAIL, temporary failures: it's sort of short-term XFAIL. When a test starts to fail, it might be because someone uncovered a bug elsewhere, and that person might not be competent to address the real bug. Yet, leaving the test as is, failing, is a problem for everybody else who might waste a lot of time to discover that they have not introduced the problem, it was already there. So we make it TFAIL, which means should be processed soon. I'm not sure I understand the additional value of TFAIL. You have to mark it anyway, so why not mark it XFAIL? Put the other way round, what's the value of XFAIL if you are effectively going to ignore the expected failures anyway? Thanks, Ralf
Re: Parallel test execution: new option `parallel-tests': [1/4]
RW == Ralf Wildenhues [EMAIL PROTECTED] writes: +case fail=$$fail:xpass=$$xpass:xfail=$$xfail in \ + fail=0:xpass=0:xfail=0) \ +msg=$$All$$all $$tests passed. ; \ +exit=true;; \ + fail=0:xpass=0:xfail=*) \ +msg=$$All$$all $$tests behaved as expected; \ +if test $$xfail -eq 1; then xfailures=failure;\ +else xfailures=failures; fi;\ +msg=$$msg ($$xfail expected $$xfailures). ; \ +exit=true;; \ + fail=*:xpass=0:xfail=*) \ +msg=$$fail of $$all $$tests failed. ;\ +exit=false;;\ + fail=*:xpass=*:xfail=*) \ +msg=$$failures of $$all $$tests did not behave as expected; \ +if test $$xpass -eq 1; then xpasses=pass; \ +else xpasses=passes; fi;\ +msg=$$msg ($$xpass unexpected $$xpasses). ; \ +exit=false;;\ + *)\ +echo 2 incorrect case; exit 4;; \ +esac; \ +if test $$skip -ne 0; then\ + if test $$skip -eq 1; then \ +msg=$$msg($$skip test was not run). ;\ + else \ +msg=$$msg($$skip tests were not run). ; \ + fi; \ +fi; \ The logic to create this kind of message is really complex, and in retrospect, I don't think that the message delivered is much easier to read than something more regular. So currently check.mk does as follows: case fail=$$fail:xpass=$$xpass in \ fail=0:xpass=0) \ msg=The test suite passed. ; \ exit=true;; \ *)\ msg=The test suite failed. ; \ exit=false;;\ esac; \ msg=$$msg - $$pass/$$all tests passed. ; \ test $$fail -eq 0 ||\ msg=$$msg - $$fail failures. ; \ test $$xpass -eq 0 || \ msg=$$msg - $$xpass unexpected pass. ; \ test $$xfail -eq 0 || \ msg=$$msg - $$xfail expected failures. ; \ test $$tfail -eq 0 || \ msg=$$msg - $$tfail expected temporary failures. ; \ test $$skip -eq 0 ||\ msg=$$msg - $$skip skipped tests. ; \ It's simpler to create the message, and I think, the message is also simpler. We introduced TFAIL, temporary failures: it's sort of short-term XFAIL. When a test starts to fail, it might be because someone uncovered a bug elsewhere, and that person might not be competent to address the real bug. Yet, leaving the test as is, failing, is a problem for everybody else who might waste a lot of time to discover that they have not introduced the problem, it was already there. So we make it TFAIL, which means should be processed soon.
