Re: [yocto] [ptest-runner 5/5] main: Do not return number of failed tests when calling ptest-runner
Hi Lukasz Also our test infrastructure expects an exit value not equal to 0 in case of a failed test. Regards, Adrian On Wed, 2021-07-21 at 11:46 +0200, ?ukasz Majewski wrote: > Up till now ptest-runner2 returns number of failed tests with its > exit status code. Such use case is not recommended [1] and may cause > issues when there are more than 256 tests to be executed. > > To alleviate this issue the number of total tests with number of failed > ones is printed before exit. To be more specific - failure of a single > test doesn't indicate that the ptest-runner itself encounter any issue > during its execution. > > One can test this change with executing: > ./ptest-runner -d tests/data fail > > Links: > [1] - https://www.gnu.org/software/libc/manual/html_node/Exit-Status.html > > Signed-off-by: Lukasz Majewski > --- > main.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/main.c b/main.c > index efa21b2..9f03857 100644 > --- a/main.c > +++ b/main.c > @@ -219,6 +219,9 @@ main(int argc, char *argv[]) > ptest_list_remove(run, opts.exclude[i], 1); > > > rc = run_ptests(run, opts, argv[0], stdout, stderr); > + fprintf(stdout, "TOTAL: %d FAIL: %d\n", ptest_list_length(run), rc); > + if (rc > 0) > + rc = 0; > > > ptest_list_free_all(); > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#54239): https://lists.yoctoproject.org/g/yocto/message/54239 Mute This Topic: https://lists.yoctoproject.org/mt/84352910/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [ptest-runner 5/5] main: Do not return number of failed tests when calling ptest-runner
Hi all, Just replied in other email about this situation, so yes for me is better at least return 1 when any test fails to make it backwards compatible and knows when some test fails. Regards, Anibal On Mon, 26 Jul 2021 at 02:00, Tero Kinnunen wrote: > I would favor to keep non-zero exit value if any tests failed. We have > relied on exit value since ptest started to support it, others may be doing > that as well. This would be backward incompatible change, making tests > silently fail. It is somewhat clumsy to try to parse failed tests from > output, so non-zero rc was very welcome change to me. > > Looking at other test runners, it is very common to return non-zero if any > tests failed. (Some examples: pytest, python unittest, googletest, shunit2, > robot framework.) See pytest exit codes for example: > https://docs.pytest.org/en/latest/reference/exit-codes.html > > Indeed the failed count is not a good exit value though because of > rollover, should be fixed. Could consider using own rc for "any tests > failed" differentiating from other errors. > > - Tero > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#54213): https://lists.yoctoproject.org/g/yocto/message/54213 Mute This Topic: https://lists.yoctoproject.org/mt/84352910/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto] [ptest-runner 5/5] main: Do not return number of failed tests when calling ptest-runner
I would favor to keep non-zero exit value if any tests failed. We have relied on exit value since ptest started to support it, others may be doing that as well. This would be backward incompatible change, making tests silently fail. It is somewhat clumsy to try to parse failed tests from output, so non-zero rc was very welcome change to me. Looking at other test runners, it is very common to return non-zero if any tests failed. (Some examples: pytest, python unittest, googletest, shunit2, robot framework.) See pytest exit codes for example: https://docs.pytest.org/en/latest/reference/exit-codes.html Indeed the failed count is not a good exit value though because of rollover, should be fixed. Could consider using own rc for "any tests failed" differentiating from other errors. - Tero -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#54208): https://lists.yoctoproject.org/g/yocto/message/54208 Mute This Topic: https://lists.yoctoproject.org/mt/84352910/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[yocto] [ptest-runner 5/5] main: Do not return number of failed tests when calling ptest-runner
Up till now ptest-runner2 returns number of failed tests with its exit status code. Such use case is not recommended [1] and may cause issues when there are more than 256 tests to be executed. To alleviate this issue the number of total tests with number of failed ones is printed before exit. To be more specific - failure of a single test doesn't indicate that the ptest-runner itself encounter any issue during its execution. One can test this change with executing: ./ptest-runner -d tests/data fail Links: [1] - https://www.gnu.org/software/libc/manual/html_node/Exit-Status.html Signed-off-by: Lukasz Majewski --- main.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/main.c b/main.c index efa21b2..9f03857 100644 --- a/main.c +++ b/main.c @@ -219,6 +219,9 @@ main(int argc, char *argv[]) ptest_list_remove(run, opts.exclude[i], 1); rc = run_ptests(run, opts, argv[0], stdout, stderr); + fprintf(stdout, "TOTAL: %d FAIL: %d\n", ptest_list_length(run), rc); + if (rc > 0) + rc = 0; ptest_list_free_all(); -- 2.20.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#54182): https://lists.yoctoproject.org/g/yocto/message/54182 Mute This Topic: https://lists.yoctoproject.org/mt/84352910/21656 Group Owner: yocto+ow...@lists.yoctoproject.org Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
