Re: Add a "random" target to bsd.regress.mk

[Scott Cheloha]

> On Mar 22, 2017, at 11:47 AM, Alexander Bluhm  wrote:

> 
> I would not do all this work.  I have indepentent targets for some
> of my own tests that I use regualry for development.  But for other
> people's tests which just pass, I don't care.  Making the individual
> targets independent would be work without anybody noticing it.
> 
> Running all tests in a row will leave the system in a state where
> in a second test run some tests fail.  So if you intend to work in
> that area, are you might want to fix this.  The common setup,
> cleanup, and resources that all regress tests share may be a problem.
> 
> At http://bluhm.genua.de/regress/results/regress.html are the daily
> test results.  We also have tests that fail sometimes and could be
> more stable.

Cool, I'll look into it further on a case-by-case basis.

-Scott

Re: Add a "random" target to bsd.regress.mk

[Alexander Bluhm]

On Tue, Mar 21, 2017 at 08:56:02PM -0500, Scott Cheloha wrote:
> > Do you plan to improve this situation?
> 
> It depends.  Do you think improving the independence of each of those
> cases is valuable in and of itself?

I would not do all this work.  I have indepentent targets for some
of my own tests that I use regualry for development.  But for other
people's tests which just pass, I don't care.  Making the individual
targets independent would be work without anybody noticing it.

Running all tests in a row will leave the system in a state where
in a second test run some tests fail.  So if you intend to work in
that area, are you might want to fix this.  The common setup,
cleanup, and resources that all regress tests share may be a problem.

At http://bluhm.genua.de/regress/results/regress.html are the daily
test results.  We also have tests that fail sometimes and could be
more stable.

bluhm

Re: Add a "random" target to bsd.regress.mk

[Scott Cheloha]

> On Mar 20, 2017, at 9:13 AM, Alexander Bluhm  wrote:
> 
> I would expect to find more issues in the tests than in the software.
> 

> Unfortunately I fear a lot of our tests do that.
> 

> Independent tests cases are useful when fixing tests or doing test
> driven development.  For finding regressions or getting more tests
> cases for the software they are not so relevant.

Okay, okay, agreed.

> Did you have a look how many of our regress tests have independent
> targets?

I had not, but this shell script:

cd /usr/src/regress
total_tgts=0
tests_w_indie_tgts=0
for mk in $(find . -name "Makefile"); do
targets=$(make -C $(dirname $mk) -V REGRESS_TARGETS regress)
[ -n "$targets" ] && echo "$(dirname $mk): $targets"
ntgts=$(echo $targets | wc -w)
[ $ntgts -gt 1 ] && tests_w_indie_tgts=$((tests_w_indie_tgts + 
1)) 
total_tgts=$((total_tgts + ntgts))
done
echo at least $tests_w_indie_tgts tests with independent targets
echo at least $total_tgts targets in total

leads me to believe there are at least 164 tests with independent targets.
So, ballpark, probably (for real) at least 200.

> Do you plan to improve this situation?

It depends.  Do you think improving the independence of each of those
cases is valuable in and of itself?  Is there any particular group
of tests where you think that might be valuable?  If I were working on
some specific piece of the system I might personally find it useful, but
I'm still figuring out the lay of the land, so, I can't say for sure.

> Having a feature that would fail for 80% of our tests does not make
> sense.  I think you try to implement the second step before the first.

> 
> Look at the tests and figure out wether you want to fix all
> of them.  I expect a lot of work.  After this has happened, we could
> add your target.

Agreed, noted.

--
Scott Cheloha

Re: Add a "random" target to bsd.regress.mk

[Scott Cheloha]

> On Mar 18, 2017, at 8:48 PM, trondd  wrote:
> 
> [...]
> 
> Actually, rereading what I quoted, I see you're concerned with unmasking
> false positives, which can be manually rerun in isolation to reproduce. 
> My thought still stands for the flip-side where a test fails due to a
> previous test.

I think determining a false negative *is* a false negative would require
basically the same steps:

* Run the test in isolation.

* Did it fail?  If yes, this is probably not a false negative.  If not,
  you could have a false negative.

* Try to reproduce the steps that caused it to fail the first time (you
  have a record, e.g., on your console, in your logs, etc.).

* Can you reproduce it?  If you can, you probably have a false negative.
  Otherwise, something else is off.

Not an exact science by any means, but a start.

Unless I misunderstood your question.

--
Scott Cheloha

Re: Add a "random" target to bsd.regress.mk

[Alexander Bluhm]

On Sat, Mar 18, 2017 at 05:42:20PM -0500, Scott Cheloha wrote:
> My thinking is that statically ordered regression runs can mask
> bugs in the software under test and the tests themselves.

I would expect to find more issues in the tests than in the software.

> In general, a test can put your system into a state that allows a
> subsequent test to pass when it would have otherwise failed.

Unfortunately I fear a lot of our tests do that.

> Randomly ordered regression runs could help to weed these issues
> out.  Even if no such bugs exist, your randomly ordered regression
> runs are stronger statements about your software's correctness and
> the independence of each test case.

Independent tests cases are useful when fixing tests or doing test
driven development.  For finding regressions or getting more tests
cases for the software they are not so relevant.

Did you have a look how many of our regress tests have independent
targets?  Do you plan to improve this situation?

Having a feature that would fail for 80% of our tests does not make
sense.  I think you try to implement the second step before the
first.  Look at the tests and figure out wether you want to fix all
of them.  I expect a lot of work.  After this has happened, we could
add your target.

bluhm

Re: Add a "random" target to bsd.regress.mk

[trondd]

On Sat, March 18, 2017 6:42 pm, Scott Cheloha wrote:
> In general, a test can put your system into a state that allows a
> subsequent test to pass when it would have otherwise failed.
>

>
> Any takers?  Thoughts?
>

If such a bug is revealed, how does someone rerun the tests in the same
order to reproduce the bug or verify when it's fixed?

"Manually" is tough if you don't know which previous test of many may have
caused the problematic system state.

Actually, rereading what I quoted, I see you're concerned with unmasking
false positives, which can be manually rerun in isolation to reproduce. 
My thought still stands for the flip-side where a test fails due to a
previous test.

Tim.

Add a "random" target to bsd.regress.mk

[Scott Cheloha]

Hi tech@,

The patch below adds a target, "random", to bsd.regress.mk, as
well as a description of the new target to bsd.regress.mk(5).

The "random" target is the same as "regress", except that the
ordering of the regression targets is randomized each time.

My thinking is that statically ordered regression runs can mask
bugs in the software under test and the tests themselves.

For small utilities this is unlikely, but for long-running daemons,
kernel modules, etc., it is more likely because these programs are
more complex.

In general, a test can put your system into a state that allows a
subsequent test to pass when it would have otherwise failed.

Randomly ordered regression runs could help to weed these issues
out.  Even if no such bugs exist, your randomly ordered regression
runs are stronger statements about your software's correctness and
the independence of each test case.

Any takers?  Thoughts?

--
Scott Cheloha

Index: share/man/man5/bsd.regress.mk.5
===
RCS file: /cvs/src/share/man/man5/bsd.regress.mk.5,v
retrieving revision 1.11
diff -u -p -r1.11 bsd.regress.mk.5
--- bsd.regress.mk.516 Jan 2017 07:00:03 -  1.11
+++ bsd.regress.mk.518 Mar 2017 21:24:38 -
@@ -50,6 +50,10 @@ test programs.
 .It Cm depend
 Build any dependencies required to carry out the current set
 of regression tests.
+.It Cm random
+Identical to
+.Cm regress ,
+but regression targets are executed in a random order.
 .It Cm regress
 Executes all of the regression targets defined in the Makefile.
 If one of the tests fails, the line
Index: share/mk/bsd.regress.mk
===
RCS file: /cvs/src/share/mk/bsd.regress.mk,v
retrieving revision 1.13
diff -u -p -r1.13 bsd.regress.mk
--- bsd.regress.mk  19 Jul 2014 18:15:53 -  1.13
+++ bsd.regress.mk  18 Mar 2017 22:23:40 -
@@ -83,6 +83,11 @@ REGRESS_SKIP_TARGETS+=${REGRESS_ROOT_TAR
 .  endif
 .endif 
 
+.ifmake random
+REGRESS_TARGETS!=echo ${REGRESS_TARGETS} | tr ' ' '\n' | sort -R
+.endif
+random: regress
+
 regress: .SILENT
 .if ! ${REGRESS_LOG:M/*}
echo =