> On Mar 20, 2017, at 9:13 AM, Alexander Bluhm <alexander.bl...@gmx.net> wrote:
>
> I would expect to find more issues in the tests than in the software.
>
> Unfortunately I fear a lot of our tests do that.
>
> Independent tests cases are useful when fixing tests or doing test
> driven development. For finding regressions or getting more tests
> cases for the software they are not so relevant.
Okay, okay, agreed.
> Did you have a look how many of our regress tests have independent
> targets?
I had not, but this shell script:
cd /usr/src/regress
total_tgts=0
tests_w_indie_tgts=0
for mk in $(find . -name "Makefile"); do
targets=$(make -C $(dirname $mk) -V REGRESS_TARGETS regress)
[ -n "$targets" ] && echo "$(dirname $mk): $targets"
ntgts=$(echo $targets | wc -w)
[ $ntgts -gt 1 ] && tests_w_indie_tgts=$((tests_w_indie_tgts +
1))
total_tgts=$((total_tgts + ntgts))
done
echo at least $tests_w_indie_tgts tests with independent targets
echo at least $total_tgts targets in total
leads me to believe there are at least 164 tests with independent targets.
So, ballpark, probably (for real) at least 200.
> Do you plan to improve this situation?
It depends. Do you think improving the independence of each of those
cases is valuable in and of itself? Is there any particular group
of tests where you think that might be valuable? If I were working on
some specific piece of the system I might personally find it useful, but
I'm still figuring out the lay of the land, so, I can't say for sure.
> Having a feature that would fail for 80% of our tests does not make
> sense. I think you try to implement the second step before the first.
>
> Look at the tests and figure out wether you want to fix all
> of them. I expect a lot of work. After this has happened, we could
> add your target.
Agreed, noted.
--
Scott Cheloha
