On Mon, 28 Sept 2020 at 22:49, Andrew Cagney <andrew.cag...@gmail.com> wrote: > > > > On Mon, 28 Sep 2020 at 21:53, Paul Wouters <p...@nohats.ca> wrote: >> >> On Mon, 28 Sep 2020, Andrew Cagney wrote: >> >> > I'm planning on removing the sanitizer ipsec-auto-up.n.sed. >> >> I understand why, but I also understand's Antony's point. >> >> To make it harder, once I rewrite the tests to also be documentation, >> we have an additionally issue of adding weird things to the configs >> people might copy. Like slow retransmit timeouts or impairs. > > > slow-retransmits should be deleted (I thought I had, I see I hadn't) > Your suspicion that IKEv1 didn't suppress retransmits turned out to be > because the tests in question weren't trying to supress retransmits.
I've deleted the slow-retransmits hack. >> >> >> I'm also a bit nervous if 90% of our tests use --impair >> delete-on-retransmit At one point are we still testing "regular operation" >> code with our tests. > > > Two types of tests: > 1. regression tests - these should be deterministic - someone reports a bug > or we add a feature and we add a test for it > 2. stress testing; the whole point of these is to be non-deterministic > but we're discussing deterministic tests here > >> >> Antony's filter in ipsec-auto-up.n.sed does address most of these >> issues. >> >> > > This assertion is not true. Please look at the examples I posted. It > silently removes important contextual information from test output - namely > that a retransmit occurred - while leaving behind the consequences. The > upshot is that a retransmit still causes the test to fail; only that the > retransmit happened is obscured. > > With the sanitizer disabled I got roughly the same number of fails. I've added the sanitizer sanitize-retransmits.sed, with a line like: ipsec auto --up westnet-eastnet-ikev2 # sanitize-retransmits it will strip retransmits from the output. This way a test wanting retransmits stripped from the output can do so explicitly. (I've not removed ipsec-auto-up.n.sed) >> >> But other issues still remain, such as retransmits causing >> different traffic counters in the output and causing false positives. >> >> >> So, I guess I have no good answers or strong opinions here. Just >> concerns :) >> >> Paul _______________________________________________ Swan-dev mailing list Swan-dev@lists.libreswan.org https://lists.libreswan.org/mailman/listinfo/swan-dev