Brian,
A brute-force means of achieving your goal would be to use the "TEST="
option to run each test individually:
TEST=<filter> - filter the set of tests:
by file-name - only run tests from specified file, e.g.
TEST=test_bfd selects all tests from test_bfd.py
by file-suffix - same as file-name, but 'test_' is omitted e.g.
TEST=bfd selects all tests from test_bfd.py
by wildcard - wildcard filter is <file>.<class>.<test
function>, each can be replaced by '*'
e.g. TEST='test_bfd.*.*' is equivalent to above
example of filter by file-name
TEST='bfd.*.*' is equivalent to above
example of filter by file-suffix
TEST='bfd.BFDAPITestCase.*' selects all
tests from test_bfd.py which are part of BFDAPITestCase class
TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named
test_add_bfd from test_bfd.py/BFDAPITestCase
TEST='*.*.test_add_bfd' selects all test
functions named test_add_bfd from all files/classes
With a bit of bash scripting, this turns out to be quite easy. Here's a
one-liner which executes all of the l2xc tests in .../vpp/test/test_l2xc.py:
for test in $(grep "def test_" test/test_l2xc.py | awk -e '{print $2}' |
cut -d'(' -f1) ; do make TEST="*.*.$test" test ; done
Thanks,
-daw-
On 10/02/2017 03:44 AM, Klement Sekera -X (ksekera - PANTHEON
TECHNOLOGIES at Cisco) wrote:
Hi Brian,
FAILFAST=0 is the default, so there is no need to specify it.
Usually, when a test case 'runs forever', it means that the VPP crashed
while processing an API call (did you see a message about a core found
in temporary directory?), in which case python will be stuck waiting for
a lock. Now, the problem is that there is no way to cancel that stuck
thread in Python (which is for most cases the main thread).
So to work around this issue, the following approach is taken: First
thing the test framework does is fork - child continues running
the tests while parent monitors the child. There is a pipe open,
through which child sends keep-alive objects containing the name of
the test case, its temporary directory and some other info whenever
setUp() runs. That way parent can tell that the child is progressing
through the tests and also can say which test caused timeout.
I spent quite some time trying to make it work so that the tests can
continue, but unless we refactor the code so that each test runs in its
own forked process, we won't be able to achieve your goal. So far, 100%
of these timeouts were caused by VPP dumping a core and it seemed to me
like a waste of time spending extra time to make the tests finish.
Thanks,
Klement
Quoting Brian Brooks (2017-09-22 19:07:53)
Is there a way to make:
FAILFAST=0 TIMEOUT=t make test
actually continue on to the next test if a test has reached timeout t?
The motivation is that I see at least one test case running for forever,
and I want to be able to see how many test cases are failing in total.
I didn't see that this was possible to do out of the box, so I tried one
approach: launch a thread that waits until the timeout has passed and then
send a term signal to the vpp process. If the test case completes before
timeout, the test case's "tearDown" routine will cancel the timeout thread.
The timeout and killing part works, except the test framework still does
not continue on to the next test case.
Thoughts?
Thanks,
Brian
_______________________________________________
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev
_______________________________________________
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev
_______________________________________________
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev
