Hi, The reason why Functest-suite job is giving false positives in CI is because of the Functest CLI. This job is mainly used by verify gates and other 3rd party jobs like ODL-Netvirt, ODL-CSIT, and will be more often used.
This only happens with the 'suite' job and not with 'daily' jobs, since we call the 'run_tests.py' script directly instead of the CLI. https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/functest/functest-suite.sh;h=f28d3d0377fcc40f30c41f64a1f2619ff4648e1d;hb=2b72e01a433d886999f2159e996a65fb7d855706#l10 Looking at this line, we use the CLI command (functest testcase run ...) in our Functest-suite job, so that you can run a single test case or a given list of test cases. Functest CLI is based on click http://click.pocoo.org/5/. The way it works is that the functions that are called from the CLI are not "functions" any more, they become a CLI program, then it will always return 0 if the command execution is OK, regardless of what the implemented function returns. Thus, the command "functest testcase run" will return always 0 whatever test result we have. Example of a failed test execution here: http://pastebin.com/raw/97uUkeYj We have some options to solve this: 1) Swap the Functest-suite command to use the python script run_tests.py instead of the CLI command. Problem --> cannot run more than 1 test case at a time. With the CLI you can do things like: "functest testcase run healtcheck, vping, tempest" but with run_tests.py you can only call 1 test case at a time, i.e. 'run_tests.py -t vping.' 2) Parse the testcase name in the Jenkins Job and if it is a list separated by comas, we call run_tests.py as many times as needed. --> too much logic in the Jenkins job 3) Raise an exception in the CLI if the test case fails --> effective but ugly http://pastebin.com/raw/MEvpbm3c What are your preferences? Thanks, Jose
_______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
