Re: What is a 'program' in Section 15.3.3.1?
In looking at tap-driver.sh there doesn't appear to be a place where a 'program' is accepted on the input command line. It appears that after all options are read if the input command line '$#' is not zero then an error is declared. So, is the TAP interface different from other Custom Test Driver interfaces? The failure of the past is the challenge of the present and the success of the future.
RE: What is a 'program' in Section 15.3.3.1?
Thanks. (Also see lists.gnu.org/archive/html/automake/2015-05/msg00051.html How are data files input for to custom drivers?) The TAP interface is very different. TAP is supposed to be a custom test driver. Is the interface and manner of calling different from other custom drivers, and the API and other comments describing custom drivers in the manual? It depends on subordinate scripts printing I'm very confused here. It looks to me like tap-driver.sh is a standalone script and doesn't need any help scripts. The input data is processed in awk and all the needed functions are defined in this context. For GraphicsMagick, I created a script which is sourced by actual test scripts (.tap scripts), which provides generic functions used by those scripts. The functions support pass/fail detection based on program exit status, but also support 'xfail' by being passed a set of feature id strings which are matched which the actual available features. If the test fails, but a necessary feature is missing, then an extra diagnosis is added which Automake's TAP script will turn into an informative XFAIL message: I am confused. Using Automake the Developer can generate a reference to a class of test cases defined in the TESTS variable. Each one of the scripts is required to output the results of one of more tests that they run in quasi-TAP format. The TAP script, tap-driver.sh, takes the output and generates a trs file. Included with the tap-driver.sh there is a means to generate XFAIL and XPASS, however this seems to be global to all subtests executed in a TESTS defined test. Each ok or not ok returned will be translated (as required) to XPASS or XFAIL depending on --expect-failure=yes. As an example: TEST_EXTENSIONS =.abc ABC_LOG_DRIVER = tap-driver.sh TESTS = test1.abc test2.abc Means that tap-driver.sh is called twice, once for test1.abc, once for test2.abc. Each of these tests can return one or more of ok, not ok, skip, bail, or a YAML-like text string defined by --diagnostic-string and --comments. Another point of confusion is that the options defined for the custom test drivers have the format a=b if 'a' has a value, but tap-driver.sh does not allow a=b, it requires a b. Anyhow, thanks for the heads up, I will be looking at your files. And, as always, I am confused.
RE: What is a 'program' in Section 15.3.3.1?
On Wed, 27 May 2015, Arthur Schwarz wrote: TAP is supposed to be a custom test driver. Is the interface and manner of calling different from other custom drivers, and the API and other comments describing custom drivers in the manual? I think that the method of calling is similar to other type of tests except that the list of tests 'TESTS' is a list of scripts which produce TAP output strings. It depends on subordinate scripts printing I'm very confused here. It looks to me like tap-driver.sh is a standalone script and doesn't need any help scripts. The input data is processed in awk and all the needed functions are defined in this context. If the test program was a C program so it was a binary and printed 'ok' and 'not ok' outputs, and is able to find any input files without assistance, then no extra shim script should be required. In my case I was replacing perhaps 1000 individual classic Automake test scripts and wanted to replace them with far fewer TAP scripts. I am confused. Using Automake the Developer can generate a reference to a class of test cases defined in the TESTS variable. Each one of the scripts is required to output the results of one of more tests that they run in quasi-TAP format. The TAP script, tap-driver.sh, takes the output and generates a trs file. Included with the tap-driver.sh there is a means to generate XFAIL and XPASS, however this seems to be global to all subtests executed in a TESTS defined test. Each ok or not ok returned will be translated (as required) to XPASS or XFAIL depending on --expect-failure=yes. A TAP test program/script may contain many tests and it may output multiple test lines. The script indicates how many tests it plans to run by first printing a line like 1..532 to indicate that it plans to run 532 tests. Each test gets is own PASS, FAIL, XFAIL indication. As an example: TEST_EXTENSIONS =.abc ABC_LOG_DRIVER = tap-driver.sh TESTS = test1.abc test2.abc Means that tap-driver.sh is called twice, once for test1.abc, once for test2.abc. Each of these tests can return one or more of ok, not ok, skip, bail, or a YAML-like text string defined by --diagnostic-string and --comments. Yes, but tap-driver.sh interprets the output while the test program/scripts runs and it can handle an arbitary number of tests/results in the same test script. Anyhow, thanks for the heads up, I will be looking at your files. And, as always, I am confused. I am not surprised. :-) Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: What is a 'program' in Section 15.3.3.1?
On Wed, 27 May 2015, Arthur Schwarz wrote: In looking at tap-driver.sh there doesn't appear to be a place where a 'program' is accepted on the input command line. It appears that after all options are read if the input command line '$#' is not zero then an error is declared. So, is the TAP interface different from other Custom Test Driver interfaces? The TAP interface is very different. It depends on subordinate scripts printing 'ok' or 'not ok' (plus some optional extra diagnostic info) rather than return codes. The subordinate scripts take full responsibility for the set of tests to be performed, what programs are invoked, and the options supplied to those programs. For GraphicsMagick, I created a script which is sourced by actual test scripts (.tap scripts), which provides generic functions used by those scripts. The functions support pass/fail detection based on program exit status, but also support 'xfail' by being passed a set of feature id strings which are matched which the actual available features. If the test fails, but a necessary feature is missing, then an extra diagnosis is added which Automake's TAP script will turn into an informative XFAIL message: not ok # SKIP requires foo support See the implementation at http://hg.code.sf.net/p/graphicsmagick/code/file/b84df92f710a/scripts/tap-functions.shi and http://hg.code.sf.net/p/graphicsmagick/code/file/b84df92f710a/tests/common.shi and http://hg.code.sf.net/p/graphicsmagick/code/file/b84df92f710a/tests/rwfile.tap Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: What is a 'program' in Section 15.3.3.1?
On 2015-05-27 13:55 -0700, Arthur Schwarz wrote: In looking at tap-driver.sh there doesn't appear to be a place where a 'program' is accepted on the input command line. It appears that after all options are read if the input command line '$#' is not zero then an error is declared. So, is the TAP interface different from other Custom Test Driver interfaces? I am guessing that you are referring to this line in tap-driver.sh: test $# -gt 0 || usage_error missing test command The error is only printed if $# is *equal* to zero (i.e., additional arguments including the program name are mandatory). Regards, -- Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
