Re: Stopping unit test on failed partial ordering dependency
On Tue, 23 Apr 2019, Kip Warner wrote: Now suppose the TestStartDaemon.sh bails for some reason. There is no point on doing all the queries and having them all fail. There has to be a sane way of handling this scenario. Having all the dependent tests fail seems ok, as long as the failure is expedient. Unless the tests are reasonably ordered, it may not be obvious to the user why so many tests are failing. If there are normally 1200 tests run but the user sees only two tests run, and one of them fails, then the user might not be aware of the many tests which have not been executed. The tests which fail still need to be reported as failed. One reason why I like the TAP testing framework is that a test script has execution context across multiple tests (avoiding repeated stop/start), which improves control over the test, and executes more efficiently. You can start your server at the beginning of the test script and then execute 500 tests on it via the same script, while terminating the server at the end of the script. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt
Re: Stopping unit test on failed partial ordering dependency
On Tue, 2019-04-23 at 10:25 -0500, Bob Friesenhahn wrote: > On Mon, 22 Apr 2019, Kip Warner wrote: > > How can I solve this problem? > > By using the TAP test framework you could organize your related > tests into ordered scripts which assure that test > construction/destruction is orderly even if some tests fail. This > approach may lose parallelism if there are not enough independent TAP > tests to keep the system busy. TAP tests do need to produce the > expected number of test report messages, even if a prerequisite has > failed. This is effectively what I'm doing in my Makefile.am: check_SCRIPTS = \ TestDatabaseSetup.sh \ TestStartDaemon.sh \ TestStopDaemon.sh \ TestDatabaseTeardown.sh check_PROGRAM S = \ TestQueryA \ TestQueryB \ TestQueryC \ TestQueryD TESTS = $(check_SCRIPTS) $(check_PROGRAMS) # Define aliases to logs to use in partial ordering... ordering_TestDatabaseSetup = TestDatabaseSetup.log ordering_TestStartDaemon = TestStartDaemon.log ordering_TestStopDaemon = TestStopDaemon.log ordering_TestDatabaseTeardown = TestDatabaseTeardown.log ordering_all_daemon_tests = \ ordering_TestQueryA.log \ ordering_TestQueryB.log \ ordering_TestQueryC.log \ ordering_TestQueryD.log # Declare some partial ordering... $(ordering_TestStartDaemon): $(ordering_TestDatabaseSetup) $(ordering_all_daemon_tests): $(ordering_TestStartDaemon) $(ordering_TestStopDaemon): $(ordering_all_daemon_tests) $(ordering_TestDatabaseTeardown): $(ordering_TestStopDaemon) Now suppose the TestStartDaemon.sh bails for some reason. There is no point on doing all the queries and having them all fail. There has to be a sane way of handling this scenario. -- Kip Warner | Senior Software Engineer OpenPGP signed/encrypted mail preferred https://www.thevertigo.com signature.asc Description: This is a digitally signed message part
Re: Stopping unit test on failed partial ordering dependency
On Mon, 2019-04-22 at 22:33 -0600, John Calcote wrote: > Try using a stamp file only created on success. To do this you’ll > need to wrap your test calls in a script that creates the stamp file > on success. Hey John, Wrapping the script in another wrapper is possible, but I'd prefer not to do that because it overly complicates things because the test script is already a wrapper for a binary to run. PS Good to hear from you again. I always have your book by my side. -- Kip Warner | Senior Software Engineer OpenPGP signed/encrypted mail preferred https://www.thevertigo.com signature.asc Description: This is a digitally signed message part
Re: Stopping unit test on failed partial ordering dependency
On Mon, 22 Apr 2019, Kip Warner wrote: How can I solve this problem? By using the TAP test framework you could organize your related tests into ordered scripts which assure that test construction/destruction is orderly even if some tests fail. This approach may lose parallelism if there are not enough independent TAP tests to keep the system busy. TAP tests do need to produce the expected number of test report messages, even if a prerequisite has failed. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt
Re: Stopping unit test on failed partial ordering dependency
On Mon, Apr 22, 2019 at 10:12 PM Kip Warner wrote: > How can I solve this problem? Try using a stamp file only created on success. To do this you’ll need to wrap your test calls in a script that creates the stamp file on success. John >
Stopping unit test on failed partial ordering dependency
Hey list, I have a bunch of unit tests declared in check_PROGRAMS and check_SCRIPTS. Some tests can only be run after others, while some others can only be run before others. I've managed to solve this problem pursuant to § 15.2.1 via partial ordering constraints. This is done by declaring explicit make prerequisites between the generated log files. This works fine, even when using the parallel test harness. But only as long as all the tests pass. One problem I am having is if test B is dependent on test A to complete successfully before it can run, but A fails. B will run anyways only because B.log depended on A.log. A.log was generated regardless of whether test A was a pass or fail. Consider the situation where a database is setup, a daemon is spawned, various client requests are serviced in any order, the daemon is shutdown, and lastly the database is torn down. If the unit test to spawn the daemon fails for any reason, all the client requests are bound to fail anyways so there's no point on trying to debug them when the problem is obvious that the daemon isn't running. How can I solve this problem? -- Kip Warner | Senior Software Engineer OpenPGP signed/encrypted mail preferred https://www.thevertigo.com signature.asc Description: This is a digitally signed message part
