Re: [PATCH 5/5] {test-protocols} tap: add experimental TAP-aware driver

2011-07-18 Thread Stefano Lattarini 
On Friday 15 July 2011, Stefano Lattarini wrote:
 * doc/automake.texi (Using the TAP test protocol): New section.
 (Overview of Custom Test Drivers Support): Minor updates.
 * lib/tap-driver: New script, TAP-aware test driver for Automake;
 implemented in perl and based on TAP::Parser.
 * lib/Makefile.am (dist_script_DATA): Add it.
 * tests/tap-autonumber.test: New test.
 * tests/tap-bailout.test: Likewise.
 * tests/tap-basic.test: Likewise.
 * tests/tap-deps.test: Likewise.
 * tests/tap-diagnostic.test: Likewise.
 * tests/tap-empty.test: Likewise.
 * tests/tap-empty-diagnostic.test: Likewise.
 * tests/tap-escape-directive.test: Likewise.
 * tests/tap-exit.test: Likewise.
 * tests/tap-fancy.test: Likewise.
 * tests/tap-fancy2.test: Likewise.
 * tests/tap-global-result.test: Likewise.
 * tests/tap-html.test: Likewise.
 * tests/tap-log.test: Likewise.
 * tests/tap-merge-stdout-stderr.test: Likewise.
 * tests/tap-more.test: Likewise.
 * tests/tap-more2.test: Likewise.
 * tests/tap-no-merge-stdout-stderr.test: Likewise.
 * tests/tap-no-spurious-summary.test: Likewise.
 * tests/tap-no-spurious.test: Likewise.
 * tests/tap-not-ok-skip.test: Likewise.
 * tests/tap-numeric-description.test: Likewise.
 * tests/tap-out-of-order.test: Likewise.
 * tests/tap-passthrough.test: Likewise.
 * tests/tap-plan.test: Likewise.
 * tests/tap-plan-errors.test: Likewise.
 * tests/tap-plan-corner.test: Likewise.
 * tests/tap-realtime.test: Likewise.
 * tests/tap-recheck-logs.test: Likewise.
 * tests/tap-recheck.test: Likewise.
 * tests/tap-skip-whole.test: Likewise.
 * tests/tap-summary.test: Likewise.
 * tests/tap-todo-skip.test: Likewise.
 * tests/tap-todo-skip-together.test: Likewise.
 * tests/tap-todo-skip-whitespace.test: Likewise.
 * tests/tap-skipall-whitespace.test: Likewise.
 * tests/tap-unplanned.test: Likewise.
 * tests/tap-whitespace-normalization.test: Likewise.
 * tests/tap-with-and-without-number.test: Likewise.
 * tests/tap-xfail-tests.test: Likewise.
 * tests/tap-bad-prog.test: New xfailing test.
 * tests/tap-color.test: Likewise.
 * tests/tap-plan-corner2.test: Likewise.
 * tests/tap-message-0.test: Likewise.
 * tests/tap-signal.test: Likewise.
 * tests/Makefile.am (TESTS, XFAIL_TESTS): Update.
 ---

And consider this squashed in:

-*-*-*-

diff --git a/tests/tap-color.test b/tests/tap-color.test
index 8ed7265..82c0b51 100755
--- a/tests/tap-color.test
+++ b/tests/tap-color.test
@@ -52,7 +52,8 @@ AUTOMAKE_OPTIONS = color-tests
 AM_TEST_LOG_DRIVER_FLAGS = --comments
 TEST_LOG_COMPILER = cat
 TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
-TESTS = all.test skip.test bailout.test plan-errors.test
+TESTS = all.test skip.test bail.test badplan.test noplan.test \
+few.test many.test order.test
 END
 
 cat  all.test  'END'
@@ -66,15 +67,41 @@ ok 5 - zardoz # TODO
 END
 
 cat  skip.test  'END'
-1..0 # Whole script not run
+1..0 # SKIP whole script
 END
 
-cat  errors.test  'END'
+cat  bail.test  'END'
 1..1
 ok 1
 Bail out!
 END
 
+cat  badplan.test  'END'
+# foo
+1..1
+ok 1
+END
+
+cat  noplan.test  'END'
+ok 1
+END
+
+cat  few.test  'END'
+1..2
+ok 1
+END
+
+cat  many.test  'END'
+1..1
+ok 1
+ok 2
+END
+
+cat  order.test  'END'
+1..1
+ok 5
+END
+
 $ACLOCAL
 $AUTOCONF
 $AUTOMAKE
@@ -92,9 +119,15 @@ test_color ()
   cat stdout | grep ^${blu}SKIP${std}: all\.test 3 - baz # SKIP sk$
   cat stdout | grep ^${red}FAIL${std}: all\.test 4 - quux$
   cat stdout | grep ^${red}XPASS${std}: all\.test 5 - zardoz # TODO$
-  cat stdout | grep ^${blu}SKIP${std}: skip\.test # Whole script not run$
-  cat stdout | grep ^${grn}PASS${std}: bailout\.test 1$
-  cat stdout | grep ^${mgn}ERROR${std}: all\.test - Bail out!$
+  cat stdout | grep ^${blu}SKIP${std}: skip\.test - whole script$
+  cat stdout | grep ^${grn}PASS${std}: bail\.test 1$
+  cat stdout | grep ^${mgn}ERROR${std}: bail\.test - Bail out!$
+  cat stdout | grep ^${mgn}ERROR${std}: badplan\.test - test plan in middle 
of output$
+  cat stdout | grep ^${mgn}ERROR${std}: noplan\.test - missing test plan$
+  cat stdout | grep ^${mgn}ERROR${std}: few.test - too few tests run 
(expected 2, got 1)$
+  cat stdout | grep ^${mgn}ERROR${std}: many.test - too many tests run 
(expected 1, got 2)$
+  cat stdout | grep ^${mgn}ERROR${std}: many.test 2 # UNPLANNED$
+  cat stdout | grep ^${mgn}ERROR${std}: order.test 5 # OUT-OF-ORDER 
(expecting 1)$
   # Diagnostic messages shouldn't be colorized.
   cat stdout | grep ^# all\.test: Hi! I shouldn't be colorized!$
   :

-*-*-*-

Sorry for the noise,
  Stefano

Re: [PATCH 5/5] {test-protocols} tap: add experimental TAP-aware driver

2011-07-18 Thread Ralf Wildenhues 
* Stefano Lattarini wrote on Mon, Jul 18, 2011 at 10:17:01AM CEST:
 On Friday 15 July 2011, Stefano Lattarini wrote:
  * doc/automake.texi (Using the TAP test protocol): New section.
  (Overview of Custom Test Drivers Support): Minor updates.
  * lib/tap-driver: New script, TAP-aware test driver for Automake;
  implemented in perl and based on TAP::Parser.
  * lib/Makefile.am (dist_script_DATA): Add it.
[...]

 And consider this squashed in:

The squash-in is OK (but I'm not done with the original patch yet).

Thanks,
Ralf

 --- a/tests/tap-color.test
 +++ b/tests/tap-color.test
 @@ -52,7 +52,8 @@ AUTOMAKE_OPTIONS = color-tests
  AM_TEST_LOG_DRIVER_FLAGS = --comments
  TEST_LOG_COMPILER = cat
  TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
 -TESTS = all.test skip.test bailout.test plan-errors.test
 +TESTS = all.test skip.test bail.test badplan.test noplan.test \
 +few.test many.test order.test
  END
  
  cat  all.test  'END'
 @@ -66,15 +67,41 @@ ok 5 - zardoz # TODO
  END
  
  cat  skip.test  'END'
 -1..0 # Whole script not run
 +1..0 # SKIP whole script
  END
  
 -cat  errors.test  'END'
 +cat  bail.test  'END'
  1..1
  ok 1
  Bail out!
  END
  
 +cat  badplan.test  'END'
 +# foo
 +1..1
 +ok 1
 +END
 +
 +cat  noplan.test  'END'
 +ok 1
 +END
 +
 +cat  few.test  'END'
 +1..2
 +ok 1
 +END
 +
 +cat  many.test  'END'
 +1..1
 +ok 1
 +ok 2
 +END
 +
 +cat  order.test  'END'
 +1..1
 +ok 5
 +END
 +
  $ACLOCAL
  $AUTOCONF
  $AUTOMAKE
 @@ -92,9 +119,15 @@ test_color ()
cat stdout | grep ^${blu}SKIP${std}: all\.test 3 - baz # SKIP sk$
cat stdout | grep ^${red}FAIL${std}: all\.test 4 - quux$
cat stdout | grep ^${red}XPASS${std}: all\.test 5 - zardoz # TODO$
 -  cat stdout | grep ^${blu}SKIP${std}: skip\.test # Whole script not run$
 -  cat stdout | grep ^${grn}PASS${std}: bailout\.test 1$
 -  cat stdout | grep ^${mgn}ERROR${std}: all\.test - Bail out!$
 +  cat stdout | grep ^${blu}SKIP${std}: skip\.test - whole script$
 +  cat stdout | grep ^${grn}PASS${std}: bail\.test 1$
 +  cat stdout | grep ^${mgn}ERROR${std}: bail\.test - Bail out!$
 +  cat stdout | grep ^${mgn}ERROR${std}: badplan\.test - test plan in middle 
 of output$
 +  cat stdout | grep ^${mgn}ERROR${std}: noplan\.test - missing test plan$
 +  cat stdout | grep ^${mgn}ERROR${std}: few.test - too few tests run 
 (expected 2, got 1)$
 +  cat stdout | grep ^${mgn}ERROR${std}: many.test - too many tests run 
 (expected 1, got 2)$
 +  cat stdout | grep ^${mgn}ERROR${std}: many.test 2 # UNPLANNED$
 +  cat stdout | grep ^${mgn}ERROR${std}: order.test 5 # OUT-OF-ORDER 
 (expecting 1)$
# Diagnostic messages shouldn't be colorized.
cat stdout | grep ^# all\.test: Hi! I shouldn't be colorized!$