msopacua Wed Oct 23 16:56:09 2002 EDT
Modified files:
/php4 README.TESTING
Log:
Some english fixes
Added docs for automated testing.
Now telling the current version of the truth about ini settings :)
Be more verbose about QA goals intentions and the tester's part in the
process.
Index: php4/README.TESTING
diff -u php4/README.TESTING:1.7 php4/README.TESTING:1.8
--- php4/README.TESTING:1.7 Wed Oct 23 08:59:18 2002
+++ php4/README.TESTING Wed Oct 23 16:56:09 2002
@@ -14,10 +14,11 @@
make test
"make test" basically executes "run-tests.php" script
-under source root. Therefore you can execute the script
-as follows
+under the source root (parallel builds will not work). Therefore you
+can execute the script as follows:
-./sapi/cli/php -c /path/to/php.ini/ run-tests.php [ext/some_extension_name]
+TEST_PHP_EXECUTABLE=sapi/cli/php \
+sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
[Which "php" executable "make test" look for]
@@ -37,38 +38,39 @@
[Which php.ini is used]
-----------------------
- "make test" force to use php.ini-dist as default config file. If
-you would like to test with other configuration file, user
-"run-tests.php" script.
-
-Example:
-./sapi/cli/php -c /path/to/php.ini/ run-tests.php ext/standard
-
-If you use php.ini other than php.ini-dist, you may see more failed
-tests.
+ "make test" uses the same php.ini file as it would once installed.
+The tests have been written to be independant of that php.ini file,
+so if you find a test that is affected by a setting, please report
+this, so we can address the issue.
[Which test scripts are executed]
---------------------------------
- "run-tests.php" ("make test") executes all test scripts by default
-by looking all directory named "tests". If there are files have "phpt"
-extension, "run-tests.php" takes test php code from the file and
-executes it.
+ "run-tests.php" ("make test"), without any arguments executes all
+test scripts by looking extracting all directories named "tests"
+from the source root and any subdirectories below. If there are files,
+which have a "phpt" extension, "run-tests.php" looks at the sections
+in these files, determines whether it should run it, by evaluating
+the 'SKIP' section. If the test is elligable for execution, the 'FILE'
+section is extracted into a ".php" file, with a unique name and
+executed.
+When an argument is given or TESTS environment variable is set, the
+GLOB is expanded by the shell and any file with extension "*.phpt" is
+regarded as a testfile.
Tester can easily execute tests selectively with as follows.
Examples:
-./sapi/cli/php -c /path/to/php.ini/ run-tests.php ext/mbstring
-./sapi/cli/php -c /path/to/php.ini/ run-tests.php ext/mbstring/020.phpt
+./sapi/cli/php run-tests.php ext/mbstring/*
+./sapi/cli/php run-tests.php ext/mbstring/020.phpt
[Test results]
--------------
Test results are printed to standard output. If there is a failed test,
-"run-tests.php" script saves the result, expected result and code
-executed to the test script directory. For example, if
-ext/myext/tests/myext.phpt is failed to pass, following files are
-created:
+the "run-tests.php" script saves the result, the expected result and the
+code executed to the test script directory. For example, if
+ext/myext/tests/myext.phpt fails to pass, the following files are created:
ext/myext/tests/myext.php - actual testfile executed
ext/myext/tests/myext.log - log of test execution (L)
@@ -76,7 +78,13 @@
ext/myext/tests/myext.out - output from test script (O)
ext/myext/tests/myext.diff - diff of .out and .exp (D)
- Tester can verify these files, if failed test is actually a bug or not.
+ Failed tests are always bugs. Either the test is bugged or not considering
+factors applying to the tester's environment, or there is a bug in PHP.
+If this is a known bug, we strive to provide bugnumbers, in either the
+test name or the file name. You can check the status of such a bug, by
+going to: http://bugs.php.net/12345 where 12345 is the bugnumber.
+For clarity and automated processing, bugnumbers are prefixed by a hash
+sign '#' in testnames and/or testcases are named bug12345.phpt.
NOTE: The files generated by tests can be selected by setting the
environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
@@ -85,6 +93,21 @@
NOTE: You can set environment variable TEST_PHP_DETAILED to enable
detailed test information.
+
+[Automated testing]
+ If you like to keep up to speed, with latest developments and quality
+assurance, setting the environment variable NO_INTERACTION to 1, will not
+prompt the tester for any user input.
+
+Normally, the exit status of "make test" is zero, regardless of the results
+of independant tests. Set the environment variable REPORT_EXIT_STATUS to 1,
+to make "make test' set the exit statis ("$?") to non-zero, when a test has
+failed.
+
+NOTE: the exit status of run-tests.php will be 1 when
+REPORT_EXIT_STATUS is set. The result of "make test" may be higher
+than that. At present, gmake 3.79.1 returns 2, so it is
+advised to test for non-zero, rather then a specific value.
[Creating new test files]
--
PHP CVS Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
