Matt Thanks for your feedback. I'm assuming this is the latest cut of the LSB-FHS testsuite. Some of the failure output from the test suite would be helpful for the ones categorized as test bugs so we can analyze them further.
A general comment on the results below -- many of these below are type C assertions (optional) and return FIP result codes. Many if not all of these would resolve to PASS by a manual declaration that the subsystem/file is not supported on the platform under test. (i attach a note at the end of this mail on understanding the result codes). Some background on the process and methodology used for the certification test suites is at: http://www.opengroup.org/testing/lsb-fhs/process.html Comments below on individual tests: On Aug 15, 1:34pm in "LSB.fhs failure dige", Matt Taggart wrote: > In the comparison of LSB 1.0 and Debian described at, > http://people.debian.org/~taggart/lsb/ > I defined a Debian LSB 1.0 Sample Implementation and ran the LSB test suite > against it. This message is digest of the failures for the test suite listed. > > > LSB.fhs > ------- > /tset/LSB.fhs/root/etc/etc-tc 7 > "/etc/gettydefs: file not found" > This is only provided by the Debian "gettyps" package which is non-free. > Category: no change required This is a type C assertion and gives a PASS or FIP result, i.e. if you do not support the subsystem you can manually resolve this to pass. > /tset/LSB.fhs/root/etc/etc-tc 29 > "/etc/hosts.equiv: file not found" > Provided by the Debian "rsh-server" server. rsh is not required by the LSB. > ditto > /tset/LSB.fhs/root/etc/etc-tc 30 > "/etc/hosts.lpd: file not found" > Not in Debian > ditto > /tset/LSB.fhs/root/etc/etc-tc 38 > "/etc/sgml: directory not found" > ditto > /tset/LSB.fhs/root/etc-x11/etc-x11-tc 2 > "/etc/X11/XF86Config" > Needs to be done manually? > ditto > /tset/LSB.fhs/root/etc-x11/etc-x11-tc 3 > "/etc/X11/Xmodmap: file not found" > In "xbase-clients" Debian package > Currently this is viewed as a required file. LSB makes higher level reqts for support of X applications than the FHS > /tset/LSB.fhs/root/sbin/sbin-tc 17 > "/sbin/fastboot: file does not exist or is not executable" > Not in Debian (es and ja manpages do exist though) > Type C assertion , comments as per the ditto ones above > /tset/LSB.fhs/root/sbin/sbin-tc 18 > "/sbin/fasthalt: file does not exist or is not executable" > Not in Debian (es and ja manpages do exist though) > Type C assertion , comments as per the ditto ones above > /tset/LSB.fhs/usr/bin/bin-tc 4 > "/usr/bin/tclsh: No such file or directory" > Provided by tcl8.* (tcl8.3 is the latest currently) > Is tcl required by the LSB? > Type C assertion, coments as above.... > /tset/LSB.fhs/usr/bin/bin-tc 5 > "/usr/bin/mh: directory not found" > Provided by Debian "mh" or "nmh" packages. > Is mh required by the LSB? > Type C assertion, coments as above.... > /tset/LSB.fhs/usr/bin/bin-tc 6 > "/usr/bin/wish: does not exist or is not executable" > Provided by tk8.* (tk8.3 is the latest currently) > Is tk required by the LSB? > Type C assertion, coments as above.... > /tset/LSB.fhs/usr/bin/bin-tc 7 > "/usr/bin/expect: does not exist or is not executable" > Provided by expect* (expect5.31 is the latest currently) > Is expect required by the LSB? Type C assertion, coments as above.... > > /tset/LSB.fhs/usr/share/share-tc 3 > "/usr/share/games: directory not found" > Add to base-files Type C assertion, coments as above.... > > /tset/LSB.fhs/usr/share/share-tc 6 > "/usr/share/nls: directory not found" > Add to base-files? Type C assertion, coments as above.... > > /tset/LSB.fhs/usr/share/share-tc 8 > "/usr/share/tmac: directory not found" > Add to base-files? > Type C assertion, coments as above.... > /tset/LSB.fhs/var/cache-fonts/cache-fonts-tc 1 > "/var/cache/fonts: directory not found" > Add to base files? Nothing in Debian uses it. Type C assertion, coments as above.... > > /tset/LSB.fhs/var/games/games-tc 1 > "/var/games: directory not found" > Add to base-files > Type C assertion, coments as above.... > /tset/LSB.fhs/var/spool-lpd/spool-lpd-tc 2 > "/var/spool/lpd/lpd.lock: file not found" > Not in Debian > Type C assertion, coments as above.... > /tset/LSB.fhs/var/spool-rwho/spool-rwho-tc 1 > "/var/spool/rwho: directory not found" > Not in Debian Type C assertion, coments as above.... > > /tset/LSB.fhs/root/etc-opt/etc-opt-tc 1 > "/etc/opt: directory not found" > Add to base-files? Non compliance , this is a required directory > > /tset/LSB.fhs/root/opt/opt-tc 1 > "/opt: directory not found" > Add to base-files? > Non compliance , this is a required directory > /tset/LSB.fhs/usr/x11r6/x11r6-tc 2 > "The symbolic link /usr/bin/X11 shall exist and point to /usr/X11R6/bin" > On Debian it looks like "/usr/bin/X11 -> ../X11R6/bin" Is this acceptable? > Fix the test? Can you send the test output so we can analyze the failure > > /tset/LSB.fhs/usr/lib/lib-tc 2 > "stderr:/usr/lib/sendmail expected to be symlink to /usr/sbin/sendmail, > pointed to /usr/sbin/exim" > Test is broken Why do you think the test is broken please supply more information > > /tset/LSB.fhs/var/opt/opt-tc 1 > "/var/opt: directory not found" > Add to base-files? > Non compliance I attach some notes on how to understand result codes in the test suites UNDERSTANDING TEST RESULT CODES =============================== - Failed The test source code for failed operating system tests is located in the appropriate testset directory, in the directory hierarchy starting from tset. To analyse the results of these tests fully, you must be able to examine the test source code to understand the test strategy and identify the conditions which led to the test failure. This level of expertise requires the skills of an operating system specialist; non-specialist staff should not attempt to interpret these results. - Uninitiated or Unresolved Uninitiated means that the particular test in question did not start to execute. Unresolved means that the test started but did not reach the point where the test was able to report success or failure. When a test is reported as uninitiated or unresolved you must identify the reason why the test was not completed. These may be because of incorrect parameters, preceding failures or external events, which are described in the following paragraphs. Incorrect Parameters Most tests reported this way cannot be run because a parameter is not set correctly in the execution parameters file tetexec.cfg. The test report always identifies the tests which cannot run because of incorrect parameters. For some tests, you can correct the parameter and re-run the tests. For others, you may not be able to correct the parameter because the resources required are not available on your system. Incorrect Entries in userintf.c (Implementation Specific Routines) Tests may be reported as unresolved or uninitiated because of incorrect entries in SRC/userintf.c. If failures of user-supplied functions are reported, you will need to check this file. Preceding Failures When earlier tests have failed, some tests cannot be performed. Before you can re-run the tests, you must resolve the problem in the preceding failed tests. External Events When an external event occurs unexpectedly, tests may not be performed. Investigate the reason the test has not been run as for a failed test. - Unreported When a test is marked as unreported a major error has occurred during the testset execution. VSX tries to avoid such errors as far as possible. However, if you terminate a testset with the signal SIGTERM, tests will be unreported. Investigate the cause of the major error as for a failed test. - Warning Whenever a warning is given, the functionality is acceptable, but you should be aware that later revisions of the relevant standards or specifications may change the requirements in this area. See the appendix entitled ``TESTS GIVING WARNINGS'' for a list of the tests which may give warnings and the reasons for them. - FIP (Further Information Provided) When a test has succeeded, additional information may sometimes be given which needs to be inspected. Where information cannot be checked automatically by a test it is given for you to validate. For example, the system name and node name are given by the VSX4 uname testset. Many of the options in the FHS specification cannot be determined automatically and will return this result code. - Unsupported Unsupported means that an optional feature is not available or supported in the implementation under test. For example, in some modes the job control features are optional. VSX will recognise that they are unsupported on a particular system and report this. - Not In Use Where no macro version of an interface exists, or separate macro and function testing is not required, the macro version of the testset will report all tests as not in use. Also, some tests within a testset may not be required in a particular test mode. For example, tests for POSIX.1-1996 functionality when running in POSIX90 mode, or where test cases required in earlier versions of the FHS specification are not required in the latest version. These are not failures and require no further work. - Untested This occurs because there is no test written to check a particular feature, or an optional facility needed to perform a test is not available on the system. For example, it is not possible to check that session IDs are inherited across a fork() when job control is not available. These are generally listed on the manual pages under ``Untestable Aspects''. - Succeeded (PASS) This means the test has been executed correctly and to completion without any kind of problem. regards Andrew ----- Andrew Josey The Open Group
