Dear all,
I've put together a very small ES5 javascript engine (<10K sloc of c89, <160kb,
no external dependencies) and I'm using test262 as its test suite -- the little
engine currently passes all (for those interested, the source is at
http://www.sqliteconcepts.org/cgi-bin/jslite; log in as anonymous - when not
logged in the repo appears empty).
Recently I've started using the SpiderMonkey test suite for additional coverage
and Jeff Walden has asked me to post some user observations to this list.
In my understanding the two suites (test262 and SM test) are about the same
size, with about 30K individual tests in both. So far my observation is that
the two suites have a different perspective. test262 is about "specification
coverage", as you would expect considering its history. The SM suite is about
"code coverage" and regressions, again as you would expect considering its
history. When applied to a new implementation, the two tend to find different
bugs (although there obviously is a lot of overlap).
Other observations so far:
- The SM test suite takes a bit of fiddling to use with another engine than SM:
there seems to be a link to XUL ('xml-info') and in the end I resorted to
hacking jstests.py in order to get it to run.
- I am confused how to target a specific part of the SM test suite. How do I
test for ES5 conformance? My basic command is "jstests.py ../js --no-extensions
- t 5", this excludes SM specific extensions and long running tests. But what
next? I could add a 'ecma_5' specifier, but that would exclude many relevant
tests that are grouped under ecma_3, etc. How do I target all tests for
regexp's? Will adding a "RegExp" specifier find all? What if it is more
detailed, e.g. targeting all tests for 'substr()'?
- The test numbering system used in test262 is easier for testing specific
parts of the spec; however, there seems to be broad consensus on the test262
mailing list that its current numbering system is painful and needs to change
-- I believe the main issue is that the numbering system is tightly tied to the
spec numbering of sections and abstract algorithm process steps and that new
major editions of the spec require renumbering of the test suite. I understand
the pain, but I think it is very much worth it in usability.
- For example, running "jstests.py ../js --no-extensions -t 5 RegExp" with my
nightly currently fails 25 of the test files (a 25% failure rate at the file
level, at the individual test level it is much better), but many fails seem
spurious:
ecma_2/RegExp/function-001.js
ecma_2/RegExp/constructor-001.js
ecma_2/RegExp/properties-001.js
ecma_3/RegExp/15.10.4.1-6.js
ecma_3/RegExp/15.10.4.1-5-n.js
ecma_3/RegExp/octal-002.js
ecma_3/RegExp/regress-375711.js
ecma_3/RegExp/regress-375715-01-n.js
ecma_3/RegExp/regress-375715-02.js
ecma_3/RegExp/regress-375715-04.js
ecma_5/RegExp/exec-lastIndex-ToInteger.js
ecma_5/RegExp/exec.js
ecma_5/RegExp/regexp-space-character-class.js
js1_2/regexp/RegExp_lastMatch.js
js1_2/regexp/RegExp_dollar_number.js
js1_2/regexp/RegExp_lastMatch_as_array.js
js1_2/regexp/RegExp_lastParen_as_array.js
js1_2/regexp/RegExp_lastParen.js
js1_2/regexp/RegExp_leftContext.js
js1_2/regexp/RegExp_leftContext_as_array.js
js1_2/regexp/RegExp_multiline.js
js1_2/regexp/RegExp_multiline_as_array.js
js1_2/regexp/RegExp_rightContext.js
js1_2/regexp/RegExp_rightContext_as_array.js
ecma_3/RegExp/regress-209067.js
The 'ecma_x' tests appear (mostly) relevant to ES5 testing, the 'js1_2' tests
appear obsolete for ES5 testing. Unless I misunderstand the test runner, test
file "ecma_3/RegExp/15.10.4.1-5-n.js" will always fail, even on a compliant ES3
engine.
- My current understanding is that for testing SM a "baselining" approach is
used. First the test suite is run against a known good compile and all failed
test are logged to a file. The new code is then tested excluding all tests that
also failed for the "known good" version. As far as I know there is no standard
"baseline file" for ES3, ES5 or ES6.
- Baselining seems wasteful. Take "ecma_2/RegExp/function-001.js": I fail this
test because the "source" property is tested to be "", and in ES5 it should be
"(?:)". However, baselining the entire file also discards all the good tests in
that file, and on average there are 12 test cases per file.
- Perhaps these issues are related to meta-data or lack thereof. If there is a
standard header with information about the test, it has escaped me. Tests seem
to vary from Netscape tests with long descriptive preambles (not machine
parseable), to tests with short descriptions, to tests with no header at all. I
think I've come across tests licensed under the MPL, under the GPL and public
domain tests. In this respect, the test262 suite seems rather better organized.
Last but not least, I'm making these comments in an attempt to be helpful, and
not to trash the SM test suite or to argue that test262 is somehow better. I am
immensely grateful that both these test suites exist as open source at all and
many thanks to all the folks involved!
Regards,
Paul
_______________________________________________
dev-tech-js-engine-internals mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals
