[whatwg] HTML5 Tokenizer Test Cases and Correct Output

2013-03-28 Thread Mohammad Al Houssami (Alumni) 
Hello everyone.

I was wondering if there is some sort of tests for the Tokenizer along with the 
correct output of tokens as well as a way of representing tokens.
What I have in mind is running the tokenizer on some HTML input and printing 
the tokens in the same way the correct output is written.
I will  then be comparing the result I have with the correct one provided 
character by character. :)

Thanks in advance.
Mohammad

Re: [whatwg] HTML5 Tokenizer Test Cases and Correct Output

2013-03-28 Thread James Graham 

On 03/28/2013 12:06 PM, Mohammad Al Houssami (Alumni) wrote:

Hello everyone.

I was wondering if there is some sort of tests for the Tokenizer along with the 
correct output of tokens as well as a way of representing tokens.
What I have in mind is running the tokenizer on some HTML input and printing 
the tokens in the same way the correct output is written.
I will  then be comparing the result I have with the correct one provided 
character by character. :)


http://code.google.com/p/html5lib/source/browse/#hg%2Ftestdata%2Ftokenizer

http://wiki.whatwg.org/wiki/Parser_tests has some documentation of the 
format.