Reviewers: ihab.awad,
Description:
Issue 996: Refactor rendering pipeline to always pass tokens *and*
marks
If the original source code contains a comment inside a string,
like:
var x = '/*...@1*/';
it will get cajoled to:
var x = '';
which is not a security breach, but is incorrect. This is because
the current pipeline uses textual substitution to keep track of
file positions as the source passes through.
Got rid of textual substition.
Before the process looked like
(1) Feed tokens to SourceSpanRenderer which forwards them to a
delegate renderer along with /*...@1*/ comments wherever there is an
original file position mark.
(2) Find /*...@1*/ inside the rendered text, remove it, and build a
mapping from output characters to filepositions from the
original source.
(3) Munge that list to generate the desired debug metadata.
Now the process looks like
(1) Feed tokens to SourceSpanRenderer which forwards them to a
delegate renderer and keeps a list of all non-comment/space
tokens with marks inline.
(2) Lex the rendered output using JsLexer and match stored tokens
with rendered tokens creating a mapping from original source
file positions to rendered source positions.
(3) Turn that mapping into a mapping of output characters to
original source filepositions.
(4) Munge as before.
I also did a bit of housecleaning:
(1) The SideBySideRenderer and TabularSideBySideRenderer were
obsoleted by SourceSnippetRenderer. Deleted them.
(2) Got rid of unnecessary IOException handler to SourceSpanRenderer
since it always renders to a StringBuilder.
(3) Got rid of unnecessary output filename parameter since it seems
to be unused except in test code.
(4) Got rid of now unused breakAfterComment parameter to
JsPrettyPrinter and friends.
Since we no longer insert comment tokens (which were subsequently
removed), the rendered output matches exactly what an unwrapped pretty
printer would do.
This invalidated the existing unit tests, so please pay special
attention to the test goldens.
Please review this at http://codereview.appspot.com/32099
Affected files:
M src/com/google/caja/parser/js/CajoledModule.java
M src/com/google/caja/render/Indenter.java
M src/com/google/caja/render/JsPrettyPrinter.java
D src/com/google/caja/render/SideBySideRenderer.java
M src/com/google/caja/render/SourceSpansRenderer.java
D src/com/google/caja/render/TabularSideBySideRenderer.java
M tests/com/google/caja/AllTests.java
M tests/com/google/caja/parser/quasiliteral/testModule.co.js
M tests/com/google/caja/render/OrigSourceRendererTestCase.java
D tests/com/google/caja/render/SideBySideRendererTest.java
M tests/com/google/caja/render/SourceSnippetRendererTest.java
M tests/com/google/caja/render/SourceSpansRendererTest.java
M tests/com/google/caja/render/ssp-golden.txt
M tests/com/google/caja/render/ssp-rewritten-tokens.txt
