[
https://issues.apache.org/jira/browse/GROOVY-12435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130108#comment-18130108
]
Paul King edited comment on GROOVY-12435 at 10/22/26, 3:37 AM:
---------------------------------------------------------------
AI read:
Confirmed on master, and it is long-standing: every antlr4 (Parrot) parser
version from 3.0 onward is affected, while the old antlr2 parser handled lone
CR line endings correctly.
h3. Reproduction
Line numbers of the AST nodes for small sources (statement columns in brackets
where relevant):
||Case||master, 5.0.6, 4.0.32, 3.0.25 (antlr4)||3.0.25 (antlr2)||Expected||
|attachment shape (lone CR after {{package foo}}, CRLF elsewhere): import,
statement|2, 6|3, 7|3, 7|
|all-CR file, 3 statements|1, 1, 1 (columns 1, 11, 22)|1, 2, 4|1, 2, 4|
|statement after a {{/* */}} comment containing lone CRs|3|5|5|
|statement after a {{'''}} string containing a lone CR|2|3|3|
|stack trace line of a {{throw}} on line 3|1|3|3|
|syntax error on line 3, column 9|1:29|3:9|3:9|
So it is not only the editor that is affected: columns are wrong too, and
runtime stack traces and compile errors point at the wrong line.
h3. Cause
* The grammar is fine: {{LineTerminator : '\r'? '\n' | '\r'}} so a lone CR
lexes as an {{NL}} token.
* Line counting is done by the antlr4 runtime ({{me.sunlan:antlr4-runtime}}
4.13.2.17), where {{LexerATNSimulator.consume(input, curChar)}} increments the
line only on {{'\n'}} (the upstream behaviour linked above).
* In the fork that method is {{final}} and package-private and is called
directly from the DFA loop, so overriding the public {{consume(input)}} in
Groovy's {{PositionAdjustingLexerATNSimulator}} (in {{GroovyLangLexer}}) would
not take effect.
* Secondary: {{PositionConfigureUtils.endPosition}} computes a node's last line
by counting only {{'\n'}} in the stop token's text, so end positions after
multi-line tokens containing lone CRs would stay wrong even with a lexer fix.
* The rest of Groovy already counts lone CR as a line end:
{{AbstractReaderSource.getLine}} uses {{BufferedReader.readLine()}}, so today
the source excerpt in an error message can show a different line from the
reported line number.
h3. Fix options
# *Fork fix (cleanest):* in {{consume(input, curChar)}} also treat {{'\r'}} as
a line end when {{input.LA(2) != '\n'}}. The extra look-ahead only happens on
CR characters. Needs a new fork release, a version bump, and regenerated
{{verification-metadata.xml}}. It deliberately differs from upstream ANTLR.
# *Groovy-only:* override {{match(CharStream, int)}} in
{{PositionAdjustingLexerATNSimulator}} so that after {{super.match}} it rescans
the token just read for lone CRs and corrects {{line}} /
{{charPositionInLine}}. Since {{match}} runs for every token, including skipped
whitespace and comments, token start positions come out right. A per-file check
for any lone CR keeps the normal path free of extra work. No dependency change,
so it can be backported to 4.x and 5.x unchanged.
Either way {{PositionConfigureUtils}} needs the same treatment (count CRLF,
lone CR and lone LF each as one line end). Regression tests can cover AST
line/column, a stack-trace line, and a syntax-error position for lone-CR, mixed
and all-CR sources.
Suggestion: the Groovy-only fix for 6.0.x and the backports, and the fork fix
as an option for the fork maintainer, after which the Groovy-side workaround
can be removed. Not a release blocker (lone-CR endings are uncommon: classic
Mac files, mixed-ending files), but it affects tooling and stack traces.
[~daniel_sun] thoughts on the fork change?
was (Author: paulk):
[~daniel_sun][~daniel_sun]AI read:
Confirmed on master, and it is long-standing: every antlr4 (Parrot) parser
version from 3.0 onward is affected, while the old antlr2 parser handled lone
CR line endings correctly.
h3. Reproduction
Line numbers of the AST nodes for small sources (statement columns in brackets
where relevant):
||Case||master, 5.0.6, 4.0.32, 3.0.25 (antlr4)||3.0.25 (antlr2)||Expected||
|attachment shape (lone CR after {{package foo}}, CRLF elsewhere): import,
statement|2, 6|3, 7|3, 7|
|all-CR file, 3 statements|1, 1, 1 (columns 1, 11, 22)|1, 2, 4|1, 2, 4|
|statement after a {{/* */}} comment containing lone CRs|3|5|5|
|statement after a {{'''}} string containing a lone CR|2|3|3|
|stack trace line of a {{throw}} on line 3|1|3|3|
|syntax error on line 3, column 9|1:29|3:9|3:9|
So it is not only the editor that is affected: columns are wrong too, and
runtime stack traces and compile errors point at the wrong line.
h3. Cause
* The grammar is fine: {{LineTerminator : '\r'? '\n' | '\r'}} so a lone CR
lexes as an {{NL}} token.
* Line counting is done by the antlr4 runtime ({{me.sunlan:antlr4-runtime}}
4.13.2.17), where {{LexerATNSimulator.consume(input, curChar)}} increments the
line only on {{'\n'}} (the upstream behaviour linked above).
* In the fork that method is {{final}} and package-private and is called
directly from the DFA loop, so overriding the public {{consume(input)}} in
Groovy's {{PositionAdjustingLexerATNSimulator}} (in {{GroovyLangLexer}}) would
not take effect.
* Secondary: {{PositionConfigureUtils.endPosition}} computes a node's last line
by counting only {{'\n'}} in the stop token's text, so end positions after
multi-line tokens containing lone CRs would stay wrong even with a lexer fix.
* The rest of Groovy already counts lone CR as a line end:
{{AbstractReaderSource.getLine}} uses {{BufferedReader.readLine()}}, so today
the source excerpt in an error message can show a different line from the
reported line number.
h3. Fix options
# *Fork fix (cleanest):* in {{consume(input, curChar)}} also treat {{'\r'}} as
a line end when {{input.LA(2) != '\n'}}. The extra look-ahead only happens on
CR characters. Needs a new fork release, a version bump, and regenerated
{{verification-metadata.xml}}. It deliberately differs from upstream ANTLR.
# *Groovy-only:* override {{match(CharStream, int)}} in
{{PositionAdjustingLexerATNSimulator}} so that after {{super.match}} it rescans
the token just read for lone CRs and corrects {{line}} /
{{charPositionInLine}}. Since {{match}} runs for every token, including skipped
whitespace and comments, token start positions come out right. A per-file check
for any lone CR keeps the normal path free of extra work. No dependency change,
so it can be backported to 4.x and 5.x unchanged.
Either way {{PositionConfigureUtils}} needs the same treatment (count CRLF,
lone CR and lone LF each as one line end). Regression tests can cover AST
line/column, a stack-trace line, and a syntax-error position for lone-CR, mixed
and all-CR sources.
Suggestion: the Groovy-only fix for 6.0.x and the backports, and the fork fix
as an option for the fork maintainer, after which the Groovy-side workaround
can be removed. Not a release blocker (lone-CR endings are uncommon: classic
Mac files, mixed-ending files), but it affects tooling and stack traces.
@daniel_sun thoughts on the fork change?
> carriage-return only line ending does not increment line number
> ---------------------------------------------------------------
>
> Key: GROOVY-12435
> URL: https://issues.apache.org/jira/browse/GROOVY-12435
> Project: Groovy
> Issue Type: Bug
> Components: parser-antlr4
> Affects Versions: 3.0.25, 4.0.33, 5.1.3, 6.0.0
> Reporter: Eric Milles
> Priority: Major
> Attachments: LineEndings.groovy, image-2026-09-30-09-41-06-973.png
>
>
> [^LineEndings.groovy]I had a source file with mixed line endings that caused
> a lot of trouble with the editor. I traced this back to the parser's handling
> of line endings. It seems when a line ending is just a carriage-return (\r)
> the line number is not incremented.
> !image-2026-09-30-09-41-06-973.png!
> In this example, the import and script statement have lines 2 and 6
> respectively. They should be at lines 3 and 7 as can be seen by the line
> numbering in the editor's gutter. The antlr2 parser handled this case
> properly.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
