[
https://issues.apache.org/jira/browse/GROOVY-12435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130108#comment-18130108
]
Paul King edited comment on GROOVY-12435 at 10/22/26, 3:33 AM:
---------------------------------------------------------------
AI read:
Confirmed on master, and it is long-standing: every antlr4 (Parrot) parser
version from 3.0 onward is affected, while the old antlr2 parser handled lone
CR line endings correctly.
h3. Reproduction
Line numbers of the AST nodes for small sources (statement columns in brackets
where relevant):
||Case||master, 5.0.6, 4.0.32, 3.0.25 (antlr4)||3.0.25 (antlr2)||Expected||
|attachment shape (lone CR after {{package foo}}, CRLF elsewhere): import,
statement|2, 6|3, 7|3, 7|
|all-CR file, 3 statements|1, 1, 1 (columns 1, 11, 22)|1, 2, 4|1, 2, 4|
|statement after a {{/* */}} comment containing lone CRs|3|5|5|
|statement after a {{'''}} string containing a lone CR|2|3|3|
|stack trace line of a {{throw}} on line 3|1|3|3|
|syntax error on line 3, column 9|1:29|3:9|3:9|
So it is not only the editor that is affected: columns are wrong too, and
runtime stack traces and compile errors point at the wrong line.
h3. Cause
* The grammar is fine: {{LineTerminator : '\r'? '\n' | '\r'}} so a lone CR
lexes as an {{NL}} token.
* Line counting is done by the antlr4 runtime ({{me.sunlan:antlr4-runtime}}
4.13.2.17), where {{LexerATNSimulator.consume(input, curChar)}} increments the
line only on {{'\n'}} (the upstream behaviour linked above).
* In the fork that method is {{final}} and package-private and is called
directly from the DFA loop, so overriding the public {{consume(input)}} in
Groovy's {{PositionAdjustingLexerATNSimulator}} (in {{GroovyLangLexer}}) would
not take effect.
* Secondary: {{PositionConfigureUtils.endPosition}} computes a node's last line
by counting only {{'\n'}} in the stop token's text, so end positions after
multi-line tokens containing lone CRs would stay wrong even with a lexer fix.
* The rest of Groovy already counts lone CR as a line end:
{{AbstractReaderSource.getLine}} uses {{BufferedReader.readLine()}}, so today
the source excerpt in an error message can show a different line from the
reported line number.
h3. Fix options
# *Fork fix (cleanest):* in {{consume(input, curChar)}} also treat {{'\r'}} as
a line end when {{input.LA(2) != '\n'}}. The extra look-ahead only happens on
CR characters. Needs a new fork release, a version bump, and regenerated
{{verification-metadata.xml}}. It deliberately differs from upstream ANTLR.
# *Groovy-only:* override {{match(CharStream, int)}} in
{{PositionAdjustingLexerATNSimulator}} so that after {{super.match}} it rescans
the token just read for lone CRs and corrects {{line}} /
{{charPositionInLine}}. Since {{match}} runs for every token, including skipped
whitespace and comments, token start positions come out right. A per-file check
for any lone CR keeps the normal path free of extra work. No dependency change,
so it can be backported to 4.x and 5.x unchanged.
Either way {{PositionConfigureUtils}} needs the same treatment (count CRLF,
lone CR and lone LF each as one line end). Regression tests can cover AST
line/column, a stack-trace line, and a syntax-error position for lone-CR, mixed
and all-CR sources.
Suggestion: the Groovy-only fix for 6.0.x and the backports, and the fork fix
as an option for the fork maintainer, after which the Groovy-side workaround
can be removed. Not a release blocker (lone-CR endings are uncommon: classic
Mac files, mixed-ending files), but it affects tooling and stack traces.
was (Author: paulk):
AI read:
GROOVY-12435 is confirmed. It has been there since the antlr4 ("Parrot") parser
arrived in Groovy 3.0, and it's a regression compared with the antlr2 parser.
Wherever a line ends in a lone `\r`, line and column numbers after it are
wrong, and stack traces and compile errors point at the wrong place.
## Reproduction
I rebuilt the attachment from its byte count rather than reading it directly,
because the sandbox wouldn't let me read the saved copy. The key byte is a lone
`\r` right after `package foo`; the rest uses CRLF.
| Case | master, 5.0.6, 4.0.32, 3.0.25
(antlr4) | 3.0.25 antlr2 | Correct |
| -------------------------------------------- |
-------------------------------------- | ------------- | ------- |
| Issue shape: import, statement | lines 2, 6
| 3, 7 | 3, 7 |
| All-CR file, 3 statements | lines 1, 1, 1; columns 1, 11,
22 | 1, 2, 4 | 1, 2, 4 |
| CRs inside a `/* */` comment, next statement | line 3
| 5 | 5 |
| CR inside a `'''` string, next statement | line 2
| 3 | 3 |
| Runtime stack trace for a throw on line 3 | line 1
| 3 | 3 |
| Syntax error on line 3, column 9 | line 1, column 29
| 3:9 | 3:9 |
Master gives exactly the reporter's numbers (2 and 6).
## Cause
- The grammar is fine: `LineTerminator : '\r'? '\n' | '\r'`, so a lone `\r`
becomes a normal newline token.
- Line counting happens in the ANTLR runtime, which Groovy uses as the
`me.sunlan:antlr4-runtime` 4.13.2.17 fork. Its
`LexerATNSimulator.consume(input, curChar)` only does `line++` on `'\n'`.
That's the upstream ANTLR behaviour Eric's comment links to.
- In the fork, that method is `final` and package-private, and the main DFA
loop calls it directly. So overriding the public `consume(input)` in Groovy's
existing `PositionAdjustingLexerATNSimulator` (in `GroovyLangLexer.java`)
wouldn't help.
- A second, smaller bug: `PositionConfigureUtils.java:65` works out a node's
end line by counting only `'\n'` in the last token. A multi-line string with
lone CRs would still get a wrong end line even after the lexer is fixed.
- The rest of Groovy disagrees with the parser. `AbstractReaderSource.getLine`
uses `BufferedReader.readLine()`, which does treat a lone `\r` as a line end.
So today an error message's source excerpt shows a different line from the line
number it reports.
## Fix options
**A. Fix the fork (cleanest).** In `consume(input, curChar)`, also count `'\r'`
as a new line when `input.LA(2) != '\n'`. The extra look-ahead only happens on
`\r` characters, so it costs essentially nothing. It needs a new fork release
from Daniel, a version bump in `versions.properties`, and regenerated
`verification-metadata.xml`. It deliberately departs from upstream ANTLR, which
the fork already does for performance.
**B. Groovy-only.** Override `match(CharStream, int mode)` in
`PositionAdjustingLexerATNSimulator`. After `super.match`, rescan the token
just read for lone `\r` and correct `line` and `charPositionInLine`.
- `match` runs for every token, including skipped whitespace and comments, so
start positions come out right.
- To keep the lexer's hot path untouched, check once per file whether it
contains a lone `\r`, and skip the rescan entirely when it doesn't.
- No dependency release is needed, and it can be backported to 4.x and 5.x
unchanged.
Either way, `PositionConfigureUtils` needs the matching change: count `\r\n`, a
lone `\r`, and a lone `\n` each as one line end. Tests can follow the table:
AST line and column numbers, a stack-trace line, and a syntax-error position,
for lone-CR, mixed and all-CR files.
**Recommendation:** use B for 6.0.x and the backports, since it fixes this with
no dependency change. Ask Daniel whether he'd take A in the fork; if he does, B
can be removed later. This isn't a release blocker: lone-CR endings are rare
(old Mac files and mixed-ending files from editors). But it's a real tooling
problem, as the reporter found in his editor, and the stack-trace line numbers
are wrong too.
> carriage-return only line ending does not increment line number
> ---------------------------------------------------------------
>
> Key: GROOVY-12435
> URL: https://issues.apache.org/jira/browse/GROOVY-12435
> Project: Groovy
> Issue Type: Bug
> Components: parser-antlr4
> Affects Versions: 3.0.25, 4.0.33, 5.1.3, 6.0.0
> Reporter: Eric Milles
> Priority: Major
> Attachments: LineEndings.groovy, image-2026-09-30-09-41-06-973.png
>
>
> [^LineEndings.groovy]I had a source file with mixed line endings that caused
> a lot of trouble with the editor. I traced this back to the parser's handling
> of line endings. It seems when a line ending is just a carriage-return (\r)
> the line number is not incremented.
> !image-2026-09-30-09-41-06-973.png!
> In this example, the import and script statement have lines 2 and 6
> respectively. They should be at lines 3 and 7 as can be seen by the line
> numbering in the editor's gutter. The antlr2 parser handled this case
> properly.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
