morrySnow opened a new pull request, #67427:
URL: https://github.com/apache/doris/pull/67427
### What problem does this PR solve?
Issue Number: None
Related PR: #66951
Problem Summary: This draft PR is stacked on #66951. The SQL grammar
repeatedly predicted the optional `EXPLAIN` and CTE prefixes in
`queryOrDmlStatement`, `dmlStatement`, and `query`, causing long adaptive
lookahead for CTE DML. This change consumes the common prefix once, dispatches
the remaining query or DML body, and passes the prefix contexts to the FE plan
builder. It preserves non-explainable LOAD/EXPORT/REPLAY/COPY/TRUNCATE
branches, nested source CTEs, error positions, and FE command semantics. It
also adds focused parser/FE tests, a reproducible JMH harness, and the
evaluated ANTLR4 optimization roadmap.
### Benchmark
The benchmark measures the standalone parser facade and a pre-tokenized
parser-only path. Lower latency is better.
- Host: MacBookPro17,1, Apple M1 (8 cores, 16 GB), macOS 15.0.1
- Runtime: OpenJDK 17.0.20.1, ANTLR 4.13.1, JMH 1.37, 1 thread, 1 GB heap,
`-prof gc`
- Standard run: 3 forks, 4 x 300 ms warmup, 7 x 400 ms measurement
- Longer target run: 3 forks, 6 x 500 ms warmup, 10 x 700 ms measurement
- Baseline: `d7f44fcfedd`; benchmark jar SHA-256
`2e33becac22a27d8c40eec7c821c506cbc9bcedfda277725fa8fee40c6d35116`
- Candidate: `5e0eadb13e9`; benchmark jar SHA-256
`743c133321dbb1a2283ea1836e5a9568a14ecc2ae350b2fc239d354d7988b423`
- Harness:
`fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrDmlCommonPrefixBenchmark.java`
#### Longer parser-only target run
| Workload | Baseline us/op | Candidate us/op | Latency change | Baseline
B/op | Candidate B/op | Allocation change |
|---|---:|---:|---:|---:|---:|---:|
| CTE SELECT | 43.273 +/- 1.101 | 43.498 +/- 1.528 | +0.52% | 73,453.8 |
73,195.1 | -0.35% |
| CTE INSERT | 48.439 +/- 1.233 | 42.907 +/- 1.236 | -11.42% | 74,096.5 |
74,560.4 | +0.63% |
| EXPLAIN CTE INSERT | 48.032 +/- 1.446 | 42.356 +/- 0.580 | -11.82% |
74,360.5 | 74,963.1 | +0.81% |
#### Public-facade control cases from the standard run
| Workload | Baseline us/op | Candidate us/op | Latency change | Baseline
B/op | Candidate B/op |
|---|---:|---:|---:|---:|---:|
| SELECT control | 2.636 +/- 0.187 | 2.388 +/- 0.167 | -9.42% | 5,200.0 |
4,928.0 |
| EXPLAIN SELECT | 7.094 +/- 1.151 | 6.316 +/- 0.274 | -10.97% | 10,730.8 |
10,728.1 |
The previous grammar performed about 56 tokens of top-level lookahead for a
long CTE and then about 62 more tokens in the DML decision. Factoring the
prefix removes the repeated prediction, which accounts for the CTE INSERT
gains. CTE SELECT remains flat within overlapping confidence intervals, and
reliable target allocation changes remain below 1%. The standard end-to-end
long-CTE forks were noisy, so this PR makes no precise end-to-end long-CTE or
whole-FE latency claim.
Build and run commands:
```shell
mvn -Pbenchmark -pl fe-sql-parser-benchmark -am clean package \
-Dmaven.build.cache.enabled=false -Dskip.doc=true
java -jar
fe/fe-sql-parser-benchmark/target/doris-fe-sql-parser-benchmarks.jar \
'.*QueryOrDmlCommonPrefixBenchmark.*' \
-prof gc -rf json -rff /tmp/antlr-p1-candidate.json
java -jar
fe/fe-sql-parser-benchmark/target/doris-fe-sql-parser-benchmarks.jar \
'.*QueryOrDmlCommonPrefixBenchmark.parsePreTokenized.*' \
-p workload=cteSelect,cteInsert,explainCteInsert \
-wi 6 -w 500ms -i 10 -r 700ms -f 3 \
-prof gc -rf json -rff /tmp/antlr-p1-candidate-long.json
```
Raw local JSON files: `/tmp/antlr-p1-{baseline,candidate}.json` and
`/tmp/antlr-p1-{baseline,candidate}-long.json`.
### Semantic differential
- Corpus: all 4,610 tracked `*.sql` files; manifest SHA-256
`567e209d57e5eaf6546ff03bf887437b8d647ed5f7ecb85bc657b987dd04be10`
- Result: 4,275 parsed and 335 rejected in both artifacts
- Per-file success/statement-count or error-class/first-position signature
SHA-256: `8cfc5acf0d5197af2f3f4cfbfba35c17cf3ee6cffd9310cdbd79010698354e25` for
both artifacts
- Lexer and token behavior are unchanged by construction.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Parser and benchmark reactor: 190 tests passed
- FE targeted parser tests: 26 passed, 0 failures
- FE and parser Checkstyle: 0 violations
- Tracked SQL corpus differential: identical acceptance and first-error
signatures
- JMH standard and longer target runs completed with 3 forks
- The standard `run-fe-ut.sh` invocation was blocked before tests by the
local Thrift compiler 0.16.0 versus the repository-required 0.24.0. The FE
reactor and targeted tests passed using the existing generated Thrift sources
and an untracked temporary compiler shim.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
