morrySnow opened a new pull request, #67429:
URL: https://github.com/apache/doris/pull/67429
### What problem does this PR solve?
Issue Number: None
Related PR: #67427
Problem Summary: This draft PR is stacked on #67427. The grammar allowed
both `querySpecification` and its outer `query` to consume `ORDER BY` and
`LIMIT`, so ANTLR repeatedly entered `queryOrganization` and reported
full-context ambiguities. This change gives every clause group one owner
according to ANSI mode and query level, makes the rule structurally non-empty,
preserves the existing accepted/rejected SQL matrix and first-error positions,
and updates the FE plan builder for the labeled contexts. It also adds focused
standalone-parser and FE plan tests plus a JMH harness for the affected path.
### Benchmark
Lower latency is better. The target input is `SELECT a, b, c FROM t WHERE a
> 1 ORDER BY a, b DESC LIMIT 20 OFFSET 10`; the control input has the same
SELECT without query-organization clauses.
- Host: MacBookPro17,1, Apple M1 (8 cores, 16 GB), macOS 15.0.1
- Runtime: OpenJDK 17.0.20.1, ANTLR 4.13.1, JMH 1.37, 1 thread, 1 GB heap
- JMH: 3 forks, 4 x 300 ms warmup, 7 x 400 ms measurement; C1-B1-C2
interleaving
- Baseline: `5e0eadb13e9`; parser jar SHA-256
`530f0ed45c4bac3a096373a1e932aae0025c603a6f905c3d975a2ad53bd0e7ca`; benchmark
jar SHA-256 `7543595b2a87f55b9b53538336cfedf55c13484302ffb487efcfb4fdd66b1c0b`
- Candidate: `773c4ee1027`; parser jar SHA-256
`65730a22a5262370a832bda7a04ae4b8e0de5c107be65c5ae80f95c711842f59`; benchmark
jar SHA-256 `47284ffd6779e9d534a19a682de5a4d468f83a62a98e048982807497f1f665b6`
- Harness:
`fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrganizationBenchmark.java`
#### Ordered SELECT target
The candidate change is calculated from the mean of C1 and C2 against B1.
Values are us/op with JMH's 99.9% error estimate.
| Path | Mode | Baseline B1 | Candidate C1 | Candidate C2 | Candidate mean
change |
|---|---|---:|---:|---:|---:|
| End-to-end | Legacy | 9.926 +/- 0.347 | 9.303 +/- 0.333 | 9.122 +/- 0.267
| -7.2% |
| End-to-end | ANSI | 9.170 +/- 0.437 | 8.709 +/- 0.646 | 8.492 +/- 0.241 |
-6.2% |
| Pre-tokenized parser | Legacy | 7.653 +/- 0.266 | 7.246 +/- 0.244 | 7.069
+/- 0.090 | -6.5% |
| Pre-tokenized parser | ANSI | 7.344 +/- 0.105 | 7.181 +/- 0.099 | 7.099
+/- 0.076 | -2.8% |
#### Plain SELECT control
| Path | Mode | Baseline us/op | Candidate us/op | Change |
|---|---|---:|---:|---:|
| End-to-end | Legacy | 6.580 +/- 0.710 | 6.657 +/- 0.809 | +1.2% |
| End-to-end | ANSI | 6.944 +/- 0.648 | 6.448 +/- 0.440 | -7.1% |
| Pre-tokenized parser | Legacy | 5.684 +/- 1.205 | 5.084 +/- 0.054 | -10.6%
|
| Pre-tokenized parser | ANSI | 5.313 +/- 0.125 | 5.193 +/- 0.089 | -2.3% |
No control-path latency regression exceeds the 3% threshold. A later B2 run
was affected by large machine-level outliers (`13.841 +/- 3.137` us/op for
legacy end-to-end) and is retained as raw data but excluded from the latency
claim; the target numbers above should be rerun on a quiet host before marking
the PR ready for review.
#### Allocation
The same artifacts were measured separately with `-prof gc`. Timing under
the profiler was noisy, so this table uses only normalized allocation.
| Path | Mode | Baseline B/op | Candidate B/op | Change |
|---|---|---:|---:|---:|
| End-to-end | Legacy | 15,473.6 | 15,719.6 | +1.59% |
| End-to-end | ANSI | 15,426.8 | 15,536.1 | +0.71% |
| Pre-tokenized parser | Legacy | 12,368.1 | 12,381.4 | +0.11% |
| Pre-tokenized parser | ANSI | 12,317.4 | 12,421.4 | +0.84% |
All allocation changes are below the 3% threshold. The profiler's legacy
end-to-end timing changed by +6.6%, but its broad confidence intervals overlap;
the non-profiler interleaved results above are used for the latency conclusion.
#### Profile attribution
- Before P2, tracked SQL entered `queryOrganization` 13,196 times and
examined 49,902 lookahead tokens; SSB and Trino profiling reported 40 and 20
related ambiguities.
- After P2, the 4,275 parseable tracked SQL files enter the target rule
2,794 times and examine 8,645 lookahead tokens, a reduction of about 79% and
83% respectively.
- LL_EXACT reports zero fallback and zero ambiguity for both the
`querySpecification` and `queryOrganization` target decisions.
- The gain therefore comes from eliminating duplicate ownership and adaptive
lookahead, not from lexer or token changes.
Build and benchmark commands:
```shell
mvn -Pbenchmark -pl fe-sql-parser-benchmark -am package \
-Dmaven.build.cache.enabled=false -Dskip.doc=true
java -jar
fe/fe-sql-parser-benchmark/target/doris-fe-sql-parser-benchmarks.jar \
'.*QueryOrganizationBenchmark.*' -p workload=orderedSelect \
-rf json -rff /tmp/antlr-p2-candidate-ordered.json
java -jar
fe/fe-sql-parser-benchmark/target/doris-fe-sql-parser-benchmarks.jar \
'.*QueryOrganizationBenchmark.*' -p workload=orderedSelect \
-prof gc -rf json -rff /tmp/antlr-p2-candidate-ordered-gc.json
```
Raw local results:
- Latency:
`/tmp/antlr-p2-{baseline-ordered-current,final-ordered,final-ordered-2,baseline-ordered-current-2}.json`
- Control: `/tmp/antlr-p2-{baseline-plain-current,final-plain-current}.json`
- Allocation: `/tmp/antlr-p2-{baseline,candidate}-ordered-gc.json`
### Semantic differential
- Corpus: all 4,610 tracked `*.sql` files; manifest SHA-256
`567e209d57e5eaf6546ff03bf887437b8d647ed5f7ecb85bc657b987dd04be10`
- Result: 4,275 parsed and 335 rejected in both artifacts and in both ANSI
modes
- Per-file success/statement-count or error-class/first-position signature
SHA-256: `69c811d0d80c52b40d8cd854e925ff4930aa6601c7d1e66541f2eb29f35db50a` for
every baseline/candidate output
- An additional 15-case invalid SQL matrix has identical exception class and
first-error position in both modes
- Lexer and token behavior are unchanged by construction
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Parser and benchmark reactor: 238 tests passed
- FE targeted parser tests: 4 passed, 0 failures
- FE and parser Checkstyle: 0 violations
- Tracked SQL corpus differential: identical acceptance and first-error
signatures in legacy and ANSI modes
- LL_EXACT profiling and JMH target/control/GC-profiler runs completed
- The standard `run-fe-ut.sh` invocation was blocked before tests by the
local Thrift compiler 0.16.0 versus the repository-required 0.24.0. The
targeted FE tests passed through the Maven reactor with an untracked temporary
Thrift 0.24 shim.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
