This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7fc8e276284 [feat](sql-parser) Split SQL grammar into standalone
fe-sql-parser (#63823)
7fc8e276284 is described below
commit 7fc8e276284eac8a0155b90155f54e0e9723c0a6
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri May 29 14:30:16 2026 +0800
[feat](sql-parser) Split SQL grammar into standalone fe-sql-parser (#63823)
## Summary
Split SQL syntax parsing out of `fe-core` into a new `fe-sql-parser`
module that produces an ANTLR parse tree (CST) without semantic
analysis. The new module can be packaged as an independent jar for
external consumers (third-party tools, linters, format converters, etc.)
without dragging in `LogicalPlan`, `Catalog`, `ConnectContext`, or any
other fe-core internals.
### Module changes
- Move `DorisLexer.g4` / `DorisParser.g4` and 8 supporting Java files
(`CaseInsensitiveStream`, `Origin`, `ParserUtils`, `ParseErrorListener`,
`PostProcessor`, `ParseException`, `SyntaxParseException`,
`QueryParsingErrors`) into `fe-sql-parser`. Package names are preserved
so fe-core's hundreds of imports do not move.
- `ParseException` now extends `RuntimeException` directly to break the
chain through `nereids.exceptions.AnalysisException`, which references
`LogicalPlan`.
- Introduce an `OriginAware` SPI in `fe-sql-parser` so `ParserUtils`
keeps its per-thread field-based fast path (originally added in #52125).
`MoreFieldsThread` in fe-core implements the interface; threads that
don't fall back to ThreadLocal — correctness is identical either way.
- Add `org.apache.doris.sqlparser.DorisSqlParser` facade with
`parseStatement` / `parseStatements` / `parseExpression`.
- fe-core reverse-depends on fe-sql-parser; its own
`antlr4-maven-plugin` now only processes the Nereids pattern-generator's
`JavaLexer.g4` / `JavaParser.g4`.
The new module's only runtime dependency is `org.antlr:antlr4-runtime`.
### Standalone CLI
`mvn -pl fe-sql-parser -Pcli package` produces a self-contained
executable jar (`fe-sql-parser-*-cli.jar`, ~1.7 MB after minimize-shade)
so the parser can be invoked directly from a shell:
```bash
$ java -jar fe-sql-parser-*-cli.jar "SELECT a FROM t WHERE a > 1"
(singleStatement (statement ...) <EOF>)
$ java -jar fe-sql-parser-*-cli.jar --pretty --multi "USE db; SELECT 1;
SELECT 2"
$ java -jar fe-sql-parser-*-cli.jar --expression "a + 1 * COALESCE(b, 0)"
$ echo "SELECT 1" | java -jar fe-sql-parser-*-cli.jar
$ java -jar fe-sql-parser-*-cli.jar -f query.sql
```
The CLI is gated behind the `cli` Maven profile so default Doris builds
do not pay the shading cost; the thin jar consumed by fe-core is
unchanged. Exit codes: `0` success, `1` parse error, `2` usage or I/O
error.
### Extension hooks for downstream tools
Downstream projects can plug in custom logic (SQL lineage, policy
enforcement, audit, rewriting, metrics) **without modifying
`fe-sql-parser`**. Four mechanisms are available:
| Mechanism | When it fires | Typical use |
|-----------|---------------|-------------|
| Subclass `DorisParserBaseVisitor<T>` | After parsing | Extract
information, rewrite, lineage |
| Subclass `DorisParserBaseListener` | After parsing | Simple
`enter`/`exit` interception |
| `parser.addParseListener(...)` via `newLexer` / `newParser` | Live,
during parsing | Token-level processing, on-the-fly mutation |
| Wrap `DorisSqlParser` | Around the call | Metrics, caching,
request-level policy |
`fe/fe-sql-parser/README.md` contains end-to-end examples for SQL
lineage extraction, policy/audit listeners, hint collection during
parsing, an instrumented facade with caching and metrics, plus tips on
locating rule names and debugging visitors with the CLI. It also
documents the build modes, library/Visitor usage, configuration flags
(`noBackslashEscapes`, `ansiSqlSyntax`), the `OriginAware` fast-path
SPI, and current caveats.
## Test plan
- [x] `fe-sql-parser` unit tests: 7 new cases in `DorisSqlParserTest`
covering `SELECT`, `SELECT FROM WHERE`, multi-statement, expression, DDL
(`CREATE TABLE` with `DISTRIBUTED` + `PROPERTIES`), malformed SQL, and
trailing-garbage expressions
- [x] Full `fe` reactor compiles (`mvn -pl fe-core -am compile`)
- [x] fe-core's 15 existing `org.apache.doris.nereids.parser.*Test`
classes pass (160 cases, 0 failures)
- [x] CLI smoke test: positional / `-e` / `-f` / stdin input modes;
`--multi` / `--expression` parse modes; `--pretty` output; parse-error
exit code
---------
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
---
fe/fe-core/pom.xml | 5 +
.../doris/nereids/exceptions/CastException.java | 2 +-
.../doris/nereids/util/MoreFieldsThread.java | 5 +-
fe/fe-sql-parser/README.md | 545 +++++++++++++++++++++
fe/fe-sql-parser/pom.xml | 165 +++++++
.../antlr4/org/apache/doris/nereids/DorisLexer.g4 | 0
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 0
.../doris/nereids/errors/QueryParsingErrors.java | 0
.../nereids/exceptions/AnalysisException.java | 18 +-
.../doris/nereids/exceptions/ParseException.java | 2 +-
.../nereids/exceptions/SyntaxParseException.java | 0
.../nereids/parser/CaseInsensitiveStream.java | 0
.../org/apache/doris/nereids/parser/Origin.java | 0
.../apache/doris/nereids/parser/OriginAware.java} | 32 +-
.../doris/nereids/parser/ParseErrorListener.java | 0
.../apache/doris/nereids/parser/ParserUtils.java | 28 +-
.../apache/doris/nereids/parser/PostProcessor.java | 0
.../org/apache/doris/sqlparser/DorisSqlParser.java | 124 +++++
.../apache/doris/sqlparser/DorisSqlParserCli.java | 237 +++++++++
.../apache/doris/sqlparser/DorisSqlParserTest.java | 75 +++
fe/pom.xml | 1 +
21 files changed, 1187 insertions(+), 52 deletions(-)
diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml
index 6e6ec0969b5..5c258a4983c 100644
--- a/fe/fe-core/pom.xml
+++ b/fe/fe-core/pom.xml
@@ -344,6 +344,11 @@ under the License.
<artifactId>antlr4-runtime</artifactId>
<version>${antlr4.version}</version>
</dependency>
+ <dependency>
+ <groupId>${project.groupId}</groupId>
+ <artifactId>fe-sql-parser</artifactId>
+ <version>${project.version}</version>
+ </dependency>
<dependency>
<groupId>com.aliyun.odps</groupId>
<artifactId>odps-sdk-core</artifactId>
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/CastException.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/CastException.java
index 5d3ac70206d..ab8b191217d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/CastException.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/CastException.java
@@ -27,7 +27,7 @@ public class CastException extends AnalysisException {
private final String message;
public CastException(String message) {
- super(ErrorCode.NONE, message, Optional.of(0), Optional.of(0),
Optional.empty());
+ super(ErrorCode.NONE, message, Optional.of(0), Optional.of(0));
this.message = message;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/MoreFieldsThread.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/MoreFieldsThread.java
index 359fb098f02..714b2ecaa1c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/MoreFieldsThread.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/MoreFieldsThread.java
@@ -18,6 +18,7 @@
package org.apache.doris.nereids.util;
import org.apache.doris.nereids.parser.Origin;
+import org.apache.doris.nereids.parser.OriginAware;
import org.apache.doris.qe.ConnectContext;
import java.util.function.Supplier;
@@ -26,7 +27,7 @@ import java.util.function.Supplier;
* This class is used to extend some thread local fields for Thread,
* so we can access the thread fields faster than ThreadLocal
*/
-public class MoreFieldsThread extends Thread {
+public class MoreFieldsThread extends Thread implements OriginAware {
private static final ThreadLocal<Boolean> keepFunctionSignatureThreadLocal
= new ThreadLocal<>();
private static final ThreadLocal<ConnectContext> connectContextThreadLocal
= new ThreadLocal<>();
@@ -65,10 +66,12 @@ public class MoreFieldsThread extends Thread {
super(group, target, name, stackSize);
}
+ @Override
public final void setOrigin(Origin origin) {
this.origin = origin;
}
+ @Override
public final Origin getOrigin() {
return this.origin;
}
diff --git a/fe/fe-sql-parser/README.md b/fe/fe-sql-parser/README.md
new file mode 100644
index 00000000000..3c39aed3db3
--- /dev/null
+++ b/fe/fe-sql-parser/README.md
@@ -0,0 +1,545 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Doris FE SQL Parser
+
+`fe-sql-parser` is a standalone ANTLR4-based syntax parser for Apache Doris
SQL. It produces an ANTLR parse tree (CST) for any Doris-dialect SQL string. It
performs **no semantic analysis**: identifiers are not resolved, tables and
columns are not validated, and types are not checked. The module has a single
runtime dependency: `org.antlr:antlr4-runtime`.
+
+The module is the single source of truth for the Doris SQL grammar; `fe-core`
consumes the parser through this module rather than maintaining its own copy.
+
+## Module Layout
+
+```
+fe-sql-parser/
+├── pom.xml
+├── src/main/antlr4/org/apache/doris/nereids/
+│ ├── DorisLexer.g4 # Doris SQL lexer grammar
+│ └── DorisParser.g4 # Doris SQL parser grammar
+└── src/main/java/
+ ├── org/apache/doris/nereids/
+ │ ├── parser/ # Parser support: CaseInsensitiveStream,
+ │ │ # Origin, OriginAware, ParserUtils,
+ │ │ # ParseErrorListener, PostProcessor
+ │ ├── exceptions/ # ParseException, SyntaxParseException
+ │ └── errors/QueryParsingErrors.java
+ └── org/apache/doris/sqlparser/
+ ├── DorisSqlParser.java # Public library facade
+ └── DorisSqlParserCli.java # Command-line entry point
+```
+
+At build time the ANTLR Maven plugin generates
`org.apache.doris.nereids.DorisLexer`, `DorisParser`, `DorisParserBaseVisitor`,
and `DorisParserBaseListener` into `target/generated-sources/antlr4/`.
+
+## Build
+
+The module has two build modes: the default mode produces a thin library jar
that `fe-core` and downstream tools depend on; the `cli` profile additionally
produces a self-contained executable jar.
+
+### Library jar (default build)
+
+```bash
+# From the fe/ directory
+mvn -pl fe-sql-parser -am package
+```
+
+Output: `fe/fe-sql-parser/target/doris-fe-sql-parser.jar` (~1.3 MB). This jar
contains only the parser classes; it expects `org.antlr:antlr4-runtime:4.13.1`
to be provided by the consuming project's classpath.
+
+To install it to your local Maven repository so other projects can resolve it:
+
+```bash
+mvn -pl fe-sql-parser -am -Pflatten install -DskipTests
+```
+
+The `flatten` profile is required so the installed POM has `${revision}`
resolved to a concrete version.
+
+### Standalone CLI jar
+
+```bash
+# From the fe/ directory
+mvn -pl fe-sql-parser -Pcli package -DskipTests
+```
+
+Output: `fe/fe-sql-parser/target/fe-sql-parser-1.2-SNAPSHOT-cli.jar` (~1.7 MB).
+
+This is a self-contained executable jar produced by `maven-shade-plugin`:
+
+- Bundles `antlr4-runtime` so the jar runs anywhere with a JRE 8+
+- Manifest sets `Main-Class: org.apache.doris.sqlparser.DorisSqlParserCli`
+- `<minimizeJar>true</minimizeJar>` strips unused classes
(transitively-inherited logging, test utilities, etc.) so the final jar
contains only the parser plus its actual reachable dependencies
+
+The CLI profile is gated so default Doris builds do not pay the shading cost.
The thin library jar produced by the default build is unaffected — `fe-core`
continues to consume it directly.
+
+## CLI Usage
+
+```
+java -jar fe-sql-parser-1.2-SNAPSHOT-cli.jar [OPTIONS] [SQL]
+```
+
+### Input sources (mutually exclusive)
+
+| Source | Example |
+|--------|---------|
+| Positional argument | `java -jar ...-cli.jar "SELECT 1"` |
+| `-e` / `--exec <SQL>` | `java -jar ...-cli.jar -e "SELECT 1"` |
+| `-f` / `--file <PATH>` | `java -jar ...-cli.jar -f query.sql` |
+| stdin (when none of the above) | `echo "SELECT 1" \| java -jar ...-cli.jar` |
+
+### Parse modes
+
+| Flag | Grammar rule | Use case |
+|------|--------------|----------|
+| (default) | `singleStatement` | One SQL statement |
+| `--multi` | `multiStatements` | Multiple statements separated by `;` |
+| `--expression` | `expressionWithEof` | A single SQL expression |
+
+### Output formats
+
+| Flag | Output |
+|------|--------|
+| (default) | ANTLR LISP-style tree on one line |
+| `--pretty` | Indented multi-line tree, two-space indent per level |
+
+### Dialect flags
+
+| Flag | Effect |
+|------|--------|
+| `--no-backslash-escapes` | Maps to MySQL's `NO_BACKSLASH_ESCAPES` sql_mode —
backslash is not a string-literal escape character |
+| `--ansi` | Enables ANSI SQL syntax variants in the few grammar rules that
branch on it |
+
+### Exit codes
+
+| Code | Meaning |
+|------|---------|
+| 0 | Parse succeeded |
+| 1 | Parse failed — `ParseException` thrown; the error message is printed to
stderr with the offending line/column and a `^^^` pointer |
+| 2 | Usage error or I/O error (bad flag, unreadable file, empty input) |
+
+### Examples
+
+Single statement, default LISP format:
+
+```bash
+$ java -jar ...-cli.jar "SELECT 1"
+(singleStatement (statement (statementBase (query (queryTerm (queryPrimary
+(querySpecification (selectClause SELECT (selectColumnClause
(namedExpressionSeq
+(namedExpression (expression (booleanExpression (valueExpression
(primaryExpression
+(constant (number 1)))))))))) queryOrganization))) queryOrganization))) <EOF>)
+```
+
+Single statement, pretty format:
+
+```bash
+$ java -jar ...-cli.jar --pretty "SELECT a FROM t WHERE a > 1"
+singleStatement
+ statement
+ statementBase
+ query
+ queryTerm
+ queryPrimary
+ querySpecification
+ selectClause
+ 'SELECT'
+ ...
+ fromClause
+ 'FROM'
+ ...
+ whereClause
+ 'WHERE'
+ ...
+ '<EOF>'
+```
+
+Multiple statements:
+
+```bash
+$ java -jar ...-cli.jar --multi "USE db1; SELECT 1; SELECT 2"
+```
+
+Single expression:
+
+```bash
+$ java -jar ...-cli.jar --expression "a + 1 * COALESCE(b, 0)"
+```
+
+From file:
+
+```bash
+$ java -jar ...-cli.jar -f path/to/my-query.sql
+```
+
+From stdin (pipe a heredoc or another command's output):
+
+```bash
+$ cat my-query.sql | java -jar ...-cli.jar
+```
+
+Parse error — note the non-zero exit code:
+
+```bash
+$ java -jar ...-cli.jar "SELEKT 1"
+mismatched input 'SELEKT' expecting {...}(line 1, pos 0)
+$ echo $?
+1
+```
+
+### Shell wrapper (optional)
+
+For frequent use, drop a wrapper on your `PATH`:
+
+```bash
+# ~/bin/doris-sql-parse
+#!/usr/bin/env bash
+exec java -jar /path/to/fe-sql-parser-1.2-SNAPSHOT-cli.jar "$@"
+```
+
+```bash
+chmod +x ~/bin/doris-sql-parse
+doris-sql-parse --pretty "SELECT 1"
+```
+
+## Library Usage
+
+If you want to embed the parser in another JVM application rather than
shelling out to the CLI.
+
+### Maven dependency
+
+```xml
+<dependency>
+ <groupId>org.apache.doris</groupId>
+ <artifactId>fe-sql-parser</artifactId>
+ <version>1.2-SNAPSHOT</version>
+</dependency>
+<!-- antlr4-runtime is pulled in transitively; declare it explicitly if you
+ want to pin a specific version -->
+<dependency>
+ <groupId>org.antlr</groupId>
+ <artifactId>antlr4-runtime</artifactId>
+ <version>4.13.1</version>
+</dependency>
+```
+
+Until the artifact is published to a public repository you need to `mvn
install` it locally (see [Library jar](#library-jar-default-build) above).
+
+### Minimal example
+
+```java
+import org.apache.doris.sqlparser.DorisSqlParser;
+import org.apache.doris.nereids.DorisParser.SingleStatementContext;
+
+DorisSqlParser parser = new DorisSqlParser();
+SingleStatementContext tree = parser.parseStatement("SELECT a, b FROM t WHERE
a > 1");
+// `tree` is a standard ANTLR ParseTree; walk it with a Visitor or Listener.
+```
+
+### Walking the tree with a Visitor
+
+```java
+import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.nereids.DorisParserBaseVisitor;
+
+import java.util.ArrayList;
+import java.util.List;
+
+DorisSqlParser parser = new DorisSqlParser();
+SingleStatementContext tree = parser.parseStatement(
+ "SELECT u.id FROM users u JOIN orders o ON u.id = o.uid");
+
+List<String> tables = new ArrayList<>();
+new DorisParserBaseVisitor<Void>() {
+ @Override
+ public Void visitTableName(DorisParser.TableNameContext ctx) {
+ tables.add(ctx.multipartIdentifier().getText());
+ return super.visitTableName(ctx);
+ }
+}.visit(tree);
+
+System.out.println(tables); // [users, orders]
+```
+
+`DorisParserBaseVisitor<T>` and `DorisParserBaseListener` are generated by
ANTLR — every grammar rule has a corresponding `visitXxx` / `enterXxx` /
`exitXxx` method you can override.
+
+### Error handling
+
+`ParseException` is a `RuntimeException`. You do not have to declare or catch
it, but you usually want to:
+
+```java
+import org.apache.doris.nereids.exceptions.ParseException;
+
+try {
+ parser.parseStatement("SELEKT 1");
+} catch (ParseException e) {
+ // e.getMessage() includes "line N, pos M" and a `^^^` pointer into the
SQL.
+ System.err.println(e.getMessage());
+}
+```
+
+### Lexer-only / token-level work
+
+If you only need tokens (SQL formatter, comment extractor, keyword finder,
hint inspector), skip the parser:
+
+```java
+import org.apache.doris.nereids.DorisLexer;
+import org.antlr.v4.runtime.Token;
+
+DorisSqlParser parser = new DorisSqlParser();
+DorisLexer lexer = parser.newLexer("SELECT /*+ HINT */ a FROM t");
+Token token;
+while ((token = lexer.nextToken()).getType() != Token.EOF) {
+ System.out.printf("%-20s %s%n",
+ DorisLexer.VOCABULARY.getSymbolicName(token.getType()),
+ token.getText());
+}
+```
+
+## Extending the Parser
+
+Downstream projects can plug in custom logic (lineage tracking, policy
enforcement, audit, SQL rewriting, metrics) **without modifying `fe-sql-parser`
itself**. There are four extension points:
+
+| Mechanism | When it fires | Typical use |
+|-----------|---------------|-------------|
+| Subclass `DorisParserBaseVisitor<T>` | After parsing, when you call
`visitor.visit(tree)` | Extract information, rewrite, lineage |
+| Subclass `DorisParserBaseListener` | After parsing, when you call
`ParseTreeWalker.walk(...)` | Simple `enter`/`exit` interception |
+| `parser.addParseListener(...)` | Live, while the parser is building the tree
| Token-level processing, on-the-fly mutation |
+| Wrap `DorisSqlParser` | Around the `parseStatement` call | Metrics, caching,
request-level policy |
+
+All ANTLR-generated classes (`DorisParser`, `DorisParserBaseVisitor`,
`DorisParserBaseListener`) and the `DorisSqlParser` facade are `public`, so
downstream code uses them directly.
+
+### Example 1: Visitor — SQL lineage (source → target)
+
+The most common pattern. Extract "which tables were read" and "which table was
written" from a single statement.
+
+```java
+import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.nereids.DorisParserBaseVisitor;
+import org.apache.doris.sqlparser.DorisSqlParser;
+
+import java.util.LinkedHashSet;
+import java.util.Set;
+
+public class LineageExtractor extends DorisParserBaseVisitor<Void> {
+ public final Set<String> sources = new LinkedHashSet<>();
+ public String target;
+
+ // INSERT INTO target_db.target_tbl SELECT ... FROM source ...
+ @Override
+ public Void visitInsertTable(DorisParser.InsertTableContext ctx) {
+ target = ctx.tableName.getText();
+ return super.visitInsertTable(ctx); // keep descending to collect
sources
+ }
+
+ // Any FROM <table> / JOIN <table> hits this
+ @Override
+ public Void visitTableName(DorisParser.TableNameContext ctx) {
+ sources.add(ctx.multipartIdentifier().getText());
+ return null;
+ }
+}
+
+// Usage
+DorisSqlParser parser = new DorisSqlParser();
+LineageExtractor lineage = new LineageExtractor();
+lineage.visit(parser.parseStatement(
+ "INSERT INTO sink SELECT a.x, b.y FROM src1 a JOIN src2 b ON a.id =
b.id"));
+
+System.out.println(lineage.target); // sink
+System.out.println(lineage.sources); // [src1, src2]
+```
+
+For **column-level lineage**, also override `visitColumnReference` /
`visitNamedExpression` and maintain a stack of "current SELECT scope" so each
column reference can be attributed to the right output column.
+
+### Example 2: Listener — policy enforcement / audit
+
+Use the listener pattern when you only care whether the parser entered a
certain rule, not its return value.
+
+```java
+import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.nereids.DorisParserBaseListener;
+import org.antlr.v4.runtime.tree.ParseTreeWalker;
+
+public class DropGuardListener extends DorisParserBaseListener {
+ @Override
+ public void
enterSupportedDropStatement(DorisParser.SupportedDropStatementContext ctx) {
+ throw new SecurityException("DROP statements are not allowed: " +
ctx.getText());
+ }
+}
+
+// Usage
+ParseTreeWalker.DEFAULT.walk(
+ new DropGuardListener(),
+ parser.parseStatement(userSql));
+```
+
+Audit-style collection:
+
+```java
+public class AuditListener extends DorisParserBaseListener {
+ public final List<String> writes = new ArrayList<>();
+
+ @Override public void enterInsertTable(DorisParser.InsertTableContext ctx)
{
+ writes.add("INSERT " + ctx.tableName.getText());
+ }
+ @Override public void enterUpdate(DorisParser.UpdateContext ctx) {
+ writes.add("UPDATE " + ctx.tableName.getText());
+ }
+ @Override public void enterDelete(DorisParser.DeleteContext ctx) {
+ writes.add("DELETE " + ctx.tableName.getText());
+ }
+ @Override public void
enterSupportedDropStatement(DorisParser.SupportedDropStatementContext ctx) {
+ writes.add("DROP " + ctx.getText());
+ }
+}
+```
+
+### Example 3: Live `ParseTreeListener` — fire during parsing
+
+Most cases are covered by Examples 1 and 2. If you need to intervene **while
the parser is building each node** (mutating tokens, injecting metadata,
streaming work), attach a listener with `parser.addParseListener(...)`. This is
exactly how `fe-sql-parser`'s internal `PostProcessor` rewrites identifier case
at parse time.
+
+`DorisSqlParser.parseStatement` does not expose the parser instance; use
`newLexer` + `newParser` to take ownership:
+
+```java
+import org.apache.doris.nereids.DorisLexer;
+import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.nereids.DorisParserBaseListener;
+import org.apache.doris.sqlparser.DorisSqlParser;
+
+public class HintCollectorListener extends DorisParserBaseListener {
+ public final List<String> hints = new ArrayList<>();
+
+ @Override
+ public void exitOptimizeHint(DorisParser.OptimizeHintContext ctx) {
+ hints.add(ctx.getText());
+ }
+}
+
+DorisSqlParser facade = new DorisSqlParser();
+DorisLexer lexer = facade.newLexer(sql);
+DorisParser parser = facade.newParser(lexer);
+
+HintCollectorListener hintListener = new HintCollectorListener();
+parser.addParseListener(hintListener);
+
+DorisParser.SingleStatementContext tree = parser.singleStatement();
+System.out.println(hintListener.hints);
+```
+
+`newParser` already attaches `PostProcessor` and `ParseErrorListener`; your
listener is added on top.
+
+### Example 4: Wrap the facade — metrics, caching, rewriting
+
+For "do something before and after every parse" (instrumentation, PII
redaction, request-level routing), composition is the cleanest pattern:
+
+```java
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import io.micrometer.core.instrument.MeterRegistry;
+import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.sqlparser.DorisSqlParser;
+
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+
+public class InstrumentedDorisSqlParser {
+ private final DorisSqlParser delegate;
+ private final Cache<String, DorisParser.SingleStatementContext> cache;
+ private final MeterRegistry metrics;
+
+ public InstrumentedDorisSqlParser(MeterRegistry metrics) {
+ this.delegate = new DorisSqlParser();
+ this.cache = Caffeine.newBuilder().maximumSize(10_000).build();
+ this.metrics = metrics;
+ }
+
+ public DorisParser.SingleStatementContext parse(String sql) {
+ // pre-hook: redact literals so semantically equivalent queries share
a cache entry
+ String normalized = redactLiterals(sql);
+ return cache.get(normalized, key -> {
+ long start = System.nanoTime();
+ try {
+ return delegate.parseStatement(key);
+ } finally {
+ metrics.timer("sql.parse").record(System.nanoTime() - start,
NANOSECONDS);
+ }
+ });
+ }
+}
+```
+
+### Example 5: Stack multiple hooks on the same tree
+
+Different teams can maintain their own hook classes; you do not need to merge
them into one giant visitor. `ParseTreeWalker` can walk the same tree multiple
times:
+
+```java
+ParseTree tree = parser.parseStatement(sql);
+
+LineageExtractor lineage = new LineageExtractor();
+AuditListener audit = new AuditListener();
+HintCollectorListener hints = new HintCollectorListener();
+
+lineage.visit(tree);
+ParseTreeWalker.DEFAULT.walk(audit, tree);
+ParseTreeWalker.DEFAULT.walk(hints, tree);
+```
+
+### Tips
+
+- **Finding the rule names**: every overrideable `visitXxx` / `enterXxx` /
`exitXxx` corresponds 1:1 to a `xxx:` rule in `DorisParser.g4`. Open
`DorisParserBaseVisitor` in your IDE to see the full list, or run the CLI with
`--pretty` to see the actual rule names that appear in the tree for your SQL,
then target them in your visitor.
+- **Remember to call `super.visitXxx(ctx)`**: a visitor's default behavior is
to recurse into children. If you forget `super`, nothing below the current node
will be visited. Either `return super.visitXxx(ctx)` to keep recursing, or
`return null` to explicitly prune.
+- **Don't throw arbitrary runtime exceptions from hooks**: they bypass
`fe-sql-parser`'s own error-location plumbing. If you need to fail inside a
visitor, throw an exception that carries `Origin`-style line/column info (see
`ParserUtils.position(Token)`).
+- **Debug your visitor with the CLI first**: the CLI doesn't know about your
visitor, but `--pretty` output tells you exactly what rule names show up for
any SQL — much faster than guessing.
+
+## Configuration Knobs
+
+`DorisSqlParser` is configured via constructor flags. Both default to `false`,
which matches the most common Doris query behavior.
+
+```java
+DorisSqlParser parser = new DorisSqlParser(
+ /* noBackslashEscapes = */ false,
+ /* ansiSqlSyntax = */ false
+);
+```
+
+| Flag | Effect |
+|------|--------|
+| `noBackslashEscapes` | When `true`, `\` inside string literals is a literal
backslash rather than an escape character. Matches MySQL's
`NO_BACKSLASH_ESCAPES` sql_mode. |
+| `ansiSqlSyntax` | When `true`, enables ANSI SQL behavior in a small number
of grammar rules (mainly around `GROUP BY` / `ORDER BY` resolution). Matches
the `enable_ansi_query_organization_behavior` Doris session variable. |
+
+## Performance Notes
+
+### Origin tracking fast path
+
+`ParserUtils.withOrigin` pushes the current ANTLR rule's line/column onto a
per-thread stack so that `ParseException` can report the exact source location
of any error raised during tree construction. By default this uses a
`ThreadLocal`; threads that run the parser on a hot path can opt into a faster
field-based storage by implementing
`org.apache.doris.nereids.parser.OriginAware`:
+
+```java
+public class MyParserThread extends Thread implements OriginAware {
+ private Origin origin;
+ @Override public Origin getOrigin() { return origin; }
+ @Override public void setOrigin(Origin o) { this.origin = o; }
+}
+```
+
+Any thread that does not implement `OriginAware` falls back to the
`ThreadLocal` path. Correctness is identical either way; the fast path saves
one `ThreadLocal` hash lookup per `withOrigin` call.
+
+### Thread safety
+
+`DorisSqlParser` is stateless aside from its constructor flags and can be
reused as a shared singleton across threads. Each parse call constructs a fresh
`Lexer`, `TokenStream`, and `Parser` internally.
+
+## Caveats
+
+- The grammar covers the full Doris SQL surface (DDL + DML + administrative
commands). If you only care about `SELECT`, you still parse with the full
parser and just visit the relevant subtree.
+- No semantic analysis: identifiers like `t`, `a`, `u.id` come back as
syntactic tokens. Resolving them against a catalog requires additional logic in
your application.
+- `antlr4-runtime:4.13.1` is a transitive dependency of the thin jar. Align
with this version in your project or you will hit `NoSuchMethodError` at
runtime.
+- The CLI jar bundles `antlr4-runtime` so it has no classpath conflicts when
run with `java -jar`.
+- The module is not yet published to Maven Central. Until it is, consumers
need to install it locally with `mvn install -Pflatten` or pull it from an
internal repository.
diff --git a/fe/fe-sql-parser/pom.xml b/fe/fe-sql-parser/pom.xml
new file mode 100644
index 00000000000..50a50c6c0aa
--- /dev/null
+++ b/fe/fe-sql-parser/pom.xml
@@ -0,0 +1,165 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xmlns="http://maven.apache.org/POM/4.0.0";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.doris</groupId>
+ <version>${revision}</version>
+ <artifactId>fe</artifactId>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
+ <artifactId>fe-sql-parser</artifactId>
+ <packaging>jar</packaging>
+ <name>Doris FE SQL Parser</name>
+ <description>Standalone ANTLR4-based SQL syntax parser for Apache Doris.
Produces a parse tree (CST) without
+ any semantic analysis. Designed to be consumed as an independent jar
by external tools.</description>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.antlr</groupId>
+ <artifactId>antlr4-runtime</artifactId>
+ <version>${antlr4.version}</version>
+ </dependency>
+ <!-- Test dependencies -->
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <finalName>doris-fe-sql-parser</finalName>
+ <directory>${project.basedir}/target/</directory>
+ <plugins>
+ <!-- antlr -->
+ <plugin>
+ <groupId>org.antlr</groupId>
+ <artifactId>antlr4-maven-plugin</artifactId>
+ <version>${antlr4.version}</version>
+ <executions>
+ <execution>
+ <id>antlr</id>
+ <goals>
+ <goal>antlr4</goal>
+ </goals>
+ </execution>
+ </executions>
+ <configuration>
+ <visitor>true</visitor>
+ <sourceDirectory>src/main/antlr4</sourceDirectory>
+ <treatWarningsAsErrors>true</treatWarningsAsErrors>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>prepare-test-jar</id>
+ <phase>test-compile</phase>
+ <goals>
+ <goal>test-jar</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-javadoc-plugin</artifactId>
+ <configuration>
+ <skip>true</skip>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+ <profiles>
+ <profile>
+ <!-- Build a self-contained executable jar with antlr4-runtime
bundled and a
+ Main-Class manifest. Produces
target/doris-fe-sql-parser-cli.jar.
+ Usage: java -jar target/doris-fe-sql-parser-cli.jar "SELECT
1" -->
+ <id>cli</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>build-cli-jar</id>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+
<shadedArtifactAttached>true</shadedArtifactAttached>
+
<shadedClassifierName>cli</shadedClassifierName>
+
<createDependencyReducedPom>false</createDependencyReducedPom>
+ <minimizeJar>true</minimizeJar>
+ <transformers>
+ <transformer
+
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+
<mainClass>org.apache.doris.sqlparser.DorisSqlParserCli</mainClass>
+ </transformer>
+ </transformers>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+
<exclude>META-INF/*.SF</exclude>
+
<exclude>META-INF/*.DSA</exclude>
+
<exclude>META-INF/*.RSA</exclude>
+
<exclude>module-info.class</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ <profile>
+ <id>release</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-source-plugin</artifactId>
+ <configuration>
+ <attach>true</attach>
+ </configuration>
+ <executions>
+ <execution>
+ <id>create-source-jar</id>
+ <goals>
+ <goal>jar-no-fork</goal>
+ <goal>test-jar-no-fork</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+</project>
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
similarity index 100%
rename from fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
rename to
fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
similarity index 100%
rename from fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
rename to
fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/errors/QueryParsingErrors.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/errors/QueryParsingErrors.java
similarity index 100%
rename from
fe/fe-core/src/main/java/org/apache/doris/nereids/errors/QueryParsingErrors.java
rename to
fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/errors/QueryParsingErrors.java
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/AnalysisException.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/exceptions/AnalysisException.java
similarity index 84%
rename from
fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/AnalysisException.java
rename to
fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/exceptions/AnalysisException.java
index e8b5d142f8f..fe01fd22f77 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/AnalysisException.java
+++
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/exceptions/AnalysisException.java
@@ -17,8 +17,6 @@
package org.apache.doris.nereids.exceptions;
-import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
-
import java.util.Optional;
/** Nereids's AnalysisException. */
@@ -27,36 +25,33 @@ public class AnalysisException extends RuntimeException {
private final String message;
private final Optional<Integer> line;
private final Optional<Integer> startPosition;
- private final Optional<LogicalPlan> plan;
/** Constructor of AnalysisException. */
public AnalysisException(ErrorCode errorCode, String message, Throwable
cause, Optional<Integer> line,
- Optional<Integer> startPosition, Optional<LogicalPlan> plan) {
+ Optional<Integer> startPosition) {
super(message, cause);
this.errorCode = errorCode;
this.message = message;
this.line = line;
this.startPosition = startPosition;
- this.plan = plan;
}
/** Constructor of AnalysisException. */
public AnalysisException(ErrorCode errorCode, String message,
Optional<Integer> line,
- Optional<Integer> startPosition, Optional<LogicalPlan> plan) {
+ Optional<Integer> startPosition) {
super(message);
this.errorCode = errorCode;
this.message = message;
this.line = line;
this.startPosition = startPosition;
- this.plan = plan;
}
public AnalysisException(ErrorCode errorCode, String message, Throwable
cause) {
- this(errorCode, message, cause, Optional.empty(), Optional.empty(),
Optional.empty());
+ this(errorCode, message, cause, Optional.empty(), Optional.empty());
}
public AnalysisException(ErrorCode errorCode, String message) {
- this(errorCode, message, Optional.empty(), Optional.empty(),
Optional.empty());
+ this(errorCode, message, Optional.empty(), Optional.empty());
}
public AnalysisException(String message, Throwable cause) {
@@ -69,11 +64,6 @@ public class AnalysisException extends RuntimeException {
@Override
public String getMessage() {
- String planAnnotation = plan.map(p -> ";\n" +
p.treeString()).orElse("");
- return getSimpleMessage() + planAnnotation;
- }
-
- private String getSimpleMessage() {
if (line.isPresent() || startPosition.isPresent()) {
String lineAnnotation = line.map(l -> "line " + l).orElse("");
String positionAnnotation = startPosition.map(s -> " pos " +
s).orElse("");
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/ParseException.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/exceptions/ParseException.java
similarity index 99%
rename from
fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/ParseException.java
rename to
fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/exceptions/ParseException.java
index e3c72f1d6e7..a3af0c38164 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/ParseException.java
+++
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/exceptions/ParseException.java
@@ -37,7 +37,7 @@ public class ParseException extends AnalysisException {
}
public ParseException(String message, Origin start, Optional<String>
command) {
- super(ErrorCode.NONE, message, start.line, start.startPosition,
Optional.empty());
+ super(ErrorCode.NONE, message, start.line, start.startPosition);
this.message = message;
this.start = start;
this.command = command;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/SyntaxParseException.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/exceptions/SyntaxParseException.java
similarity index 100%
rename from
fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/SyntaxParseException.java
rename to
fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/exceptions/SyntaxParseException.java
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/CaseInsensitiveStream.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/CaseInsensitiveStream.java
similarity index 100%
rename from
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/CaseInsensitiveStream.java
rename to
fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/CaseInsensitiveStream.java
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/Origin.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/Origin.java
similarity index 100%
rename from fe/fe-core/src/main/java/org/apache/doris/nereids/parser/Origin.java
rename to
fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/Origin.java
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/CastException.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/OriginAware.java
similarity index 56%
copy from
fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/CastException.java
copy to
fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/OriginAware.java
index 5d3ac70206d..471ae72acad 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/exceptions/CastException.java
+++
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/OriginAware.java
@@ -15,28 +15,20 @@
// specific language governing permissions and limitations
// under the License.
-package org.apache.doris.nereids.exceptions;
-
-import java.util.Optional;
+package org.apache.doris.nereids.parser;
/**
- * cast exception.
+ * Optional fast-path SPI for the parser's "current Origin" stack. A Thread
+ * subclass that runs the parser hot-path may implement this so {@link
ParserUtils}
+ * can store and load the Origin in a plain field instead of paying a
ThreadLocal
+ * lookup on every {@code withOrigin} call. Threads that don't implement this
+ * fall back to a ThreadLocal — correctness is identical either way.
+ *
+ * Implementations must only be called by the owning thread; no synchronization
+ * is required or expected.
*/
-public class CastException extends AnalysisException {
-
- private final String message;
-
- public CastException(String message) {
- super(ErrorCode.NONE, message, Optional.of(0), Optional.of(0),
Optional.empty());
- this.message = message;
- }
-
- public CastException(String message, Throwable t) {
- super(message, t);
- this.message = message;
- }
+public interface OriginAware {
+ Origin getOrigin();
- public String getMessage() {
- return message;
- }
+ void setOrigin(Origin origin);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/ParseErrorListener.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/ParseErrorListener.java
similarity index 100%
rename from
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/ParseErrorListener.java
rename to
fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/ParseErrorListener.java
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/ParserUtils.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/ParserUtils.java
similarity index 77%
rename from
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/ParserUtils.java
rename to
fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/ParserUtils.java
index b36a5eb4729..0b3cc03c8cb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/ParserUtils.java
+++
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/ParserUtils.java
@@ -17,8 +17,6 @@
package org.apache.doris.nereids.parser;
-import org.apache.doris.nereids.util.MoreFieldsThread;
-
import org.antlr.v4.runtime.CharStream;
import org.antlr.v4.runtime.ParserRuleContext;
import org.antlr.v4.runtime.Token;
@@ -31,18 +29,18 @@ import java.util.function.Supplier;
* Utils for parser.
*/
public class ParserUtils {
- private static final ThreadLocal<Origin> slowThreadLocal = new
ThreadLocal<>();
+ private static final ThreadLocal<Origin> threadLocal = new ThreadLocal<>();
/** getOrigin */
public static Optional<Origin> getOrigin() {
Thread thread = Thread.currentThread();
Origin origin;
- if (thread instanceof MoreFieldsThread) {
+ if (thread instanceof OriginAware) {
// fast path
- origin = ((MoreFieldsThread) thread).getOrigin();
+ origin = ((OriginAware) thread).getOrigin();
} else {
// slow path
- origin = slowThreadLocal.get();
+ origin = threadLocal.get();
}
return Optional.ofNullable(origin);
}
@@ -56,27 +54,27 @@ public class ParserUtils {
);
Thread thread = Thread.currentThread();
- if (thread instanceof MoreFieldsThread) {
+ if (thread instanceof OriginAware) {
// fast path
- MoreFieldsThread moreFieldsThread = (MoreFieldsThread) thread;
- Origin outerOrigin = moreFieldsThread.getOrigin();
+ OriginAware aware = (OriginAware) thread;
+ Origin outerOrigin = aware.getOrigin();
try {
- moreFieldsThread.setOrigin(origin);
+ aware.setOrigin(origin);
return f.get();
} finally {
- moreFieldsThread.setOrigin(outerOrigin);
+ aware.setOrigin(outerOrigin);
}
} else {
// slow path
- Origin outerOrigin = slowThreadLocal.get();
+ Origin outerOrigin = threadLocal.get();
try {
- slowThreadLocal.set(origin);
+ threadLocal.set(origin);
return f.get();
} finally {
if (outerOrigin != null) {
- slowThreadLocal.set(outerOrigin);
+ threadLocal.set(outerOrigin);
} else {
- slowThreadLocal.remove();
+ threadLocal.remove();
}
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/PostProcessor.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/PostProcessor.java
similarity index 100%
rename from
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/PostProcessor.java
rename to
fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/PostProcessor.java
diff --git
a/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParser.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParser.java
new file mode 100644
index 00000000000..e006228cf3f
--- /dev/null
+++
b/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParser.java
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser;
+
+import org.apache.doris.nereids.DorisLexer;
+import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.nereids.DorisParser.ExpressionContext;
+import org.apache.doris.nereids.DorisParser.ExpressionWithEofContext;
+import org.apache.doris.nereids.DorisParser.MultiStatementsContext;
+import org.apache.doris.nereids.DorisParser.SingleStatementContext;
+import org.apache.doris.nereids.parser.CaseInsensitiveStream;
+import org.apache.doris.nereids.parser.ParseErrorListener;
+import org.apache.doris.nereids.parser.PostProcessor;
+
+import org.antlr.v4.runtime.CharStreams;
+import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.ParserRuleContext;
+import org.antlr.v4.runtime.atn.PredictionMode;
+import org.antlr.v4.runtime.misc.ParseCancellationException;
+
+import java.util.function.Function;
+
+/**
+ * Standalone facade for Doris SQL syntax parsing. Produces an ANTLR parse tree
+ * (CST) without any semantic analysis. Thread-safe (stateless).
+ */
+public final class DorisSqlParser {
+ private static final ParseErrorListener PARSE_ERROR_LISTENER = new
ParseErrorListener();
+ private static final PostProcessor POST_PROCESSOR = new PostProcessor();
+
+ private final boolean noBackslashEscapes;
+ private final boolean ansiSqlSyntax;
+
+ public DorisSqlParser() {
+ this(false, false);
+ }
+
+ /**
+ * @param noBackslashEscapes maps to MySQL's NO_BACKSLASH_ESCAPES sql
mode; controls how
+ * the lexer treats backslashes inside string
literals.
+ * @param ansiSqlSyntax enables ANSI behavior in a few corners of the
grammar
+ * (matches
GlobalVariable.enable_ansi_query_organization_behavior).
+ */
+ public DorisSqlParser(boolean noBackslashEscapes, boolean ansiSqlSyntax) {
+ this.noBackslashEscapes = noBackslashEscapes;
+ this.ansiSqlSyntax = ansiSqlSyntax;
+ }
+
+ /** Parse a single SQL statement. */
+ public SingleStatementContext parseStatement(String sql) {
+ return (SingleStatementContext) toAst(sql,
DorisParser::singleStatement);
+ }
+
+ /** Parse one or more SQL statements separated by semicolons. */
+ public MultiStatementsContext parseStatements(String sql) {
+ return (MultiStatementsContext) toAst(sql,
DorisParser::multiStatements);
+ }
+
+ /** Parse a single SQL expression (no trailing tokens allowed). */
+ public ExpressionContext parseExpression(String sql) {
+ ExpressionWithEofContext ctx = (ExpressionWithEofContext) toAst(sql,
DorisParser::expressionWithEof);
+ return ctx.expression();
+ }
+
+ /** Build a freshly configured lexer for advanced callers that want to
walk tokens directly. */
+ public DorisLexer newLexer(String sql) {
+ DorisLexer lexer = new DorisLexer(new
CaseInsensitiveStream(CharStreams.fromString(sql)));
+ lexer.isNoBackslashEscapes = noBackslashEscapes;
+ return lexer;
+ }
+
+ /** Build a freshly configured parser bound to {@code lexer}. Caller owns
the returned parser. */
+ public DorisParser newParser(DorisLexer lexer) {
+ CommonTokenStream tokenStream = new CommonTokenStream(lexer);
+ tokenStream.fill();
+ return configure(new DorisParser(tokenStream));
+ }
+
+ private ParserRuleContext toAst(String sql, Function<DorisParser,
ParserRuleContext> parseFunction) {
+ CommonTokenStream tokenStream = tokenize(sql);
+ DorisParser parser = configure(new DorisParser(tokenStream));
+ try {
+ // first, try parsing with potentially faster SLL mode
+ parser.getInterpreter().setPredictionMode(PredictionMode.SLL);
+ return parseFunction.apply(parser);
+ } catch (ParseCancellationException ex) {
+ // if we fail, parse with LL mode
+ tokenStream.seek(0);
+ parser.reset();
+ parser.getInterpreter().setPredictionMode(PredictionMode.LL);
+ return parseFunction.apply(parser);
+ }
+ }
+
+ private CommonTokenStream tokenize(String sql) {
+ DorisLexer lexer = newLexer(sql);
+ CommonTokenStream tokenStream = new CommonTokenStream(lexer);
+ tokenStream.fill();
+ return tokenStream;
+ }
+
+ private DorisParser configure(DorisParser parser) {
+ parser.ansiSQLSyntax = ansiSqlSyntax;
+ parser.addParseListener(POST_PROCESSOR);
+ parser.removeErrorListeners();
+ parser.addErrorListener(PARSE_ERROR_LISTENER);
+ return parser;
+ }
+}
diff --git
a/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParserCli.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParserCli.java
new file mode 100644
index 00000000000..c8b39676501
--- /dev/null
+++
b/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParserCli.java
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser;
+
+import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.nereids.exceptions.ParseException;
+
+import org.antlr.v4.runtime.ParserRuleContext;
+import org.antlr.v4.runtime.tree.ParseTree;
+import org.antlr.v4.runtime.tree.TerminalNode;
+import org.antlr.v4.runtime.tree.Trees;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.PrintStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.Arrays;
+
+/**
+ * Command-line entry point for fe-sql-parser. Reads a SQL string from
+ * argument, file, or stdin and prints the ANTLR parse tree (CST).
+ *
+ * Exit codes: 0 success, 1 parse error, 2 usage or I/O error.
+ */
+public final class DorisSqlParserCli {
+
+ private DorisSqlParserCli() {
+ }
+
+ public static void main(String[] args) {
+ Options opts;
+ try {
+ opts = parseArgs(args);
+ } catch (UsageException e) {
+ System.err.println("doris-sql-parse: " + e.getMessage());
+ printUsage(System.err);
+ System.exit(2);
+ return;
+ }
+ if (opts.help) {
+ printUsage(System.out);
+ return;
+ }
+
+ String sql;
+ try {
+ sql = readSql(opts);
+ } catch (IOException e) {
+ System.err.println("doris-sql-parse: failed to read SQL: " +
e.getMessage());
+ System.exit(2);
+ return;
+ }
+ if (sql == null || sql.trim().isEmpty()) {
+ System.err.println("doris-sql-parse: empty SQL input");
+ System.exit(2);
+ return;
+ }
+
+ DorisSqlParser parser = new DorisSqlParser(opts.noBackslashEscapes,
opts.ansiSql);
+ try {
+ ParserRuleContext tree;
+ switch (opts.mode) {
+ case MULTI:
+ tree = parser.parseStatements(sql);
+ break;
+ case EXPRESSION:
+ tree = parser.parseExpression(sql);
+ break;
+ case SINGLE:
+ default:
+ tree = parser.parseStatement(sql);
+ break;
+ }
+ if (opts.pretty) {
+ printTreeIndented(tree, DorisParser.ruleNames, System.out, 0);
+ } else {
+ System.out.println(Trees.toStringTree(tree,
Arrays.asList(DorisParser.ruleNames)));
+ }
+ } catch (ParseException e) {
+ System.err.println(e.getMessage());
+ System.exit(1);
+ }
+ }
+
+ private static void printTreeIndented(ParseTree tree, String[] ruleNames,
PrintStream out, int indent) {
+ StringBuilder pad = new StringBuilder();
+ for (int i = 0; i < indent; i++) {
+ pad.append(" ");
+ }
+ String label;
+ if (tree instanceof ParserRuleContext) {
+ label = ruleNames[((ParserRuleContext) tree).getRuleIndex()];
+ } else if (tree instanceof TerminalNode) {
+ label = "'" + tree.getText() + "'";
+ } else {
+ label = tree.getClass().getSimpleName();
+ }
+ out.println(pad.toString() + label);
+ for (int i = 0; i < tree.getChildCount(); i++) {
+ printTreeIndented(tree.getChild(i), ruleNames, out, indent + 1);
+ }
+ }
+
+ private static String readSql(Options opts) throws IOException {
+ if (opts.execSql != null) {
+ return opts.execSql;
+ }
+ if (opts.file != null) {
+ return new String(Files.readAllBytes(Paths.get(opts.file)),
StandardCharsets.UTF_8);
+ }
+ StringBuilder sb = new StringBuilder();
+ try (BufferedReader br = new BufferedReader(new
InputStreamReader(System.in, StandardCharsets.UTF_8))) {
+ String line;
+ while ((line = br.readLine()) != null) {
+ sb.append(line).append('\n');
+ }
+ }
+ return sb.toString();
+ }
+
+ private enum Mode { SINGLE, MULTI, EXPRESSION }
+
+ private static final class Options {
+ Mode mode = Mode.SINGLE;
+ boolean pretty;
+ boolean noBackslashEscapes;
+ boolean ansiSql;
+ boolean help;
+ String execSql;
+ String file;
+ }
+
+ private static Options parseArgs(String[] args) {
+ Options o = new Options();
+ int i = 0;
+ while (i < args.length) {
+ String a = args[i];
+ switch (a) {
+ case "-h":
+ case "--help":
+ o.help = true;
+ return o;
+ case "-e":
+ case "--exec":
+ if (++i >= args.length) {
+ throw new UsageException("--exec requires a SQL
argument");
+ }
+ o.execSql = args[i];
+ break;
+ case "-f":
+ case "--file":
+ if (++i >= args.length) {
+ throw new UsageException("--file requires a path");
+ }
+ o.file = args[i];
+ break;
+ case "--multi":
+ o.mode = Mode.MULTI;
+ break;
+ case "--expression":
+ o.mode = Mode.EXPRESSION;
+ break;
+ case "--pretty":
+ o.pretty = true;
+ break;
+ case "--no-backslash-escapes":
+ o.noBackslashEscapes = true;
+ break;
+ case "--ansi":
+ o.ansiSql = true;
+ break;
+ default:
+ if (o.execSql == null && o.file == null &&
!a.startsWith("-")) {
+ o.execSql = a;
+ } else {
+ throw new UsageException("unknown argument: " + a);
+ }
+ break;
+ }
+ i++;
+ }
+ return o;
+ }
+
+ private static void printUsage(PrintStream out) {
+ out.println("Usage: doris-sql-parse [OPTIONS] [SQL]");
+ out.println();
+ out.println("Read a Doris SQL string and print its ANTLR parse tree
(CST).");
+ out.println();
+ out.println("Input (mutually exclusive; reads from stdin if none of
these are given):");
+ out.println(" SQL SQL string as a positional
argument");
+ out.println(" -e, --exec <SQL> SQL string");
+ out.println(" -f, --file <PATH> Read SQL from a UTF-8 text
file");
+ out.println();
+ out.println("Parse mode:");
+ out.println(" (default) Parse as a single statement");
+ out.println(" --multi Parse as multiple statements
(separated by ;)");
+ out.println(" --expression Parse as a single
expression");
+ out.println();
+ out.println("Output:");
+ out.println(" (default) ANTLR LISP-style toStringTree
on one line");
+ out.println(" --pretty Indented multi-line tree");
+ out.println();
+ out.println("Dialect flags:");
+ out.println(" --no-backslash-escapes MySQL NO_BACKSLASH_ESCAPES
sql_mode behavior");
+ out.println(" --ansi Enable ANSI SQL syntax
variants");
+ out.println();
+ out.println("Other:");
+ out.println(" -h, --help Show this help");
+ out.println();
+ out.println("Exit codes: 0 success, 1 parse error, 2 usage or I/O
error.");
+ }
+
+ private static final class UsageException extends RuntimeException {
+ UsageException(String msg) {
+ super(msg);
+ }
+ }
+}
diff --git
a/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/DorisSqlParserTest.java
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/DorisSqlParserTest.java
new file mode 100644
index 00000000000..928160e85c4
--- /dev/null
+++
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/DorisSqlParserTest.java
@@ -0,0 +1,75 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser;
+
+import org.apache.doris.nereids.DorisParser.ExpressionContext;
+import org.apache.doris.nereids.DorisParser.MultiStatementsContext;
+import org.apache.doris.nereids.DorisParser.SingleStatementContext;
+import org.apache.doris.nereids.exceptions.ParseException;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+class DorisSqlParserTest {
+ private final DorisSqlParser parser = new DorisSqlParser();
+
+ @Test
+ void parsesSimpleSelect() {
+ SingleStatementContext ctx = parser.parseStatement("SELECT 1");
+ Assertions.assertNotNull(ctx);
+ Assertions.assertNotNull(ctx.statement());
+ }
+
+ @Test
+ void parsesSelectWithFromAndWhere() {
+ SingleStatementContext ctx = parser.parseStatement("SELECT a, b FROM t
WHERE a > 1");
+ Assertions.assertNotNull(ctx);
+ Assertions.assertNotNull(ctx.statement());
+ }
+
+ @Test
+ void parsesMultipleStatements() {
+ MultiStatementsContext ctx = parser.parseStatements("SELECT 1; SELECT
2; SELECT 3");
+ Assertions.assertNotNull(ctx);
+ Assertions.assertEquals(3, ctx.statement().size());
+ }
+
+ @Test
+ void parsesExpression() {
+ ExpressionContext expr = parser.parseExpression("a + 1 * b");
+ Assertions.assertNotNull(expr);
+ }
+
+ @Test
+ void parsesDdl() {
+ SingleStatementContext ctx = parser.parseStatement(
+ "CREATE TABLE t (id INT, name VARCHAR(64)) DISTRIBUTED BY
HASH(id) BUCKETS 4 "
+ + "PROPERTIES (\"replication_num\" = \"1\")");
+ Assertions.assertNotNull(ctx);
+ }
+
+ @Test
+ void rejectsMalformedSql() {
+ Assertions.assertThrows(ParseException.class, () ->
parser.parseStatement("SELEKT 1"));
+ }
+
+ @Test
+ void rejectsTrailingGarbageInExpression() {
+ Assertions.assertThrows(ParseException.class, () ->
parser.parseExpression("1 + 2 BAD GARBAGE"));
+ }
+}
diff --git a/fe/pom.xml b/fe/pom.xml
index af8947325cc..c5e88ad3d5a 100644
--- a/fe/pom.xml
+++ b/fe/pom.xml
@@ -224,6 +224,7 @@ under the License.
<module>fe-extension-spi</module>
<module>fe-connector</module>
<module>fe-extension-loader</module>
+ <module>fe-sql-parser</module>
<module>fe-core</module>
<module>hive-udf</module>
<module>be-java-extensions</module>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
