andygrove opened a new issue, #2910: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/2910
This issue proposes a design for how dialects work in sqlparser v2. It follows on from the v2 proposal in #2893 and covers one part of it: how the parser is generic over a dialect, and how dialects override and reuse grammar rules. This is not just a sketch. There is a working implementation of the whole design: - **Parser:** [andygrove/datafusion-sqlparser-rs@`6f8e464`](https://github.com/andygrove/datafusion-sqlparser-rs/tree/6f8e464b133487405dcd3b8779429b70377ef1b0) (branch `v2-proposal`). It implements all 14 built-in dialects other than ANSI and Generic. It accepts 99.1% of the statements that the current crate's tests write for a specific dialect, in that dialect. - **Tested with DataFusion:** apache/datafusion#27304 ports DataFusion's whole SQL frontend to it: `DFParser`, the SQL planner, the unparser and DataFusion's own SQL dialect. All of DataFusion's tests pass, including all 546 sqllogictest files. The PR is a draft and is not meant to be merged. It shows that the design works for the crate's largest user. Feedback on the design is welcome, especially from people who maintain a dialect or a custom dialect downstream. ## The problem Today, dialect logic lives in the core parser. A dialect can only step in at a few entry points (`parse_statement`, `parse_prefix`, `parse_infix`, ...). So any difference in the middle of a statement becomes a `dialect_of!` check or a `supports_*` flag inside a parser of more than 23,000 lines. Every dialect shares one grammar, so syntax leaks between dialects, and reserved words from one dialect affect all the others. ## 1. The parser is generic over the dialect `Parser<'a, D: Dialect>` is generic over a dialect type rather than holding a `&dyn Dialect`: ```rust let statements = parse_sql::<Postgres>("SELECT x::int FROM t")?; ``` Every grammar rule is a method on the `Dialect` trait. Each method's default implementation is the core rule, which is a free function in `sqlparser::grammar`: ```rust pub trait Dialect: 'static + Sized { type Ext: AstExt; fn parse_statement<'a>(p: &mut Parser<'a, Self>) -> PResult<Statement<'a, Self::Ext>> { grammar::parse_statement(p) } fn parse_join<'a>(p: &mut Parser<'a, Self>) -> PResult<Option<Join<'a, Self::Ext>>> { grammar::parse_join(p) } fn parse_limit<'a>(p: &mut Parser<'a, Self>) -> PResult<Limit<'a, Self::Ext>> { grammar::parse_limit(p) } // ... about 150 hooks in all, from lexing (`lex_token`, `is_reserved`) // through expressions (`parse_prefix`, `infix_rule`) to statements. } ``` The core grammar has no knowledge of any specific dialect. `Core` is simply `impl Dialect for Core { type Ext = Builtin; }`, with every rule left as the default. Because the dialect is a type parameter, every call to a hook is resolved at compile time and can be inlined. The core parser has no `dialect_of!` checks and no runtime flag lookups. For code that only learns the dialect at runtime, `DialectKind` and `builtin::parse_sql_as(kind, sql)` dispatch once at the top to the right monomorphized parser. ## 2. Rules call each other through `D::`, so an override applies everywhere This is the key rule of the design. A core rule never calls another core rule directly. It always goes through the dialect, either as `D::rule(p)` or as a `Parser` method that forwards to `D::`: ```rust pub fn parse_statement<'a, D: Dialect>(p: &mut Parser<'a, D>) -> PResult<Statement<'a, D::Ext>> { if p.at_keyword("INSERT") { return D::parse_insert_statement(p); } if p.at_keyword("CREATE") { return D::parse_create(p); } // ... } impl<'a, D: Dialect> Parser<'a, D> { pub fn parse_table_ref(&mut self) -> PResult<TableRef<'a, D::Ext>> { spanned!(self, D::parse_table_ref(self)) } } ``` So when MySQL overrides `parse_limit` to handle `LIMIT 5, 10`, the override applies in every place a `LIMIT` can appear: top-level queries, subqueries, CTEs and set operations. None of those call sites need to change. A dialect overrides exactly the rules that differ and inherits the rest: ```rust impl Dialect for MySql { type Ext = Builtin; fn parse_limit<'a>(p: &mut Parser<'a, Self>) -> PResult<Limit<'a, Builtin>> { // Parses `LIMIT count` and MySQL's `LIMIT offset, count`. } } ``` An override usually handles its own syntax first and then falls back to the core rule by calling `grammar::rule(p)`, much like calling `super` in an object-oriented language. ## 3. Dialects build on each other by delegating to generic rules Rust has no implementation inheritance between types, so a dialect cannot "extend" another dialect type. Instead, each dialect exports its rules as **public functions that are generic over any dialect**, and other dialects delegate to them hook by hook: ```rust // In spark.rs: Spark's join rule, usable by any dialect. pub fn parse_join<'a, D: Dialect>(p: &mut Parser<'a, D>) -> PResult<Option<Join<'a, D::Ext>>> { let Some(kind) = eat_semi_or_anti(p)? else { return grammar::parse_join(p); // Fall back to the core }; let table = p.parse_table_ref()?; // Dispatches to D::parse_table_ref let constraint = grammar::parse_join_constraint(p)?; Ok(Some(Join { kind, spelled_out: false, table, constraint })) } // In databricks.rs impl Dialect for Databricks { type Ext = Builtin; fn parse_join<'a>(p: &mut Parser<'a, Self>) -> PResult<Option<Join<'a, Builtin>>> { spark::parse_join(p) } fn parse_qualify<'a>(p: &mut Parser<'a, Self>) -> PResult<Option<Expr<'a, Builtin>>> { snowflake::parse_qualify(p) } fn is_reserved(word: &str) -> bool { !word.eq_ignore_ascii_case("ALL") && grammar::is_reserved(word) } // ... } ``` ### How the "inheritance" works When Databricks calls `spark::parse_join`, the call is monomorphized as `spark::parse_join::<Databricks>`. Inside Spark's rule, every nested call (`p.parse_table_ref()`, `D::parse_expr`, ...) dispatches to **Databricks'** hooks, not Spark's. This is the same "late binding" that virtual methods on `self` give in an object-oriented language. A borrowed rule is not a frozen copy of the other dialect's behavior. It runs in the borrowing dialect's grammar. For example, Spark's join rule parses the joined table using Databricks' table factor rule, along with any Databricks-only syntax that rule accepts. So the model is: | Object-oriented idea | v2 equivalent | | ---------------------------- | -------------------------------------------------------------- | | Base class | The core grammar (`grammar::*`), the defaults of `Dialect` | | Overriding a virtual method | Implementing a `Dialect` hook | | Calling `super` | Calling `grammar::rule(p)` from an override | | Calling `self.method()` | `D::rule(p)` / `p.parse_*()`, which reach the running dialect | | Inheriting from a subclass | Forwarding a hook to `other_dialect::rule(p)` | There are two differences from class inheritance, and they are deliberate: - **Inheritance is per rule, not per dialect.** A dialect is not tied to one parent. Databricks takes most of its rules from Spark, `QUALIFY` from Snowflake, and others from BigQuery and Oracle. Redshift is built on PostgreSQL. A dialect takes exactly the rules it wants from whichever dialects have them, as mixins. - **Inheritance is explicit.** A dialect writes one short forwarding method for each rule it borrows. It never picks up another dialect's syntax by accident, so a change to Spark can only affect Databricks through rules that Databricks chose to share. This is what keeps dialects strict: PostgreSQL rejects backtick-quoted names, and only PostgreSQL has `CREATE OPERATOR`. Rules are written so that they can be shared. For example, `spark::parse_join` is generic over any `D: Dialect`. A rule that builds nodes of the shared built-in AST is bounded by `D: Dialect<Ext = Builtin>` instead. ## 4. Operators: Pratt parsing with rules from the dialect Expressions use a Pratt parser. For each token after an operand, the parser asks the dialect's `infix_rule` hook how to handle it. The answer is an `InfixRule` that holds the binding power and the parse action together, so the two cannot drift apart: ```rust fn infix_rule<'a>(p: &Parser<'a, Self>) -> Option<InfixRule<'a, Self>> { if p.at_op("??") { return Some(InfixRule::binary(prec::ADD, BinaryOp::Named("??".into()))); } grammar::infix_rule(p) // The core operators } ``` Binding powers are constants in `grammar::prec`, and a dialect places its own operators relative to them. Keywords and operators are matched by their source text rather than through a central `Keyword` or `Token` enum. So a dialect can add a token (`lex_token`), an operator (`infix_rule`) or a keyword without any change to the core. ## 5. Extending the AST: `AstExt` A dialect chooses its AST through an associated type, `type Ext: AstExt`. `AstExt` has slots for the places a dialect can add nodes: expressions, statements, table factors, join conditions, order-by items, the end of a query, and the data after `SELECT` and `INSERT`: ```rust impl AstExt for Acme { type Statement<'a> = Ping<'a>; // Your own node type Expr<'a> = NoExt; // Slots you do not use // ... } ``` All built-in dialects share one extension type, `Builtin`, so downstream tools handle a single tree whichever built-in dialect parsed it. `AstExt` is separate from `Dialect` for exactly that reason: many dialects can share one AST type. A third-party dialect can reuse `Builtin`, and borrow any built-in rule, or define its own extension type to add nodes. With the `visitor` feature, the new nodes join in through `#[derive(Visit, VisitMut)]`. ## 6. Third-party dialects are equal to built-in ones Built-in dialects use only the public API, so a dialect in another crate can do everything a built-in one can: add tokens, operators, statements and AST nodes, and borrow rules from any built-in dialect. [`examples/custom_dialect.rs`](https://github.com/andygrove/datafusion-sqlparser-rs/blob/6f8e464b133487405dcd3b8779429b70377ef1b0/examples/custom_dialect.rs) is a complete example. DataFusion's port is the real-world test of this. v2 has no `GenericDialect`, so DataFusion defines [its own dialect](https://github.com/andygrove/datafusion/blob/090eb2b0beeb89812ce352b7d91e79d9d37d95ee/datafusion/sql/src/dialect.rs) in its own crate, using only the public API. The dialect is DuckDB's rules plus the syntax DataFusion takes from other dialects: - `LEFT SEMI`/`ANTI` joins, `DISTRIBUTE BY` and `INSERT OVERWRITE`, from Spark - `ASOF JOIN` and `value:key` paths, from Snowflake - `LIMIT offset, count`, `<=>`, `XOR` and backtick-quoted names, from MySQL - `DISTINCT ON` and `RETURNING`, from PostgreSQL - `INSERT OR REPLACE`, from SQLite - a few additions of DataFusion's own, such as `RIGHT SEMI`/`ANTI` joins and its bitwise operator precedence For example, its join rule adds `RIGHT SEMI` and `RIGHT ANTI` joins, hands `ASOF` to `snowflake::parse_join`, and leaves the rest to the rules underneath. Every hook either delegates to a built-in dialect's rule or is one of these additions. A side benefit is that DataFusion's SQL syntax becomes a defined list of rules rather than "whatever `GenericDialect` accepts". DataFusion's own statements (`CREATE EXTERNAL TABLE`, `COPY TO`, ...) are built on the same `Parser` API. ## Trade-offs - **More code is generated.** Each dialect gets its own monomorphized copy of the rules it uses. This is the cost of static dispatch and inlining. - **Borrowing a rule takes some boilerplate.** A dialect writes one forwarding method for each rule it borrows, instead of inheriting all of another dialect's rules at once. This is intended (see above), but it is visible: `databricks.rs` is mostly forwarding methods. - **Choosing where the hooks go.** A dialect can only change the grammar where there is a hook. Too few hooks bring back dialect checks inside rules, and too many make the trait hard to navigate. The current set grew from porting all 14 dialects and DataFusion. Where to draw that line is the main open question. - **No `GenericDialect`.** Strict dialects mean you have to know which database the SQL is for. A downstream crate that wants a permissive dialect can build one by borrowing rules from several dialects, as DataFusion does. ## Questions for discussion 1. Is per-rule delegation the right model for sharing rules between dialects, or would people prefer a way to inherit all of another dialect's rules (for example, a macro that forwards every hook to a parent dialect)? 2. Is the current set of hooks in the right places, and is about 150 hooks the right size for the `Dialect` trait? 3. Are the `AstExt` slots enough for the custom dialects people maintain today? The README and `AGENTS.md` in the branch go into more detail on the rest of the design, including the zero-copy AST and the strict dialects. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
