andygrove opened a new pull request, #27304:
URL: https://github.com/apache/datafusion/pull/27304
## Which issue does this PR close?
- Related to apache/datafusion-sqlparser-rs#2893, the proposal for a v2 of
sqlparser.
This draft is not intended to be merged. It shows what DataFusion would look
like on the proposed v2 parser and tracks what is still missing.
## Rationale for this change
The v2 proposal changes sqlparser substantially: a dialect-free core with
dialects that extend it through hooks, an AST that borrows from the SQL text,
and strict dialects instead of `GenericDialect`. Porting DataFusion is the best
test of whether the new design works for its largest user, and it finds the
gaps in v2 before the design settles.
## What changes are included in this PR?
- **Dependency:** `sqlparser` points at a snapshot of v2 on a fork
(`andygrove/datafusion-sqlparser-rs`, branch `v2-proposal`).
- **A DataFusion SQL dialect** (`datafusion/sql/src/dialect.rs`). v2 has no
`GenericDialect`, so DataFusion gets its own dialect, built only from v2's
public API. It is DuckDB's dialect plus syntax DataFusion uses from other
dialects:
- `FROM VALUES` without parentheses, and `LEFT SEMI` and `LEFT ANTI`
joins, as in Spark
- `RIGHT SEMI` and `RIGHT ANTI` joins
- `ASOF JOIN ... MATCH_CONDITION`, as in Snowflake
- `STRUCT<...>`, `ARRAY<...>` and `MAP<...>` types
- `UNSIGNED` integer types, as in MySQL
The `generic`, `ansi` and `datafusion` dialect names all select it. The
other dialects remain selectable by name. A side benefit is that DataFusion's
SQL syntax is now a defined, documentable list of rules rather than "whatever
`GenericDialect` accepts".
- **`DFParser`:** DataFusion's own statements (`CREATE EXTERNAL TABLE`,
`COPY TO`, `EXPLAIN` options, `RESET`, `DROP CATALOG`, ...) are reimplemented
on v2's parser API. `DFParserBuilder::with_dialect` now takes a dialect value
instead of `&dyn Dialect`.
- **Planner:** the expression, query and statement planners are ported to
v2's AST. AST types now carry a lifetime: `Statement<'a>`, `Expr<'a>` and so on.
- **`datafusion-expr`:** `LogicalPlan` no longer holds sqlparser types. The
wildcard option types always use datafusion-expr's own copies. The planner
extension traits (`ExprPlanner`, `RelationPlanner`, `TypePlanner`) take v2
types. `RawBinaryExpr::op` is an owned `BinaryOp<'static>`.
- **Temporarily disabled:**
- The unparser (the `unparser` feature is off by default).
- Line and column spans in diagnostics: v2's spans are byte offsets, and
converting them needs the SQL text.
- The parser's unit tests, until they are ported.
## What is the testing strategy for this PR?
The existing sqllogictests. On this commit, 476 of 546 test files pass
completely. The remaining 70 files have about 670 failing records between them,
out of roughly 25,000. On the same machine, `main` passes all 546 files, so
every remaining failure comes from the port.
The failures fall into these groups:
| Cause | Approx. records |
|---|---|
| Tests that switch to the Databricks dialect for lambdas, then use bare
`[...]` list literals, which Databricks itself does not accept. The DataFusion
dialect parses these lambdas, so those dialect switches can probably be
dropped. | 63 |
| Tests that match the old parser's error messages | ~77 |
| `CREATE EXTERNAL TABLE hits` failing during setup, so the rest of the file
has no table (mostly `clickbench.slt`) | ~95 |
| Multi-column `IN` with prepared statements | ~40 |
| Syntax v2 does not parse yet: `SHOW a.b.c`, `SET TIME ZONE = ...`,
`TIMESTAMP WITH TIME ZONE '...'`, `0.e0` | ~35 |
| DataFusion dialect rules not added yet: `\|>` pipes, `<=>`, negative
decimal scales, `CREATE FUNCTION ... RETURNS`, `struct(x AS name)`, `ASOF ...
USING` | ~50 |
| Result mismatches and other planner gaps | the rest |
Rust tests outside sqllogictest that use the sqlparser AST directly (in
`datafusion/sql/tests`, `datafusion/core/tests` and `datafusion-examples`) are
not ported yet and do not compile.
## Are there any user-facing changes?
Yes, if this direction were adopted:
- **API changes:** sqlparser's AST is re-exported and used by public APIs.
The changed APIs include `SessionState::sql_to_statement`, `DFParser`,
`SqlToRel::statement_to_plan`, and the planner extension traits, and their
types gain lifetimes.
- **Default dialect:** `generic` now selects the DataFusion dialect. It is
stricter than `GenericDialect`, which accepted most dialects' syntax at once.
- **Error messages:** parse errors have different wording.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
