edmondop opened a new issue, #2612:
URL: https://github.com/apache/datafusion-sqlparser-rs/issues/2612
Follow-up to the discussion in #2610.
**Problem**
The parser lets a dialect take over at specific points (`parse_prefix`,
`parse_infix`, `parse_statement`): it receives `&mut Parser`, and returning
`None` falls back to the default. The tokenizer has no equivalent. A dialect
can only answer yes/no questions the tokenizer asks (`is_identifier_part`,
`supports_triple_quoted_string`, and about 20 others), so it can toggle
existing lexemes but not add new ones.
Dialect-specific lexing therefore lives in `Tokenizer::next_token` behind
`dialect_of!` checks:
- `b'...'` byte strings (BigQuery, MySQL) and bit strings (Postgres)
- `r'...'` raw strings (BigQuery)
- `//` as a line comment (Snowflake) or integer division (DuckDB)
- `#` as a line comment (Snowflake, BigQuery, MySQL, Hive)
- `\r` ending a line comment (Postgres)
A downstream dialect cannot add a branch like these. Its only option is
`tokenize_with_location_into_buf_with_mapper`, which rewrites one finished
token at a time and cannot see characters or merge tokens.
Example: in dbt models, a Jinja comment `{# ... #}` breaks under every
dialect that treats `#` as a line comment, because the tokenizer consumes the
rest of the line, including `#}`.
**Proposal**
1. Expose the character stream that `next_token` reads from (today the
private `State`: `peek`, `next`, `location`), or a narrow public wrapper around
it.
2. Add a hook with the same contract as the parser hooks:
```rust
/// Dialect-specific tokenizer override, called before the default rules.
///
/// If `None` is returned, falls back to the default behavior.
fn next_token(
&self,
_chars: &mut State,
_prev_token: Option<&Token>,
) -> Option<Result<Token, TokenizerError>> {
None
}
```
3. As a separate step, move the existing `dialect_of!` branches into the
hook implementations of their dialects. That would show the hook covers syntax
already in the tree, and would remove `dialect_of!` from the tokenizer.
**Open questions**
- Lookahead: a hook that returns `None` must not consume input. `State`
wraps `Peekable<Chars>` and allows one character of lookahead. Should `State`
be `Clone`, or offer `peek_nth`?
- `Token` is a closed enum, so a dialect-specific lexeme must map to an
existing variant (`Word`, `CustomBinaryOperator`, ...). Is that enough, or
should there be a `Token::Custom` variant?
- Cost: one extra dynamic call per token. The tokenizer already calls
several dialect methods per token, and `sqlparser_bench` can measure the
difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
