RPG-Alex opened a new pull request, #2069:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2069
### Refactor: Extract comment variants from `Whitespace` into a dedicated
`Comment` enum
This PR refactors the internal representation of comments in the tokenizer
and parser.
The `SingleLine` and `MultiLine` comment variants previously defined in the
`Whitespace` enum are now represented by a new, dedicated `Comment` enum.
`Whitespace` now references comments via `InterstitialComment(Comment)`,
allowing clear differentiation between `LeadingComment(Comment)` and inline
whitespace comments.
### Rationale
This change improves the semantic clarity of comment handling in the SQL
tokenizer and parser.
As discussed in **#2065**, comments preceding a table or column definition
may serve as **inline documentation**, and should be distinguishable from
interstitial (whitespace) comments.
For example:
```sql
-- Table for storing values for users
CREATE TABLE IF NOT EXISTS users (
id BIGINT PRIMARY KEY,
-- Stores the username as text, required for creating a user account
name TEXT NOT NULL
);
```
By separating comment handling from generic whitespace, the parser can now
support more context-aware interpretations and contribute to a **lossless
syntax tree**, addressing **#175** and complementing **PR #189**.
### Summary of Changes
* **Added:** `Comment` enum encapsulating `SingleLine` and `MultiLine`
comment variants.
* **Refactored:**
* `Whitespace` to include `InterstitialComment(Comment)` variant.
* `Tokenizer.rs` to emit `Comment` values instead of
`Whitespace::SingleLineComment` / `Whitespace::MultiLineComment`.
* `parser/mod.rs` to align with new comment handling.
* **Added logic** for differentiating `InterstitialComment` and
`LeadingComment`.
* **Propagated refactors** to all dependent parser components.
### Impact
* Enables improved handling of comments in a lossless AST context.
* Provides a foundation for interpreting `LeadingComment` nodes as
documentation.
* Simplifies comment handling for future parser extensions and tooling
integration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
