LucaCappelletti94 commented on issue #2893: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/2893#issuecomment-6752884942
The issue is not "pretty much all the features", the issue is exactly the few cursed corner cases that do not allow for it. That is why in [my list above](https://github.com/apache/datafusion-sqlparser-rs/issues/2893#issuecomment-6747749176) I put in the first point: > Identify the various points where the current tokenizer & parser allocates a new string to handle dialect oddities, and determine whether they can be partitioned within that dialect, a particular subset of statements, or whether it is actually possible to remove them entirely. I spent more time reasoning how to build this in a way to have the user spend and loose performance exactly by how much AST they desire to cover and handle, so that if they need the whole thing they can, and it should compose reasonably modularly. This would allow for instance datafusion to discard all statements that are outside its scope, such as all of the RLS stuff for instance. A possible "have your cake and eat it too", could be to create a trait like (drafted here just as a concept, but basically just head-tail variadics and a trait generality ordering over structs): ``` trait LeastGeneralCommonVariant<T> { type Variant; } /// The empty set struct Empty; impl<Other> LeastGeneralCommonVariant<Other> for Empty { type Variant = Other; } ``` and then you can define generalization rankings such as: ``` impl<'a> LeastGeneralCommonVariant<&'a str> for &'a str { type Variant = &'a str; } impl<'a, G> LeastGeneralCommonVariant<Cow<'a, G>> for G { type Variant = Cow<'a, G>; } // Of course these traits relations will grow quadratically (specialization does not exist yet sadly) // with the number of relations in that group but I do not expect we would make an // immense amount thereof, and should likely be feature-gated, like // in diesel the number of columns that a table can have for the same reason. ``` or in the case of tokenizers: ``` trait Tokenizer { type Error; type Token; type Tokens: Iterator<Item=Result<Self::Token, Self::Error>>; fn tokenize(&mut self, stream: &str) -> Result<Self::Tokens, Self::Error>; } struct TokenizerAbleToTokenizeDialectSubset {} struct TokenizerAbleToTokenizeWholeDialect { subset_tokenizer: TokenizerAbleToTokenizeDialectSubset, other_attributes: ... } impl LeastGeneralCommonVariant<TokenizerAbleToTokenizeWholeDialect> for TokenizerAbleToTokenizeDialectSubset { type Variant = TokenizerAbleToTokenizeWholeDialect; } impl LeastGeneralCommonVariant<TokenizerAbleToTokenizeDialectSubset> for TokenizerAbleToTokenizeWholeDialect { type Variant = TokenizerAbleToTokenizeWholeDialect; } ``` We could even have a generalization order for dialects if we want to really categorize well in which dialect a statement exists: ``` struct SQLStandard; struct PostgreSQL; struct MySQL; struct GeneralDialect; trait Dialect { // Not sure yet what are the properties of the dialect, I think most of this // stuff would live in the statement impl now. } impl LeastGeneralCommonVariant<SQLStandard> for PostgreSQL { type Variant = PostgreSQL; } // When comparing two incompatible dialects, we fallback to the general dialect impl LeastGeneralCommonVariant<MySQL> for PostgreSQL { type Variant = GeneralDialect; } impl LeastGeneralCommonVariant<GeneralDialect> for PostgreSQL { type Variant = GeneralDialect; } // and so on ``` Then, we could have a `StatementBuilder` trait with the properties to build statements: ``` trait Statement { type Dialect; type Builder; } /// A builder for statements trait StatementBuilder { type Tokenizer; type Statement; type Error; fn digest(tokens: &mut <Self::Tokenizer as Tokenizer>::Tokens) -> Result<Self::Statement, Self::Error>; } // Then there needs to be a builder able to build that statement // or an enum of statements, with the fallback most general one // which functions as the worst case scenario, unless a less general // builder is made available. ``` Now, given a `(Head, Tail)` nested n-uple, we can efficiently define: ``` pub trait LeastGeneralCommonRequiremets { type Dialect: Dialect; type Builder: Builder; type Tokenizer: Tokenizer; } impl<Head, Tail> LeastGeneralCommonRequiremets for (Head, Tail) where Head: Statement, Tail: LeastGeneralCommonRequiremets { type Dialect = <<Head as Statement>::Dialect as LeastGeneralCommonVariant<<Tail as LeastGeneralCommonRequiremets>::Dialect>::Variant; // Same for builder, tokenizer } impl<Head> LeastGeneralCommonRequiremets for (Head,) { type Dialect = <Head as Statement>::Dialect; // samesies } impl LeastGeneralCommonRequiremets for () { type Dialect = Empty; // samesies } ``` Then the user given a set of statements that we (or they) provide, creates an enum, uses a derive to build the necessary trait impls, and should get the least computationally expensive of the variants available. ``` struct MyStatementA<Ident> { /// All of the attrs of the statement A } struct MyStatementB<Ident> { /// All of the attrs of the statement B } #[derive(Statements)] /// An enum of the bare essential statements I need, /// where the derive implements Statements for this type, /// such as creating the minimal enum of errors that applies /// for this reduced set of statements, implementing HasErrorEnum /// for it. The derive primarely just needs to construct the (Head, Tail) /// nested n-uple of the statements, so the implementation of the compile /// time most general dialects, builders, tokenizers can be defined once /// with the blanket defined above. enum MyStatements<Ident = String> { StatementA(MyStatementA<Ident>), StatementB(MyStatementB<Ident>), ... } ``` What do you think of the general concept? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
