LucaCappelletti94 commented on issue #2893: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/2893#issuecomment-6747749176
Once the current version has a richer AST and is less buggy, we certainly can: 1. Use allocators to measure precisely the number of allocations needed in a benchmark corpus and gradually minimize them 2. Include compiled-size as a quality metric 3. Include compile time as a quality metric 4. While copy-less is likely not possible due to dialect needs (I have already slammed against this in my previous attempts at making the iterator a copy-less `Lender`) and slices/`&str` are not viable, possibly as I said before we can go for `Cow` + `&str` for the subset of dialects where no allocation-requiring syntax is needed 5. Maybe allocation arenas are a better imperfect-world solution than strictly aiming for copt-less 6. The existing tests are often tautological and can be relied upon only with caution, as I found already several instances of the tests pinning the wrong behavior. The differential fuzzers are a much better ground truth than the test suite, but we need (ideally) to make one per-dialect. For instance @s5dsn-eqee is [adding one for Clickhouse here](https://github.com/apache/datafusion-sqlparser-rs/pull/2646). @andygrove I suggest you: 1. Identify the various points where the current tokenizer & parser allocates a new string to handle dialect oddities, and determine whether they can be partitioned within that dialect, a particular subset of statements, or whether it is actually possible to remove them entirely. 2. If copy-less is possible, I suggest you go for https://github.com/WanderLanz/Lender so we can have the widest applicability, including in cases where the text is streamed. 3. In an ideal scenario, we could make a compile-time composable AST which depending on which statements are expected, using head-tail variadics (e.g. https://github.com/connetto-rs/tuplities https://github.com/connetto-rs/diesel-builders), has different bounds and guarantees, maybe built with a derive on top of a user-defined enum of statements. Basically, pay-by-AST, regardless of which is the specific dialect, and would come with quite strong compile-time guarantees. Such a solution would, for instance, stop datafusion from having to deal with the superset of statements that it does not use and reduce the size of the statements enum considerably. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
