tobixdev commented on issue #19795: URL: https://github.com/apache/datafusion/issues/19795#issuecomment-3749776429
In our [SPARQL query engine](https://github.com/tobixdev/rdf-fusion) we also often encounter some queries where the planning time is quite large compared to execution time. We have deeply netsted joins in largery graph patterns and large union types. The low hanging fruits would be looking into which passes of the DF optimizer are actually needed and it's been on my TODO list for quite some time. But improving the base speed on the optimizations will surely help. Testing them in DataFusion directly will be complicated as we're using a lot of custom operators (most of them get mapped to a DF operator immediately, but still the parser relies on that). I am happy to run further tests and all benchmarks are available on [Codspeed](https://codspeed.io/tobixdev/rdf-fusion/benchmarks). If I can find the time I would also like to help out but realistically this won't be possible till End of February / March. If I can help wth something feel free to ping me. Here is one example (the dataset is very small in this case and in-memory and this is an extreme case). <img width="1499" height="700" alt="Image" src="https://github.com/user-attachments/assets/67270986-8564-4a93-9b19-38a3ccb1f6ae"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
