Hi all, after a short discussion I had today with Jialin I took some minutes to see if / how code generation could help us to impove the performance on Queries.
The first thing I looked at was the Filterin in the SeriesReader. Currently many Filters have if conditions / branches in their `satisfy` method. As this method is called on every datapoint (that is not filtered out on another level) there is lot of potential for improvement. I pushed a branch with my code changes (which are not that much, actually), see [1] for all technical details. What did i do? Basically I changed the existing filters to also return an AST of their "filter operation". So I can visit all filters and dsynamically generate the Java Code for the "perfect" Filter in that situation. This is then dynamically compiled in a Java class and live loaded. Of course this process takes some time (about 100 ms or a bit below, I think) but then improves the speed on every data point. What are my results so far? I generated an example file with about 50 Mio. datapoints and three different types of filters (simple, standard, complex). And so far, my results show that the performance can be improved to up to 20%!!! For simple queries its not beneficial and can make it even a bit slower (see overhead above). For standard queries I saw about 5% improvement in query times. For complex queries i saw between 10% and 20% improvement in query time. I will do some more benchmarks the next days and would love tow ork together with other devs from the community to get this feature in the next release! Best! Julian PS.: Happy new year for everybody [1] Branch: https://github.com/apache/iotdb/tree/experimental/code-generation [https://opengraph.githubassets.com/4a5319a07e99ead0a4adb97783f0ba1682e78fa6c074c1a9204c2ea82a2a347b/apache/iotdb]<https://github.com/apache/iotdb/tree/experimental/code-generation> GitHub - apache/iotdb at experimental/code-generation<https://github.com/apache/iotdb/tree/experimental/code-generation> Apache IoTDB. Contribute to apache/iotdb development by creating an account on GitHub. github.com
