Aha, I'm using Apache Kylin which uses Calcite to generate a logical plan, then convert to Spark plan to execute a query. Given that Calcite has more operations for aggregations, and Kylin wants to take full advantage of precomputed cubes (something like Calcite's materialized views), it uses both Calcite and Spark(for distribution computing). Maybe it's wild and a little fun, but it does works well on many scenarios.
On Fri, Aug 4, 2023 at 8:10 PM Soumyadeep Mukhopadhyay < soumyamy...@gmail.com> wrote: > I am curious about your use case. Are you not losing out on the > optimisations of Calcite when you are using Spark? Is it possible for you > to share a general approach where we will be able to keep the optimisations > done by Calcite and use Spark on top of it? > > > On Fri, 4 Aug 2023 at 5:19 PM, P.F. ZHAN <dethr...@gmail.com> wrote: > > > Generally speaking, the SEARCH operator is very good, but when we use > > Calcite to optimize the logical plan and then use Spark to execute, this > is > > unsupported. So is there a more elegant way to close the SEARCH operator? > > Or how to convert the SEARCH operator to the IN operator before > converting > > the Calcite logical plan to the Spark logical plan? If we do this, we > need > > to consider Join / Filter, are there any other RelNodes? > > > > Maybe, this optimization is optional more better at present for many > query > > execution engine does not support this operator? > > >
