Thank you, shall explore more on this! :)
On Fri, 4 Aug 2023 at 5:53 PM, P.F. ZHAN <dethr...@gmail.com> wrote: > Aha, I'm using Apache Kylin which uses Calcite to generate a logical plan, > then convert to Spark plan to execute a query. Given that Calcite has more > operations for aggregations, and Kylin wants to take full advantage of > precomputed cubes (something like Calcite's materialized views), it uses > both Calcite and Spark(for distribution computing). Maybe it's wild and a > little fun, but it does works well on many scenarios. > > On Fri, Aug 4, 2023 at 8:10 PM Soumyadeep Mukhopadhyay < > soumyamy...@gmail.com> wrote: > > > I am curious about your use case. Are you not losing out on the > > optimisations of Calcite when you are using Spark? Is it possible for you > > to share a general approach where we will be able to keep the > optimisations > > done by Calcite and use Spark on top of it? > > > > > > On Fri, 4 Aug 2023 at 5:19 PM, P.F. ZHAN <dethr...@gmail.com> wrote: > > > > > Generally speaking, the SEARCH operator is very good, but when we use > > > Calcite to optimize the logical plan and then use Spark to execute, > this > > is > > > unsupported. So is there a more elegant way to close the SEARCH > operator? > > > Or how to convert the SEARCH operator to the IN operator before > > converting > > > the Calcite logical plan to the Spark logical plan? If we do this, we > > need > > > to consider Join / Filter, are there any other RelNodes? > > > > > > Maybe, this optimization is optional more better at present for many > > query > > > execution engine does not support this operator? > > > > > >
