Re: Using Calcite as a Distributed Optimizer
Hi Atri, you can refer to this list [1]. I also aware that Apache Ignite and Hazelcast made some efforts to employ Calcite as a distributed query planner. [1] https://calcite.apache.org/docs/powered_by.html -- Roman Kondakov On 14.05.2021 14:31, Atri Sharma wrote: Is there an example I can refer to? On Fri, 14 May 2021, 04:59 Haisheng Yuan, wrote: Yes, definitely. Many distributed big data systems use Apache Calcite to optimize queries and generate distributed plans. On 2021/05/13 23:16:10, Atri Sharma wrote: Thank you. So my use case is such that I wish to use Calcite as a two phase optimizer -- Get a SQL query, compile it and optimize it and convert it to a SQL fragment. Then run the query on worker nodes, get results on master and merge results. This question spans both Calcite and Avatica, but I wanted to understand if achieving the above is possible with Calcite today. Atri On Fri, 14 May 2021, 01:20 Julian Hyde, wrote: Calcite has no user@ list. So, ask away here! On May 13, 2021, at 10:52 AM, Atri Sharma wrote: Sorry, didn't realize I had sent to the dev list. I will send to the user list On Thu, 13 May 2021, 15:57 Atri Sharma, wrote: Hi All, Are there examples of using Calcite to compile and optimize queries to be run on a set of nodes, and then merge partial results back? Atri -- Regards, Atri l'apprenant
Re: Using Calcite as a Distributed Optimizer
I think you can refer to Flink Best, Liya Fan On Fri, May 14, 2021 at 12:32 PM Atri Sharma wrote: > Is there an example I can refer to? > > On Fri, 14 May 2021, 04:59 Haisheng Yuan, wrote: > > > Yes, definitely. Many distributed big data systems use Apache Calcite to > > optimize queries and generate distributed plans. > > > > On 2021/05/13 23:16:10, Atri Sharma wrote: > > > Thank you. > > > > > > So my use case is such that I wish to use Calcite as a two phase > > optimizer > > > -- Get a SQL query, compile it and optimize it and convert it to a SQL > > > fragment. > > > > > > Then run the query on worker nodes, get results on master and merge > > results. > > > > > > This question spans both Calcite and Avatica, but I wanted to > understand > > if > > > achieving the above is possible with Calcite today. > > > > > > Atri > > > > > > On Fri, 14 May 2021, 01:20 Julian Hyde, > wrote: > > > > > > > Calcite has no user@ list. So, ask away here! > > > > > > > > > On May 13, 2021, at 10:52 AM, Atri Sharma > > wrote: > > > > > > > > > > Sorry, didn't realize I had sent to the dev list. I will send to > the > > user > > > > > list > > > > > > > > > > On Thu, 13 May 2021, 15:57 Atri Sharma, > wrote: > > > > > > > > > >> Hi All, > > > > >> > > > > >> Are there examples of using Calcite to compile and optimize > queries > > to > > > > >> be run on a set of nodes, and then merge partial results back? > > > > >> > > > > >> Atri > > > > >> > > > > >> -- > > > > >> Regards, > > > > >> > > > > >> Atri > > > > >> l'apprenant > > > > >> > > > > > > > > > > > > > >
Re: Using Calcite as a Distributed Optimizer
Is there an example I can refer to? On Fri, 14 May 2021, 04:59 Haisheng Yuan, wrote: > Yes, definitely. Many distributed big data systems use Apache Calcite to > optimize queries and generate distributed plans. > > On 2021/05/13 23:16:10, Atri Sharma wrote: > > Thank you. > > > > So my use case is such that I wish to use Calcite as a two phase > optimizer > > -- Get a SQL query, compile it and optimize it and convert it to a SQL > > fragment. > > > > Then run the query on worker nodes, get results on master and merge > results. > > > > This question spans both Calcite and Avatica, but I wanted to understand > if > > achieving the above is possible with Calcite today. > > > > Atri > > > > On Fri, 14 May 2021, 01:20 Julian Hyde, wrote: > > > > > Calcite has no user@ list. So, ask away here! > > > > > > > On May 13, 2021, at 10:52 AM, Atri Sharma > wrote: > > > > > > > > Sorry, didn't realize I had sent to the dev list. I will send to the > user > > > > list > > > > > > > > On Thu, 13 May 2021, 15:57 Atri Sharma, wrote: > > > > > > > >> Hi All, > > > >> > > > >> Are there examples of using Calcite to compile and optimize queries > to > > > >> be run on a set of nodes, and then merge partial results back? > > > >> > > > >> Atri > > > >> > > > >> -- > > > >> Regards, > > > >> > > > >> Atri > > > >> l'apprenant > > > >> > > > > > > > > >
Re: Using Calcite as a Distributed Optimizer
Yes, definitely. Many distributed big data systems use Apache Calcite to optimize queries and generate distributed plans. On 2021/05/13 23:16:10, Atri Sharma wrote: > Thank you. > > So my use case is such that I wish to use Calcite as a two phase optimizer > -- Get a SQL query, compile it and optimize it and convert it to a SQL > fragment. > > Then run the query on worker nodes, get results on master and merge results. > > This question spans both Calcite and Avatica, but I wanted to understand if > achieving the above is possible with Calcite today. > > Atri > > On Fri, 14 May 2021, 01:20 Julian Hyde, wrote: > > > Calcite has no user@ list. So, ask away here! > > > > > On May 13, 2021, at 10:52 AM, Atri Sharma wrote: > > > > > > Sorry, didn't realize I had sent to the dev list. I will send to the user > > > list > > > > > > On Thu, 13 May 2021, 15:57 Atri Sharma, wrote: > > > > > >> Hi All, > > >> > > >> Are there examples of using Calcite to compile and optimize queries to > > >> be run on a set of nodes, and then merge partial results back? > > >> > > >> Atri > > >> > > >> -- > > >> Regards, > > >> > > >> Atri > > >> l'apprenant > > >> > > > > >
Re: Using Calcite as a Distributed Optimizer
Thank you. So my use case is such that I wish to use Calcite as a two phase optimizer -- Get a SQL query, compile it and optimize it and convert it to a SQL fragment. Then run the query on worker nodes, get results on master and merge results. This question spans both Calcite and Avatica, but I wanted to understand if achieving the above is possible with Calcite today. Atri On Fri, 14 May 2021, 01:20 Julian Hyde, wrote: > Calcite has no user@ list. So, ask away here! > > > On May 13, 2021, at 10:52 AM, Atri Sharma wrote: > > > > Sorry, didn't realize I had sent to the dev list. I will send to the user > > list > > > > On Thu, 13 May 2021, 15:57 Atri Sharma, wrote: > > > >> Hi All, > >> > >> Are there examples of using Calcite to compile and optimize queries to > >> be run on a set of nodes, and then merge partial results back? > >> > >> Atri > >> > >> -- > >> Regards, > >> > >> Atri > >> l'apprenant > >> > >
Re: Using Calcite as a Distributed Optimizer
Calcite has no user@ list. So, ask away here! > On May 13, 2021, at 10:52 AM, Atri Sharma wrote: > > Sorry, didn't realize I had sent to the dev list. I will send to the user > list > > On Thu, 13 May 2021, 15:57 Atri Sharma, wrote: > >> Hi All, >> >> Are there examples of using Calcite to compile and optimize queries to >> be run on a set of nodes, and then merge partial results back? >> >> Atri >> >> -- >> Regards, >> >> Atri >> l'apprenant >>
Re: Using Calcite as a Distributed Optimizer
Sorry, didn't realize I had sent to the dev list. I will send to the user list On Thu, 13 May 2021, 15:57 Atri Sharma, wrote: > Hi All, > > Are there examples of using Calcite to compile and optimize queries to > be run on a set of nodes, and then merge partial results back? > > Atri > > -- > Regards, > > Atri > l'apprenant >
