Hi Zhaohui, Thanks for sharing. Materialized views are crucial to performance optimization in the data lake. We mentioned that in the blog post too as a future work. Would love to discuss this further.
Thanks, Walaa. On Sat, Dec 12, 2020 at 11:49 AM 953396112 <13282155...@qq.com> wrote: > Hi Walaa and Tao: > I am very happy to see your sharing. Our team is also > working on SQL rewriting, analysis and optimization. Using Calcite's > materialized view recognition capabilities to speed up user queries, many > materialized view recognition algorithms[1] and normalization algorithms[2] > have been implemented, and a lot of work has been done in preprocessing > available materialized views, which has excellent performance. Welcome to > discuss issues related to materialized view identification. > > > [1] https://github.com/apache/calcite/pull/2094 > [2] https://github.com/apache/calcite/pull/2262 > > > Regards! > Zhaohui Xu > > > ------------------ 原始邮件 ------------------ > 发件人: > "dev" > < > taojia...@gmail.com>; > 发送时间: 2020年12月12日(星期六) 晚上9:23 > 收件人: "dev"<dev@calcite.apache.org>; > > 主题: Re: Using Calcite at LinkedIn > > > > Hi Walaa > Very happy to see this, our team basically do the same thing, a unified SQL > layer: > 1. Spark: RelNode -> Spark DataFrame plan > 2. Presto: RelNode -> In string SQL > 3. Clickhouse: RelNode -> Serialized RelNode > 4. Flink -> TBD(with datastream API or table API) > > > I do point 1 both in my previous company and current company, maybe I can > participate in this part: analyze and translate Spark Catalyst plans. > > > Regards! > > Aron Tao > > > Walaa Eldin Moustafa <wa.moust...@gmail.com> 于2020年12月12日周六 上午5:34写道： > > > Hi Calcite community, > > > > I wanted to share a recently published LinkedIn's blog series article > [1] > > on how Calcite helps us build a smarter data lake using Coral [2]. > Hope you > > find it interesting. Also, if you want to discuss with our team and > the > > data lake + Calcite community, please feel free to join our Coral > Slack > > workspace [3]. > > > > [1] https://engineering.linkedin.com/blog/2020/coral > > [2] https://github.com/linkedin/coral > > [3] > > > > > https://join.slack.com/t/coral-sql/shared_invite/zt-j9jw5idg-mkt3fjA~wgoUEMXXZqMr6g > > > <https://join.slack.com/t/coral-sql/shared_invite/zt-j9jw5idg-mkt3fjA~wgoUEMXXZqMr6g>> > ; > > Thanks, > > Walaa. > >
