+1 Thank you Shaofeng for the details.
I've connected with my colleagues at Microsoft and the project roadmap is super exciting to me. As my colleagues are part of various Apache projects Spark, Hadoop, Flink, Trino projects, I believe I can provide insights and add diversity to this project. Happy to be a mentor on this proposal during the incubation process, if you are interested. Thanks, Rakesh On Thu, Dec 28, 2023 at 6:47 AM ShaoFeng Shi <shaofeng...@apache.org> wrote: > Hi Rakesh, > > Yeah, the team aware that, they plan to merge the Trino plugin into the > main repository at some point of time, depends on its maturity. The Hive > plugin is not on current roadmap I think, but this is open for discussion. > > Best regards, > > Shaofeng Shi 史少锋 > Apache Kylin PMC, > Apache Incubator PMC, > Email: shaofeng...@apache.org > > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html > Join Kylin user mail group: user-subscr...@kylin.apache.org > Join Kylin dev mail group: dev-subscr...@kylin.apache.org > > > > > Rakesh Radhakrishnan <rake...@apache.org> 于2023年12月27日周三 14:06写道： > > > OK, that's really interesting. I hope you will add "Gluten-Trino > > <https://github.com/oap-project/Gluten-Trino>" and "Gluten-Flink" as > > sub-projects(plugin module) under the main Gluten project and follow a > > single release cycle? > > > > Is it technically/architecturally feasible to develop a "Gluten-Hive" > > plugin (as we know Hive uses Apache Tez DAG) as a long term plan ? > > > > Thanks, > > Rakesh > > > > On Tue, Dec 26, 2023 at 4:30 PM ShaoFeng Shi <shaofeng...@apache.org> > > wrote: > > > > > Hi Rakesh, > > > > > > Thanks for your comment. The Trino plugin is under PoC, not ready for > > > widely use at this moment, so it is staged in another git repository: > > > https://github.com/oap-project/Gluten-Trino > > > Flink support is still in early discussion stage, need more > collaboration > > > in the future. > > > > > > Best regards, > > > > > > Shaofeng Shi 史少锋 > > > Apache Kylin PMC, > > > Apache Incubator PMC, > > > Email: shaofeng...@apache.org > > > > > > Apache Kylin FAQ: > https://kylin.apache.org/docs/gettingstarted/faq.html > > > Join Kylin user mail group: user-subscr...@kylin.apache.org > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org > > > > > > > > > > > > > > > Rakesh Radhakrishnan <rake...@apache.org> 于2023年12月26日周二 11:24写道： > > > > > > > Thanks Shaofeng and team for the efforts! > > > > > > > > Its really interesting to see that Flink and Trino projects can get > the > > > > benefits of Gluten and boost their performance. Much appreciated, if > > you > > > > could share more on this integration, perf_numbers, > > > pullrequest_reference, > > > > jira_reference etc. Will Flink and Trino already have a plugin > > mechanism > > > in > > > > place to seamlessly integrate with Gluten ? > > > > > > > > Thanks, > > > > Rakesh > > > > > > > > On Sat, Dec 23, 2023 at 6:13 PM ShaoFeng Shi <shaofeng...@apache.org > > > > > > wrote: > > > > > > > > > Hi Enrico, > > > > > > > > > > This is a good question. Actually I asked the same to the team when > > > they > > > > > find me for coaching. > > > > > > > > > > As the proposal mentioned, Gluten is a middle layer between the > > > JVM-based > > > > > SQL engine and native libraries. It connects the two sides, and > build > > > > > common services like unified plan transformation, seamless native > > > > > integration, clear JNI interfaces, etc. Spark is the first engine > it > > > > > supports now, but the framework and mechanism can also work for > other > > > > > engines like Trino, Flink SQL in the future (the proposal also > > > mentioned > > > > > this). Underlying Gluten, it can integrate with different > libararies, > > > > like > > > > > Velox, Clickhouse and Arrow (not ready yet). So, its mission and > > scope > > > is > > > > > different, that's why we put it as a new project. > > > > > > > > > > Best regards, > > > > > > > > > > Shaofeng Shi 史少锋 > > > > > Apache Kylin PMC, > > > > > Apache Incubator PMC, > > > > > Email: shaofeng...@apache.org > > > > > > > > > > Apache Kylin FAQ: > > > https://kylin.apache.org/docs/gettingstarted/faq.html > > > > > Join Kylin user mail group: user-subscr...@kylin.apache.org > > > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > Enrico Olivelli <eolive...@gmail.com> 于2023年12月22日周五 22:43写道： > > > > > > > > > > > Shaofeng, > > > > > > > > > > > > Il giorno ven 22 dic 2023 alle ore 13:59 ShaoFeng Shi > > > > > > <shaofeng...@apache.org> ha scritto: > > > > > > > > > > > > > > Hi IPMC members, > > > > > > > > > > > > > > I would like to propose a new project to the ASF incubator - > > > Gluten. > > > > > > > > > > > > > > Gluten[1] is a middle layer responsible for offloading Apache > > Spark > > > > SQL > > > > > > > queries to native engines. This project aims to address the CPU > > > > > > > computational bottleneck to offload SparkSQL operators to > native > > > > > engines > > > > > > in > > > > > > > data loading scenarios based on Apache Spark. > > > > > > > > > > > > Have you considered making this project a subproject of Apache > > Spark > > > ? > > > > > > > > > > > > Enrico > > > > > > > > > > > > > > > > > > > > Here is the proposal - > > > > > > > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/GlutenProposal > > > > > > > > > > > > > > I would be the Champion of the project. I will mentor and help > > the > > > > > > project > > > > > > > through the incubator with Yu Li [l...@apache.org], Kent Yao [ > > > > > > y...@apache.org] > > > > > > > and Wenli Zhang [ovi...@apache.org] . > > > > > > > > > > > > > > We are open to hearing the feedback from the incubator. > > > > > > > > > > > > > > Best, > > > > > > > Shaofeng Shi. > > > > > > > > > > > > > > [1] https://oap-project.github.io/gluten/ > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Shaofeng Shi 史少锋 > > > > > > > Apache Kylin PMC, > > > > > > > Apache Incubator PMC, > > > > > > > Email: shaofeng...@apache.org > > > > > > > > > > > > > > Apache Kylin FAQ: > > > > > https://kylin.apache.org/docs/gettingstarted/faq.html > > > > > > > Join Kylin user mail group: user-subscr...@kylin.apache.org > > > > > > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > > > > For additional commands, e-mail: > general-h...@incubator.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > >
