Hi, I think what we are landing on finally is.
- Keep pushing for SparkSQL support using Spark extensions route - Calcite effort will be a separate/orthogonal approach, down the line Please feel free to correct me, if I got this wrong. On Mon, Dec 21, 2020 at 3:30 AM pzwpzw <pengzhiwei2...@icloud.com.invalid> wrote: > Hi 受春柏 ,here is my point. We can use Calcite to build a common sql layer > to process engine independent SQL, for example most of the DDL、Hoodie CLI > command and also provide parser for the common SQL extensions(e.g. Merge > Into). The Engine-related syntax can be taught to the respective engines to > process. If the common sql layer can handle the input sql, it handle > it.Otherwise it is routed to the engine for processing. In long term, the > common layer will more and more rich and perfect. > 2020年12月21日 下午4:38,受春柏 <sc...@126.com> 写道: > > Hi,all > > > That's very good,Hudi SQL syntax can support Flink、hive and other analysis > components at the same time, > But there are some questions about SparkSQL. SparkSQL syntax is in > conflict with Calctite syntax.Is our strategy > user migration or syntax compatibility? > In addition ,will it also support write SQL? > > > > > > > > > > > > > > > > > > > > > 在 2020-12-19 02:10:16,"Nishith" <n3.nas...@gmail.com> 写道: > > That’s awesome. Looks like we have a consensus on Calcite. Look forward to > the RFC as well! > > > -Nishith > > > On Dec 18, 2020, at 9:03 AM, Vinoth Chandar <vin...@apache.org> wrote: > > > Sounds good. Look forward to a RFC/DISCUSS thread. > > > Thanks > > Vinoth > > > On Thu, Dec 17, 2020 at 6:04 PM Danny Chan <danny0...@apache.org> wrote: > > > Yes, Apache Flink basically reuse the DQL syntax of Apache Calcite, i would > > add support for SQL connectors of Hoodie Flink soon ~ > > Currently, i'm preparing a refactoring to the current Flink writer code. > > > Vinoth Chandar <vin...@apache.org> 于2020年12月18日周五 上午6:39写道: > > > Thanks Kabeer for the note on gmail. Did not realize that. :) > > > My desired use case is user use the Hoodie CLI to execute these SQLs. > > They can choose what engine to use by a CLI config option. > > > Yes, that is also another attractive aspect of this route. We can build > > out > > a common SQL layer and have this translate to the underlying engine > > (sounds > > like Hive huh) > > Longer term, if we really think we can more easily implement a full DML + > > DDL + DQL, we can proceed with this. > > > As others pointed out, for Spark SQL, it might be good to try the Spark > > extensions route, before we take this on more fully. > > > The other part where Calcite is great is, all the support for > > windowing/streaming in its syntax. > > Danny, I guess if we should be able to leverage that through a deeper > > Flink/Hudi integration? > > > > On Thu, Dec 17, 2020 at 1:07 PM Vinoth Chandar <vin...@apache.org> > > wrote: > > > I think Dongwook is investigating on the same lines. and it does seem > > better to pursue this first, before trying other approaches. > > > > > On Tue, Dec 15, 2020 at 1:38 AM pzwpzw <pengzhiwei2...@icloud.com > > .invalid> > > wrote: > > > Yeah I agree with Nishith that an option way is to look at the > > ways > > to > > plug in custom logical and physical plans in Spark. It can simplify > > the > > implementation and reuse the Spark SQL syntax. And also users > > familiar > > with > > Spark SQL will be able to use HUDi's SQL features more quickly. > > In fact, spark have provided the SparkSessionExtensions interface for > > implement custom syntax extensions and SQL rewrite rule. > > > > > > https://spark.apache.org/docs/2.4.5/api/java/org/apache/spark/sql/SparkSessionExtensions.html > > . > > We can use the SparkSessionExtensions to extended hoodie sql syntax > > such > > as MERGE INTO and DDL syntax. > > > 2020年12月15日 下午3:27,Nishith <n3.nas...@gmail.com> 写道: > > > Thanks for starting this thread Vinoth. > > In general, definitely see the need for SQL style semantics on Hudi > > tables. Apache Calcite is a great option to considering given > > DatasourceV2 > > has the limitations that you described. > > > Additionally, even if Spark DatasourceV2 allowed for the flexibility, > > the > > same SQL semantics needs to be supported in other engines like Flink > > to > > provide the same experience to users - which in itself could also be > > considerable amount of work. > > So, if we’re able to generalize on the SQL story along Calcite, that > > would > > help reduce redundant work in some sense. > > Although, I’m worried about a few things > > > 1) Like you pointed out, writing complex user jobs using Spark SQL > > syntax > > can be harder for users who are moving from “Hudi syntax” to “Spark > > syntax” > > for cross table joins, merges etc using data frames. One option is to > > look > > at the if there are ways to plug in custom logical and physical plans > > in > > Spark, this way, although the merge on sparksql functionality may not > > be > > as > > simple to use, but wouldn’t take away performance and feature set for > > starters, in the future we could think of having the entire query > > space > > be > > powered by calcite like you mentioned > > 2) If we continue to use DatasourceV1, is there any downside to this > > from > > a performance and optimization perspective when executing plan - I’m > > guessing not but haven’t delved into the code to see if there’s > > anything > > different apart from the API and spec. > > > On Dec 14, 2020, at 11:06 PM, Vinoth Chandar <vin...@apache.org> > > wrote: > > > > Hello all, > > > > Just bumping this thread again > > > > thanks > > > vinoth > > > > On Thu, Dec 10, 2020 at 11:58 PM Vinoth Chandar <vin...@apache.org> > > wrote: > > > > Hello all, > > > > One feature that keeps coming up is the ability to use UPDATE, MERGE > > sql > > > syntax to support writing into Hudi tables. We have looked into the > > Spark 3 > > > DataSource V2 APIs as well and found several issues that hinder us in > > > implementing this via the Spark APIs > > > > - As of this writing, the UPDATE/MERGE syntax is not really opened up > > to > > > external datasources like Hudi. only DELETE is. > > > - DataSource V2 API offers no flexibility to perform any kind of > > > further transformations to the dataframe. Hudi supports keys, > > indexes, > > > preCombining and custom partitioning that ensures file sizes etc. All > > this > > > needs shuffling data, looking up/joining against other dataframes so > > forth. > > > Today, the DataSource V1 API allows this kind of further > > > partitions/transformations. But the V2 API is simply offers partition > > level > > > iteration once the user calls df.write.format("hudi") > > > > One thought I had is to explore Apache Calcite and write an adapter > > for > > > Hudi. This frees us from being very dependent on a particular > > engine's > > > syntax support like Spark. Calcite is very popular by itself and > > supports > > > most of the key words and (also more streaming friendly syntax). To > > be > > > clear, we will still be using Spark/Flink underneath to perform the > > actual > > > writing, just that the SQL grammar is provided by Calcite. > > > > To give a taste of how this will look like. > > > > A) If the user wants to mutate a Hudi table using SQL > > > > Instead of writing something like : spark.sql("UPDATE ....") > > > users will write : hudiSparkSession.sql("UPDATE ....") > > > > B) To save a Spark data frame to a Hudi table > > > we continue to use Spark DataSource V1 > > > > The obvious challenge I see is the disconnect with the Spark > > DataFrame > > > ecosystem. Users would write MERGE SQL statements by joining against > > other > > > Spark DataFrames. > > > If we want those expressed in calcite as well, we need to also invest > > in > > > the full Query side support, which can increase the scope by a lot. > > > Some amount of investigation needs to happen, but ideally we should > > be > > > able to integrate with the sparkSQL catalog and reuse all the tables > > there. > > > > I am sure there are some gaps in my thinking. Just starting this > > thread, > > > so we can discuss and others can chime in/correct me. > > > > thanks > > > vinoth > > > > > > > >