Okay that sounds perfect. Thank you! On Mon, Jan 3, 2022 at 4:52 AM Bertty Contreras <[email protected]> wrote:
> So nice, I will organize my schedule for it and i will come back to you > with options. Mean while I will collecting all the designs and other > elements that are done and could help you with the implementation of the > new feature ;), > > Best regards, > Bertty > > On Mon 3. Jan 2022 at 04:41, kamalesh palanisamy <[email protected]> > wrote: > >> Hi, >> Thank you for the explanation. Yes, I feel it would be better if we could >> discuss it so that everything is clear. I am free from Wednesday-Saturday >> anytime after 3 PM Germany time. You can select which every day suits your >> schedule best during this time. >> >> Thanks, >> Kamalesh P >> >> >> On Sun, Jan 2, 2022 at 6:28 PM Bertty Contreras <[email protected]> >> wrote: >> >>> The main is idea of wayang is to provide a layer that pick the best >>> combination of platform to process a query, you can see the details on the >>> paper rheemix[1] >>> >>> Then providing a SQL-API will allow to transform a query into different >>> operators of wayang that will allow optimization with platform that only >>> have SQL like postgres with platforms that don’t SQL lenguaje like giraph. >>> >>> The idea to use calcite, is coming from the intermediate representation >>> that calcite generates that will allows us to create the wayang plan with >>> an “udf” that are translateble again to SQL or translatable to a executable >>> code that can be executed by flink, as an example. >>> >>> Imagen the query that it said something like: >>> >>> Select A.a,A.b,A.c from A join A.a = X.a …. >>> >>> Then X(10TB) is on HDFS and A(100MB) is on postgres, then the plan to >>> execute will something like: >>> >>> Select A.a from A(1MB), this file is small then you can do broadcast and >>> filter using flink. >>> >>> Then the join results are just 2 records, the wayang will perform the >>> query on postgres using the 2 record as condition. >>> >>> But also could occurs that the join answer is 1TB, in that case, the >>> data of postgres will be move to HDFS and the all the rest of the process >>> will be on using flink. >>> >>> Currently the optimizer is taking the decision of what platform will be >>> used depending on the amount of data to process and data movement. Then the >>> SQL-API will provide an way of “freedom” the decisions because we will have >>> all the intermediate representation to performs changes. >>> >>> After we have the SQL-API we will be adding platforms that just support >>> and SQL ;), as you said. >>> >>> The idea of using the intermediate representation it maybe sound weird >>> to you, but we can have a meeting to explain you better, then you can >>> understand better the full concept and also give us your feedback, let me >>> if hyou are available and when and I will freedom my schedule for it ;). >>> I’m in Germany just to you figure if we have some timezone differences ;). >>> >>> Best regards, >>> Bertty >>> >>> [1] >>> https://wayang.apache.org/assets/pdf/paper/journal_vldb.pdf >>> >>> >>> On Sun 2. Jan 2022 at 17:43, kamalesh palanisamy <[email protected]> >>> wrote: >>> >>>> Hi Bertty, >>>> Thank you for the information! I would love to work on adding the SQL >>>> API for Wayang. Basically, now I need to add a new platform for the >>>> wayang-platforms that supports SQL through apache calcite? Am I right? >>>> Please do correct me if I am wrong. >>>> >>>> Thanks, >>>> Kamalesh P >>>> >>>> >>>> On Sun, Jan 2, 2022 at 3:36 AM Bertty Contreras <[email protected]> >>>> wrote: >>>> >>>>> Hi Kamalesh, >>>>> >>>>> Currently, Apache Wayang(Incubating) has the issues listed in Jira >>>>> [1]. One feature that the community didn't have time to work on is the SQL >>>>> API for Apache Wayang(Incubating) [2]; the main idea is to use Apache >>>>> Calcite [3] as the parser of the SQL and then do something like Spark >>>>> adapter of calcite [4]. If you want to contribute to this feature, it will >>>>> be so awesome :D. >>>>> >>>>> If you found another issue interesting, let me know, or even if you >>>>> have some idea of a feature will be so awesome too :D >>>>> >>>>> Best regards, >>>>> Bertty >>>>> >>>>> [1] https://issues.apache.org/jira/projects/WAYANG >>>>> [2] >>>>> https://issues.apache.org/jira/projects/WAYANG/issues/WAYANG-25?filter=allopenissues >>>>> [3] https://calcite.apache.org >>>>> [4] https://github.com/apache/calcite/tree/master/spark >>>>> >>>>> On Sun, Jan 2, 2022 at 6:50 AM kamalesh palanisamy < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> My name is Kamalesh and I am currently looking to contribute to the >>>>>> project, but I couldn't find any proper issues. Can you help me with >>>>>> any >>>>>> features you would like me to contribute to?. Thanks! >>>>>> Thanks, >>>>>> Kamalesh P >>>>>> >>>>> -- Thanks, Kamalesh P
