@Siddharth
> Secondly, the existing runner does not support Beam's windowing and > triggering semantics in the Flink DataStream API Can you please expand on this? AFAIK the current Flink runner is feature complete w.r.t. windowing support. which provides more control over the execution of the pipeline but requires > more manual configuration > What manual configuration are you referring to? The user should be completely abstracted away from the Flink APIs. If this is not the case, it's something to be fixed. @Jeff one idea in my mind is to integrate the beam sql api with flink table api > (this might be a bit outdated, please correct me if I'm wrong) AFAIK Beam SQL is just a DSL over Beam's low-level APIs. This makes introducing new runners fairly straightforward because you only need to support a few primitive transformations to have a fully working runner (even though it might not be optimal performance-wise); everything else is just built on top of those. This prevents us from directly translating Beam SQL into Flink SQL / Flink Table API (you always need to go through Beam's low-level API). Unless there is a good reason, we should not introduce another Flink runner into the Beam codebase. The current one, which is very advanced feature-wise (it took years to get there), already has a need for more maintainers. If you see weak spots in the current implementation, would you consider improving the existing runner instead of trying to write a new one from scratch? Best, D. On Fri, Apr 28, 2023 at 11:29 AM Siddharth Aryan < siddhartharyan...@gmail.com> wrote: > Hello Jeff, > Thank you for the idea, as it will allow beam users to write sql > queries using the Beam SQL API and execute them on the Flink Table API.I > will look into it later as my current focus is to implement an integration > between Apache Beam and the Flink DataStream API. While the existing Flink > runner is based on DataStream and Operator API, my project aims to create a > new runner that specifically utilizes the Flink DataStream API. > And thanks for the feedback. > > Best Regards, > Siddharth Aryan > > On Thu, Apr 27, 2023 at 1:39 PM Jeff Zhang <zjf...@gmail.com> wrote: > >> Same question as David, one idea in my mind is to integrate the beam sql >> api with flink table api, this does not exist in the current flink runner. >> >> On Thu, Apr 27, 2023 at 3:46 PM David Morávek <d...@apache.org> wrote: >> >>> Hi Siddharth, >>> >>> Thanks for your interest in the Flink Runner for Beam. Reading through >>> the project, one thing that immediately strikes me is that there already is >>> a Flink runner based on DataStream and Operator (one level below >>> DataStream) API in the code base. Are you aware of this? If yes, how does >>> the runner you want to introduce differ from the existing one? >>> >>> Best, >>> D. >>> >>> On Sun, Apr 2, 2023 at 9:41 PM Svetak Sundhar via dev < >>> dev@beam.apache.org> wrote: >>> >>>> Hi Siddharth, >>>> I left some comments as well on the sentiment analysis proposal. >>>> >>>> Thanks, >>>> >>>> >>>> Svetak Sundhar >>>> >>>> Technical Solutions Engineer, Data >>>> s <nellywil...@google.com>vetaksund...@google.com >>>> >>>> >>>> >>>> On Sun, Apr 2, 2023 at 1:58 PM Anand Inguva via dev < >>>> dev@beam.apache.org> wrote: >>>> >>>>> I left some comments on the sentiment analysis proposal. >>>>> >>>>> Thanks, >>>>> Anand >>>>> >>>>> On Thu, Mar 30, 2023 at 9:59 AM Danny McCormick via dev < >>>>> dev@beam.apache.org> wrote: >>>>> >>>>>> Thanks Siddharth! I left some comments on the sentiment analysis >>>>>> proposal, I am probably not the best person to comment on the flink >>>>>> datastream api one though. >>>>>> >>>>>> Thanks, >>>>>> Danny >>>>>> >>>>>> On Fri, Mar 24, 2023 at 11:53 PM Siddharth Aryan < >>>>>> siddhartharyan...@gmail.com> wrote: >>>>>> >>>>>>> Hello , >>>>>>> I am Siddharth Aryan a undergrad and I am looking forward to someone >>>>>>> who can help me reviewing my proposal and give me a feedback on the them >>>>>>> which help me to create a good proposal. >>>>>>> Here ,I am attaching my both the project proposals: >>>>>>> >Sentimental Analysis Pipeline with the help of Machine Learnig: >>>>>>> >>>>>>> https://docs.google.com/document/d/1U6zcXAWsDCrWlbf14f5VlLqPZFucwXR48tD7mrERW-g/edit?usp=sharing >>>>>>> >>>>>>> >Integrating Apache Beam with Flink Datastream API: >>>>>>> >>>>>>> https://docs.google.com/document/d/1sQEe9eVuoHX9QWS9Zj5wVl7MLmfk7QO09pjZOsk-TFY/edit?usp=sharing >>>>>>> >>>>>>> Best Regards >>>>>>> Siddharth Aryan >>>>>>> >>>>>>> Github :https://github.com/nervoussidd >>>>>>> >>>>>> >> >> -- >> Best Regards >> >> Jeff Zhang >> >
