Hi Martijin, Thanks for the feedback. I am not proposing to bundle the new graph library with Alink. I am +1 for dropping the DataSet-based Gelly library, but we probably need a new graph library in Flink for the possible migration.
We haven't decided what to do yet and probably need more discussion. There are some possible solutions: 1. We include a new DataStream-based graph library in FlinkML[1], given that graphs and machine learning algorithms are more often used together [2][3][4]. To achieve this, we could reuse the `AlgoOperator` interface in FlinkML. 2. We include a new DataStream-based graph library as a separate module/repo. This is consistent with existing libraries like Spark [5]. What do you think? [1] https://github.com/apache/flink-ml [2] https://arxiv.org/abs/1403.6652 [3] https://arxiv.org/abs/1503.03578 [4] https://github.com/apache/spark Best, Zhipeng Martijn Visser <mart...@ververica.com> 于2022年1月4日周二 15:27写道： > Hi Zhipeng, > > Good that you've reached out, I wasn't aware that Gelly is being used in > Alink. Are you proposing to write a new graph library as a successor of > Gelly and bundle that with Alink? > > Best regards, > > Martijn > > On Tue, 4 Jan 2022 at 02:57, Zhipeng Zhang <zhangzhipe...@gmail.com> > wrote: > >> Hi everyone, >> >> Thanks for starting the discussion :) >> >> We (Alink team [1]) are actually using part of the Gelly library to >> support graph algorithms (connected component, single source shortest path, >> etc.) for users in Alibaba Inc. >> >> As DataSet API is going to be dropped, shall we also provide a new graph >> library based on DataStream runtime (similar as we did for machine >> learning)? >> >> [1] https://github.com/Alibaba/alink >> >> David Anderson <dander...@apache.org> 于2022年1月4日周二 00:01写道： >> >>> Most of the inquiries I've had about Gelly in recent memory have been >>> from folks looking for a streaming solution, and it's only been a handful. >>> >>> +1 for dropping Gelly >>> >>> David >>> >>> On Mon, Jan 3, 2022 at 2:41 PM Till Rohrmann <trohrm...@apache.org> >>> wrote: >>> >>>> I haven't seen any changes or requests to/for Gelly in ages. Hence, I >>>> would assume that it is not really used and can be removed. >>>> >>>> +1 for dropping Gelly. >>>> >>>> Cheers, >>>> Till >>>> >>>> On Mon, Jan 3, 2022 at 2:20 PM Martijn Visser <mart...@ververica.com> >>>> wrote: >>>> >>>>> Hi everyone, >>>>> >>>>> Flink is bundled with Gelly, a Graph API library [1]. This has been >>>>> marked as approaching end-of-life for quite some time [2]. >>>>> >>>>> Gelly is built on top of Flink's DataSet API, which is deprecated and >>>>> slowly being phased out [3]. It only works on batch jobs. Based on the >>>>> activity in the Dev and User mailing lists, I don't see a lot of questions >>>>> popping up regarding the usage of Gelly. Removing Gelly would reduce CI >>>>> time and resources because we won't need to run tests for this anymore. >>>>> >>>>> I'm cross-posting this to the User mailing list to see if there are >>>>> any users of Gelly at the moment. >>>>> >>>>> Let me know your thoughts. >>>>> >>>>> Martijn Visser | Product Manager >>>>> >>>>> mart...@ververica.com >>>>> >>>>> >>>>> [1] >>>>> https://nightlies.apache.org/flink/flink-docs-stable/docs/libs/gelly/overview/ >>>>> >>>>> [2] https://flink.apache.org/roadmap.html >>>>> >>>>> [3] https://lists.apache.org/thread/b2y3xx3thbcbtzdphoct5wvzwogs9sqz >>>>> >>>>> <https://www.ververica.com/> >>>>> >>>>> >>>>> Follow us @VervericaData >>>>> >>>>> -- >>>>> >>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink >>>>> Conference >>>>> >>>>> Stream Processing | Event Driven | Real Time >>>>> >>>>> >> >> -- >> best, >> Zhipeng >> >> -- best, Zhipeng
