Yes, Community is a better place. You can also add the Dataflow Runner https://github.com/dataArtisans/flink-dataflow.
On Fri, Oct 9, 2015 at 3:32 PM, Vasiliki Kalavri <vasilikikala...@gmail.com> wrote: > Thank you Matthias! > > I'm not sure where the "Downloads" section is the right place for this. > I would actually put it under "Community", with a header "External > Contributions" or something like this, but I'm not feeling strong about > this :) > > -Vasia. > > > On 9 October 2015 at 15:29, Matthias J. Sax <mj...@apache.org> wrote: > >> I was not sure what we should add and was hoping for input from the >> community. >> >> I am aware of the following projects we might want to add: >> >> - Zeppelin >> - SAMOA >> - Mahout >> - Cascading (dataartisan repo) >> - BigPetStore >> - Gradoop >> >> >> -Matthias >> >> >> >> On 10/09/2015 03:07 PM, Maximilian Michels wrote: >> > Cool. Right now the list is empty. Do you already have a list you >> > could include in the upcoming pull request? :) >> > >> > On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <mj...@apache.org> >> wrote: >> >> Hi, >> >> >> >> I just started this. Please see >> >> https://github.com/mjsax/flink-web/tree/flink-external-page >> >> >> >> I think, it is the best way to extend the "Downloads" page. I would also >> >> add a link to this on the main page's "Getting Started" section. >> >> >> >> As a first try, I started like this: >> >>> Third party packages >> >>> >> >>> This is a list of third party packages (ie, libraries, system >> extensions, or examples) build for Flink. The Flink community only collects >> links to those packages but does not maintain them. Thus, they do not >> belong to the Apache Flink project and the community cannot give any >> support for them. >> >>> Package Name >> >>> >> >>> Available for Flink 0.8.x and 0.9.x >> >>> >> >>> Short description >> >>> >> >>> Please let us know, if we missed to list your package. Be aware, that >> we might remove listed packages without notice. >> >> >> >> Can you please give me some input, what projects I should add initially? >> >> >> >> >> >> -Matthias >> >> >> >> >> >> On 10/08/2015 04:03 PM, Maximilian Michels wrote: >> >>> IMHO we can do that. There should be a disclaimer that the third party >> >>> software is not officially supported. >> >>> >> >>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <mj...@apache.org> >> wrote: >> >>>> Should we add a new page at Flink project web page? >> >>>> >> >>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote: >> >>>>> +1 for your pragmatic approach, Vasia. A simple collection of third >> >>>>> party software using Flink should be enough for now; of course, >> >>>>> outside the Apache realm. >> >>>>> >> >>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <chiwanp...@apache.org> >> wrote: >> >>>>>> +1 for Vasia’s suggestion. From a long-term perspective, the site >> like Spark Packages [1] would be helpful to manage external contribution. >> >>>>>> >> >>>>>> [1] http://spark-packages.org >> >>>>>> >> >>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <mj...@apache.org> >> wrote: >> >>>>>>> >> >>>>>>> Thanks for the feedback. >> >>>>>>> >> >>>>>>> I think, the repository does not need to build on a single Flink >> >>>>>>> release. From my point of view, there should be a single parent >> module >> >>>>>>> that contains *independent modules* for each extension/library >> (there >> >>>>>>> should be no "cross dependencies" between the modules and each >> module >> >>>>>>> can specify the flink dependencies it needs by itself). This make >> is >> >>>>>>> most flexible. And if a library works on an old release, it might >> just >> >>>>>>> stay there as is. If a library changes (due to Flink changes), it >> might >> >>>>>>> just be contained multiple times for different Flink releases. >> >>>>>>> >> >>>>>>> Each module should provide a short doc (README) that shows how to >> use an >> >>>>>>> integrate it with Flink. Thus, the responsibility goes to the >> >>>>>>> contributor to maintain the library. If it breaks and is not >> maintained >> >>>>>>> any further, we can simple remove it. >> >>>>>>> >> >>>>>>> I agree, that the community might not be able to maintain those >> >>>>>>> extension/libraries right now. I would put the responsibility >> (more or >> >>>>>>> less completely) on the contributor and delete project that do not >> fix >> >>>>>>> any more. >> >>>>>>> >> >>>>>>> @Vasia: a link to a library could be included in the README. If >> anybody >> >>>>>>> only wants to share a library but not contribute code, the parent >> README >> >>>>>>> could contain a list of additional links. >> >>>>>>> >> >>>>>>> >> >>>>>>> -Matthias >> >>>>>>> >> >>>>>>> >> >>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote: >> >>>>>>>> How about, for now, we simply create a page where we gather >> links/short >> >>>>>>>> descriptions of all these contributions >> >>>>>>>> and let the maintenance and dependency management to the >> tool/library >> >>>>>>>> creators? >> >>>>>>>> This way we will at least have these contributions in one place >> and link to >> >>>>>>>> them somewhere from the website. >> >>>>>>>> >> >>>>>>>> -Vasia. >> >>>>>>>> >> >>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <m...@apache.org> >> wrote: >> >>>>>>>> >> >>>>>>>>> Hi Matthias, >> >>>>>>>>> >> >>>>>>>>> Thanks for bringing up this idea. Actually, it has been >> discussed a >> >>>>>>>>> couple of times on the mailing list whether we should have a >> central >> >>>>>>>>> place for third-party extensions/contributions/libraries. This >> could >> >>>>>>>>> either be something package-based or, like you proposed, another >> >>>>>>>>> repository. >> >>>>>>>>> >> >>>>>>>>> An external place for contributions raises a couple of questions >> >>>>>>>>> >> >>>>>>>>> - Which version should the external contributions be based on? >> >>>>>>>>> - How do we make sure, the extensions are continuously updated? >> >>>>>>>>> (dedicated maintainers or automatic compatibility checks) >> >>>>>>>>> - How do we easily plug-in the external modules into Flink? >> >>>>>>>>> >> >>>>>>>>> In the long term, we really need a solution for these questions. >> The >> >>>>>>>>> code base of Flink is growing and more and more packages go to >> >>>>>>>>> flink-contrib/flink-staging. I would find something >> packaged-based >> >>>>>>>>> better than a repository. Quite frankly, momentarily, I think >> >>>>>>>>> developing such a plugin system is out of scope for most Flink >> >>>>>>>>> developers. At the current pace of Flink development, collecting >> these >> >>>>>>>>> contributions externally without properly maintaining them, >> doesn't >> >>>>>>>>> make much sense to me. >> >>>>>>>>> >> >>>>>>>>> Cheers, >> >>>>>>>>> Max >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax < >> mj...@apache.org> wrote: >> >>>>>>>>>> >> >>>>>>>>>> Hi, >> >>>>>>>>>> >> >>>>>>>>>> many people are building quite exiting stuff on top of Flink. >> It is hard >> >>>>>>>>>> to keep an good overview on what stuff is available and what >> not. What >> >>>>>>>>>> do you think about starting a second git repository >> "flink-external" >> >>>>>>>>>> that collects all those code? >> >>>>>>>>>> >> >>>>>>>>>> The ideas would be to collect stuff in a central point, such >> that people >> >>>>>>>>>> can access it easily and get an overview what is already >> available (this >> >>>>>>>>>> might also avoid duplicate development). It might also be a >> good point >> >>>>>>>>>> to show common patterns. In order to collect as much as >> possible, the >> >>>>>>>>>> contributing requirement (with respect to testing etc) could be >> lower >> >>>>>>>>>> than for Flink itself. >> >>>>>>>>>> >> >>>>>>>>>> For example, I recently started a small flink-clojure module >> with a >> >>>>>>>>>> simple word-count example to answer a question on SO. Including >> this in >> >>>>>>>>>> Flink would not be appropriate. However, for a flink-external >> repro it >> >>>>>>>>>> might be nice to have. >> >>>>>>>>>> >> >>>>>>>>>> What do you think about it? >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> -Matthias >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> Regards, >> >>>>>> Chiwan Park >> >>>>>> >> >>>>>> >> >>>>>> >> >>>> >> >> >> >>
