I agree that we should keep all runners in Matrix table but it should be clear which runners are part of Beam code base, which are not. In case of non-Beam code repo runners we need to provide a link to its documentation.
MapReduce runner is another case since it’s not yet production ready and it lives in separate branch. We need to mention this clearly as well, it took me a while to find out where actually this runner is. > On 21 Sep 2018, at 16:06, Maximilian Michels <[email protected]> wrote: > > > not sure I fully follow you there. > > @JB I simply meant to ask whether it make sense to have Runners in the matrix > whose code/documentation is not part of Beam. For it to become a part of > Beam, it could be as easy as adding a link to the external Runner page. > >> I don't know that we need to limit the matrix to runners in the Beam codebase > > @Robert I think we used to only allow Runners in the matrix which were in the > Beam code base. However, you are right, it is not necessary for Runners to > live in the Beam repo. But IMHO they should be documented and linked before > entries to the matrix are made. > >> +1 but perhaps we should having a table listing Runners under development >> like we do for IOs. > > @Tim Yes, I didn't even know the Hadoop Runner was in a branch. > > > I don't want to remove any Runners from the matrix but I propose to require > some form of documentation on the Beam website in addition to the > compatibility matrix entry. > > The current state: > > Runners documented > ================== > > Direct Runner > Apache Apex > Apache Flink > Apache Gearpump > Apache Samza > Apache Spark > Google Cloud Dataflow > > Runners according to the matrix > =============================== > > Apache Apex > Apache Flink > Apache Gearpump > Apache Samza > Apache Spark > Google Cloud Dataflow > Apache Hadoop MapReduce > JStorm > IBM Streams > > > If we can fix the diff between these two lists, I'd feel more comfortable the > next time somebody asks about a Runner I haven't used yet :) > > Thanks, > Max > > On 21.09.18 14:51, Thomas Weise wrote: >> The MapReduce runner IMHO should not be in the matrix. >> For the external runners, is there any public documentation available that >> explains how they can be used and how they are supported? >> Thanks, >> Thomas >> On Fri, Sep 21, 2018 at 5:14 AM Jean-Baptiste Onofré <[email protected] >> <mailto:[email protected]>> wrote: >> You are right Tim. >> M/R runner is on a branch (in stale for now to be honest ;)). >> I think I got Max's remark. >> So, agree to focus only Beam coverage in the runner compatibility >> matrix. However, it's also important for the community to have some >> insights about runners generally speaking. >> Regards >> JB >> On 21/09/2018 14:00, Tim Robertson wrote: >> > "what do you think about limiting the matrix to Runners in the >> Beam code >> > base" >> > >> > +1 but perhaps we should having a table listing Runners under >> > development like we do for IOs. >> > >> > As a concrete example we have MapReduce listed in the matrix [1], >> a page >> > documenting it [2] stating it is in Beam 2.6.0 but unless I'm >> mistaken >> > the code exists only on a branch [3] and hasn't been touched for >> a while. >> > >> > Thanks, >> > Tim >> > >> > [1] https://beam.apache.org/documentation/runners/capability-matrix/ >> > [2] https://beam.apache.org/documentation/runners/mapreduce/ >> > [3] https://github.com/apache/beam/tree/mr-runner >> > >> > On Fri, Sep 21, 2018 at 1:37 PM Jean-Baptiste Onofré >> <[email protected] <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>>> wrote: >> > >> > Hi Max, >> > >> > not sure I fully follow you there. You mean that we would >> have kind of >> > compability matrix on dedicated page of each runner ? >> > >> > Regards >> > JB >> > >> > On 21/09/2018 10:57, Maximilian Michels wrote: >> > > Hi Beamers, >> > > >> > > There have been occasions where people asked me about >> Runner XY and I >> > > had to find out that it only exists in the compatibility >> matrix, >> > but not >> > > as part of our code base. More interestingly, I couldn't >> even find its >> > > code or documentation via my favorite search engine. >> > > >> > > This seems to be the case for multiple Runners in the matrix. >> > > >> > > The compatibility matrix will need an overhaul anyways with the >> > > portability changes, but what do you think about limiting the >> > matrix to >> > > Runners in the Beam code base? >> > > >> > > Thanks, >> > > Max >> > >> > -- >> > Jean-Baptiste Onofré >> > [email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> > http://blog.nanthrax.net >> > Talend - http://www.talend.com >> > >> -- Jean-Baptiste Onofré >> [email protected] <mailto:[email protected]> >> http://blog.nanthrax.net >> Talend - http://www.talend.com
