1) Instead of enabling it easier to write features I think more users would care about being able to move their pipeline between different runners and one of the key missing features is dynamic work rebalancing in all runners (except Dataflow). Also, portability is meant to help make a crisp line between what are the responsibilities of the Runner and the SDK which would help make it easier to write features in an SDK and to support features in Runners.
2) To realize portability there are a lot of JIRAs being tracked under the portability label[1] that need addressing to be able to run an existing pipeline in a portable manner before we even get to more advanced features. 1: https://issues.apache.org/jira/browse/BEAM-3515?jql=project%20%3D%20BEAM%20AND%20labels%20%3D%20portability 3) Ben, do you want to design and run a couple of polls (similar to the Java 8 poll) to get feedback from our users based upon the list of major features being developed? 4) Yes, plenty. It would be worthwhile to have someone walk through the open JIRAs and mark them with a label and also summarize what groups they fall under as there are plenty of good ideas there. On Tue, Jan 23, 2018 at 5:25 PM, Robert Bradshaw <rober...@google.com> wrote: > In terms of features, I think a key thing we should focus on is making > simple things simple. Beam is very powerful, but it doesn't always > make easy things easy. Features like schema'd PCollections could go a > long way here. Also fully fleshing out/smoothing our runner > portability story is part of this too. > > For beam 3.x we could also reason about if there's any complexity that > doesn't hold its weight (e.g. side inputs on CombineFns). > > On Mon, Jan 22, 2018 at 9:20 PM, Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > Hi Ben, > > > > about the "technical roadmap", we have a thread about "Beam 3.x roadmap". > > > > It already provides ideas for points 3 & 4. > > > > Regards > > JB > > > > On 01/22/2018 09:15 PM, Ben Chambers wrote: > >> Thanks Davor for starting the state of the project discussions [1]. > >> > >> > >> In this fork of the state of the project discussion, I’d like to start > the > >> discussion of the feature roadmap for 2018 (and beyond). > >> > >> > >> To kick off the discussion, I think the features could be divided into > several > >> areas, as follows: > >> > >> 1. > >> > >> Enabling Contributions: How do we make it easier to add new > features to the > >> supported runners? Can we provide a common intermediate layer below > the > >> existing functionality that features are translated to so that > runners only > >> need to support the intermediate layer and new features only need > to target > >> it? What other ways can we make it easier to contribute to the > development > >> of Beam? > >> > >> 2. > >> > >> Realizing Portability: What gaps are there in the promise of > portability? > >> For example in [1] we discussed the fact that users must write > per-runner > >> code to push system metrics from runners to their monitoring > platform. This > >> limits their ability to actually change runners. Credential > management for > >> different environments also falls into this category. > >> > >> 3. > >> > >> Large Features: What major features (like Beam SQL, Beam Python, > etc.) would > >> increase the Beam user base in 2018? > >> > >> 4. > >> > >> Improvements: What small changes could make Beam more appealing to > users? > >> Are there API improvements we could make or common mistakes we > could detect > >> and/or prevent? > >> > >> > >> Thanks in advance for participating in the discussion. I believe that > 2018 could > >> be a great year for Beam, providing easier, more complete runner > portability and > >> features that make Beam easier to use for everyone. > >> > >> > >> Ben > >> > >> > >> [1] > >> https://lists.apache.org/thread.html/f750f288af8dab3f468b869bf5a3f4 > 73094f4764db419567f33805d0@%3Cdev.beam.apache.org%3E > >> > >> [2] > >> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda > 900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E > > > > -- > > Jean-Baptiste Onofré > > jbono...@apache.org > > http://blog.nanthrax.net > > Talend - http://www.talend.com >
