In terms of features, I think a key thing we should focus on is making simple things simple. Beam is very powerful, but it doesn't always make easy things easy. Features like schema'd PCollections could go a long way here. Also fully fleshing out/smoothing our runner portability story is part of this too.
For beam 3.x we could also reason about if there's any complexity that doesn't hold its weight (e.g. side inputs on CombineFns). On Mon, Jan 22, 2018 at 9:20 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Ben, > > about the "technical roadmap", we have a thread about "Beam 3.x roadmap". > > It already provides ideas for points 3 & 4. > > Regards > JB > > On 01/22/2018 09:15 PM, Ben Chambers wrote: >> Thanks Davor for starting the state of the project discussions [1]. >> >> >> In this fork of the state of the project discussion, I’d like to start the >> discussion of the feature roadmap for 2018 (and beyond). >> >> >> To kick off the discussion, I think the features could be divided into >> several >> areas, as follows: >> >> 1. >> >> Enabling Contributions: How do we make it easier to add new features to >> the >> supported runners? Can we provide a common intermediate layer below the >> existing functionality that features are translated to so that runners >> only >> need to support the intermediate layer and new features only need to >> target >> it? What other ways can we make it easier to contribute to the >> development >> of Beam? >> >> 2. >> >> Realizing Portability: What gaps are there in the promise of portability? >> For example in [1] we discussed the fact that users must write per-runner >> code to push system metrics from runners to their monitoring platform. >> This >> limits their ability to actually change runners. Credential management >> for >> different environments also falls into this category. >> >> 3. >> >> Large Features: What major features (like Beam SQL, Beam Python, etc.) >> would >> increase the Beam user base in 2018? >> >> 4. >> >> Improvements: What small changes could make Beam more appealing to users? >> Are there API improvements we could make or common mistakes we could >> detect >> and/or prevent? >> >> >> Thanks in advance for participating in the discussion. I believe that 2018 >> could >> be a great year for Beam, providing easier, more complete runner portability >> and >> features that make Beam easier to use for everyone. >> >> >> Ben >> >> >> [1] >> https://lists.apache.org/thread.html/f750f288af8dab3f468b869bf5a3f473094f4764db419567f33805d0@%3Cdev.beam.apache.org%3E >> >> [2] >> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com
