My user wishes - whatever version, it is just a number after all ;): - make coder usage simpler and consistent (PCollection TypeDescriptor and Coder are duplicated in term of API) - have a beam api (split from the sdk and internals and impl) - have SDF supported by runners - have a SDFRunner allowing to simulate the SDF lifecycle manually (same for DoFn short term - see next point for the current issue) - ensure classloader usage is consistent, ie any proxy is created into the final artifact classloader (transform if custom, dofn/source/sdf otherwise) - have a test compatibility kit (TCK) for runner. It would be a jar any runner impl can import to run with surefire - make IO configuration reflection friendly (get rid of the autovalue pattern which is not industriablizable and allow pojo like classes or alternatively support reading the conf from properties) - support pipeline implicit option based on transform names to override some attributes - change runner implementations to let the bundle size have a pipeline option defining an upper bound and not hardcode them arbitrarly - defaults can stay the current ones - better multi input/output support (just PCollection based and fully wireable) - a smoother pipeline API would be nice. I like hazelcast jet one for instance
Le 29 nov. 2017 03:29, "Robert Bradshaw" <rober...@google.com> a écrit : > On Tue, Nov 28, 2017 at 9:48 AM, Reuven Lax <re...@google.com> wrote: > > > > On Tue, Nov 28, 2017 at 9:14 AM, Jean-Baptiste Onofré <j...@nanthrax.net> > > wrote: > >> > >> Hi Reuven, > >> > >> Yes, I remember that we agreed on a release per month. However, we > didn't > >> do it before. I think the most important is not the period, it's more a > >> stable pace. I think it's more interesting for our community to have > >> "always" a release every two months, more than a tentative of a release > >> every month that end later than that. Of course, if we can do both, it's > >> perfect ;) > > > > Agree. A stable pace is the most important thing. > > +1, and I think everyone who's done a release is in favor of making it > easier and more frequent. Someone should put together a proposal of > easy things we can do to automate, etc. > > >> For Beam 3.x, I wasn't talking about breaking change, but more about > >> "marketing" announcement. I think that, even if we don't break API, some > >> features are "strong enough" to be "qualified" in a major version. > > > > Ah, good point. This doesn't stop us from checking in these new features > > into 2.x possibly tagged with an @Experimental flag. We can then use 3.0 > to > > announce all these features more broadly, and remove @Experimental tags. > > > > I would also like to see enterprise-ready BeamSQL and Java 7 deprecation > on > > the list for Beam 3.0 > > > >> > >> I think that any major idea & feature (breaking or not the API) are > >> valuables for Beam 3.x (and it's a good sign for our community again > ;)). > > I'm generally not a fan of bumping the major version number just > because enough time has passed, or enough new features have gone in > (and am mostly opposed to holding features back just because we want > to announce them (simultanously?) in a big release)--instead I find > that the need for a new major version arises out of a realization that > the model has sufficiently changed and we need to cut ties with the > old way of doing things (that's perhaps holding us back). That being > said, it could be that some of these features are large enough to > merit this. > > Regardless of the naming, I think it's a great time to have a > discussion of where we want to go in 2018. > > Top of my list is first class support for Schema'd PCollections (and > with it SQL support, etc.) and full support of the portability > framework realizing the possibility of every runner running every SDK > (and, ideally, even cross-SDK/language pipelines). I would also like > to see explorations into interactive/incremental (for Python at least, > but probably Java as well). > > - Robert > > > >> On 11/28/2017 06:09 PM, Reuven Lax wrote: > >>> > >>> > >>> > >>> On Tue, Nov 28, 2017 at 8:55 AM, Jean-Baptiste Onofré <j...@nanthrax.net > >>> <mailto:j...@nanthrax.net>> wrote: > >>> > >>> Hi guys, > >>> > >>> Even if there's no rush, I think it would be great for the > community > >>> to have > >>> a better view on our roadmap and where we are going in term of > >>> schedule. > >>> > >>> I would like to discuss the following: > >>> - a best effort to maintain a good release pace or at least > provide a > >>> rough > >>> schedule. For instance, in Apache Karaf, I have a release schedule > >>> (http://karaf.apache.org/download.html#container-schedule > >>> <http://karaf.apache.org/download.html#container-schedule>). I > think > >>> a > >>> release ~ every quarter would be great. > >>> > >>> > >>> Originally we had stated that we wanted monthly releases of Beam. So > far > >>> the releases have been painful enough that monthly hasn't happened. I > think > >>> we should address these issues and go to monthly releases as originally > >>> stated. > >>> > >>> - if I see new Beam 2.x releases for sure (according to the > previous > >>> point), > >>> it would be great to have discussion about Beam 3.x. I think that > one > >>> of > >>> interesting new feature that Beam 3.x can provide is around > >>> PCollection with > >>> Schemas. It's something that we started to discuss with Reuven and > >>> Eugene. > >>> In term of schedule, > >>> > >>> > >>> I don't think schemas require Beam 3.0 - I think we can introduce them > >>> without making breaking changes. However there are many other features > that > >>> would be very interesting for Beam 3.x, and we should start putting > together > >>> a list of them. I > >>> > >>> > >>> I would love to see your thoughts & ideas about releases schedule > and > >>> Beam 3.x. > >>> > >>> Regards > >>> JB > >>> -- Jean-Baptiste Onofré > >>> jbono...@apache.org <mailto:jbono...@apache.org> > >>> http://blog.nanthrax.net > >>> Talend - http://www.talend.com > >>> > >>> > >> > >> -- > >> Jean-Baptiste Onofré > >> jbono...@apache.org > >> http://blog.nanthrax.net > >> Talend - http://www.talend.com > > > > >
