Ps: forgot another wish: make usable beam sql. Today you need to add a fn before and after cause of that type breakage not consistent with the pipeline API. It would be nice to support pojo (extracted from the select fields or created from "views" like in jackson) bit not having to wrap the sql usage in multiple UDF would make it powerful and ready to use.
Le 29 nov. 2017 07:01, "Romain Manni-Bucau" <[email protected]> a écrit : > My user wishes - whatever version, it is just a number after all ;): > > - make coder usage simpler and consistent (PCollection TypeDescriptor and > Coder are duplicated in term of API) > - have a beam api (split from the sdk and internals and impl) > - have SDF supported by runners > - have a SDFRunner allowing to simulate the SDF lifecycle manually (same > for DoFn short term - see next point for the current issue) > - ensure classloader usage is consistent, ie any proxy is created into the > final artifact classloader (transform if custom, dofn/source/sdf otherwise) > - have a test compatibility kit (TCK) for runner. It would be a jar any > runner impl can import to run with surefire > - make IO configuration reflection friendly (get rid of the autovalue > pattern which is not industriablizable and allow pojo like classes or > alternatively support reading the conf from properties) > - support pipeline implicit option based on transform names to override > some attributes > - change runner implementations to let the bundle size have a pipeline > option defining an upper bound and not hardcode them arbitrarly - defaults > can stay the current ones > - better multi input/output support (just PCollection based and fully > wireable) > - a smoother pipeline API would be nice. I like hazelcast jet one for > instance > > Le 29 nov. 2017 03:29, "Robert Bradshaw" <[email protected]> a écrit : > >> On Tue, Nov 28, 2017 at 9:48 AM, Reuven Lax <[email protected]> wrote: >> > >> > On Tue, Nov 28, 2017 at 9:14 AM, Jean-Baptiste Onofré <[email protected]> >> > wrote: >> >> >> >> Hi Reuven, >> >> >> >> Yes, I remember that we agreed on a release per month. However, we >> didn't >> >> do it before. I think the most important is not the period, it's more a >> >> stable pace. I think it's more interesting for our community to have >> >> "always" a release every two months, more than a tentative of a release >> >> every month that end later than that. Of course, if we can do both, >> it's >> >> perfect ;) >> > >> > Agree. A stable pace is the most important thing. >> >> +1, and I think everyone who's done a release is in favor of making it >> easier and more frequent. Someone should put together a proposal of >> easy things we can do to automate, etc. >> >> >> For Beam 3.x, I wasn't talking about breaking change, but more about >> >> "marketing" announcement. I think that, even if we don't break API, >> some >> >> features are "strong enough" to be "qualified" in a major version. >> > >> > Ah, good point. This doesn't stop us from checking in these new features >> > into 2.x possibly tagged with an @Experimental flag. We can then use >> 3.0 to >> > announce all these features more broadly, and remove @Experimental tags. >> > >> > I would also like to see enterprise-ready BeamSQL and Java 7 >> deprecation on >> > the list for Beam 3.0 >> > >> >> >> >> I think that any major idea & feature (breaking or not the API) are >> >> valuables for Beam 3.x (and it's a good sign for our community again >> ;)). >> >> I'm generally not a fan of bumping the major version number just >> because enough time has passed, or enough new features have gone in >> (and am mostly opposed to holding features back just because we want >> to announce them (simultanously?) in a big release)--instead I find >> that the need for a new major version arises out of a realization that >> the model has sufficiently changed and we need to cut ties with the >> old way of doing things (that's perhaps holding us back). That being >> said, it could be that some of these features are large enough to >> merit this. >> >> Regardless of the naming, I think it's a great time to have a >> discussion of where we want to go in 2018. >> >> Top of my list is first class support for Schema'd PCollections (and >> with it SQL support, etc.) and full support of the portability >> framework realizing the possibility of every runner running every SDK >> (and, ideally, even cross-SDK/language pipelines). I would also like >> to see explorations into interactive/incremental (for Python at least, >> but probably Java as well). >> >> - Robert >> >> >> >> On 11/28/2017 06:09 PM, Reuven Lax wrote: >> >>> >> >>> >> >>> >> >>> On Tue, Nov 28, 2017 at 8:55 AM, Jean-Baptiste Onofré < >> [email protected] >> >>> <mailto:[email protected]>> wrote: >> >>> >> >>> Hi guys, >> >>> >> >>> Even if there's no rush, I think it would be great for the >> community >> >>> to have >> >>> a better view on our roadmap and where we are going in term of >> >>> schedule. >> >>> >> >>> I would like to discuss the following: >> >>> - a best effort to maintain a good release pace or at least >> provide a >> >>> rough >> >>> schedule. For instance, in Apache Karaf, I have a release schedule >> >>> (http://karaf.apache.org/download.html#container-schedule >> >>> <http://karaf.apache.org/download.html#container-schedule>). I >> think >> >>> a >> >>> release ~ every quarter would be great. >> >>> >> >>> >> >>> Originally we had stated that we wanted monthly releases of Beam. So >> far >> >>> the releases have been painful enough that monthly hasn't happened. I >> think >> >>> we should address these issues and go to monthly releases as >> originally >> >>> stated. >> >>> >> >>> - if I see new Beam 2.x releases for sure (according to the >> previous >> >>> point), >> >>> it would be great to have discussion about Beam 3.x. I think that >> one >> >>> of >> >>> interesting new feature that Beam 3.x can provide is around >> >>> PCollection with >> >>> Schemas. It's something that we started to discuss with Reuven and >> >>> Eugene. >> >>> In term of schedule, >> >>> >> >>> >> >>> I don't think schemas require Beam 3.0 - I think we can introduce them >> >>> without making breaking changes. However there are many other >> features that >> >>> would be very interesting for Beam 3.x, and we should start putting >> together >> >>> a list of them. I >> >>> >> >>> >> >>> I would love to see your thoughts & ideas about releases schedule >> and >> >>> Beam 3.x. >> >>> >> >>> Regards >> >>> JB >> >>> -- Jean-Baptiste Onofré >> >>> [email protected] <mailto:[email protected]> >> >>> http://blog.nanthrax.net >> >>> Talend - http://www.talend.com >> >>> >> >>> >> >> >> >> -- >> >> Jean-Baptiste Onofré >> >> [email protected] >> >> http://blog.nanthrax.net >> >> Talend - http://www.talend.com >> > >> > >> >
