Hello Reuven, I finally took the time to read the Drain proposal, thanks a lot for bringing this, it looks like a nice fit with the current APIs and it would be great if this could be implemented as much as possible in a Runner independent way.
I am eager now to see the snapshot and update proposal. Thanks again, Ismaël On Tue, Jun 6, 2017 at 10:03 PM, Reuven Lax <re...@google.com.invalid> wrote: > > I believe so, but it looks like the dispenser is an interface to the user > for such features. We still need to define what the semantics of features > like Drain are, and how they affect the pipeline. > > On Tue, Jun 6, 2017 at 12:06 PM, Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > Hi Reuven, > > > > In the "Apache Beam: Technical vision" document (dating from the > > incubation) (https://docs.google.com/document/d/1UyAeugHxZmVlQ5cEWo_eOPg > > XNQA1oD-rGooWOSwAqh8/edit?usp=sharing), I added a section named "Beam > > Pipelines Dispenser". > > > > The idea is to be able to bootstrap, run and control pipelines (and the > > runners). > > > > I think it's somehow related. WDYT ? > > > > Regards > > JB > > > > On 06/06/2017 07:43 PM, Reuven Lax wrote: > > > >> Hi all, > >> > >> Beam is a great programming mode, but in order to really run pipelines > >> (especially streaming pipelines which are "always on") in a production > >> setting, there is a set of features necessary. Dataflow has a couple of > >> those features built in (Drain and Update), and inspired by those I'll be > >> sending out a few proposals for similar features in Beam. > >> > >> Please note that my intention here is _not_ to simply forklift the > >> Dataflow > >> features to Beam. The Dataflow features are being used as inspiration, and > >> we have two years of experience how real users have used these feature > >> (and > >> also experienced when users have found these features limited and > >> frustrating). In every case my Beam proposals are different - hopefully > >> better! - than the actual Dataflow feature that exists today. > >> > >> I think all runners would greatly benefit from production-control features > >> like this, and I would love to see community input. The first proposal is > >> for a way of draining a streaming pipeline before stopping it, and here it > >> is > >> <https://docs.google.com/document/d/1NExwHlj-2q2WUGhSO4jTu8X > >> GhDPmm3cllSN8IMmWci8/edit> > >> . > >> > >> Reuven > >> > >> > > -- > > Jean-Baptiste Onofré > > jbono...@apache.org > > http://blog.nanthrax.net > > Talend - http://www.talend.com > >
