On Fri, Aug 30, 2019 at 6:52 PM Chad Dombrova <chad...@gmail.com> wrote:
> +dev > > I read the document on Drain, and it sounds very promising. I have a few > questions, starting with this statement from the doc: > > "This document proposes a new pipeline action called Drain. Drain can > be implemented by runners by manipulating the watermark of the pipeline." > > What is a pipeline "action" and how would this be exposed to the user? I > assume this is externally and manually initiated. Is this something that > would be invoked from a PipelineResult object (i.e. akin to "cancel")? > The term is alluding to job management (run, cancel, get metrics, ...) and yes this should be exposed as part of PipelineResult in addition to any tooling that performs job management. > > Once a Drain is initiated on a pipeline, does this trigger a loop over all > unbounded sources to set their watermark to infinity, or only certain > ones? > Conceptually yes, the runner sets the watermark to infinity for all "sources" preventing new data from being produced. The watermark then progresses downstream throughout the execution graph causing windows to close, timers to fire, state to be emit and then garbage collected and so forth. > > thanks, > -chad > > > On Fri, Aug 16, 2019 at 2:47 PM Jose Delgado <jdelg...@bandainamcoent.com> > wrote: > >> I see, thank you Lukasz. >> >> >> >> Regards, >> Jose >> >> *From: *Lukasz Cwik <lc...@google.com> >> *Reply-To: *"user@beam.apache.org" <user@beam.apache.org> >> *Date: *Monday, August 5, 2019 at 11:11 AM >> *To: *user <user@beam.apache.org> >> *Subject: *Re: Save state on tear down >> >> >> >> This is not possible today. >> >> >> >> There have been discussions about pipeline drain, snapshot and update [1, >> 2] which may provide additional details of what is planned and could use >> your feedback. >> >> >> >> 1: >> https://docs.google.com/document/d/1NExwHlj-2q2WUGhSO4jTu8XGhDPmm3cllSN8IMmWci8 >> >> 2: >> https://docs.google.com/document/d/1UWhnYPgui0gUYOsuGcCjLuoOUlGA4QaY91n8p3wz9MY >> >> >> >> On Thu, Aug 1, 2019 at 3:44 PM Jose Delgado <jdelg...@bandainamcoent.com> >> wrote: >> >> Hello All, >> >> >> >> I wondering if there is a way or pattern to save state on tear down ? >> >> >> >> In case of a failure or a pipeline we cannot update(due to significant >> changes ) we would like to save the state and re-load it on the next >> creation of the pipeline. >> >> >> >> Note: we are currently using Google Dataflow runner >> >> >> >> Regards, >> >> Jose >> >>