it will offer the most
> > >> stable and predictable performance with the lowest checkpoint latency.
> > It’s
> > >> not perfect, it won’t solve all of the use cases, but frankly all of
> the
> > >> other options have their own blind spots, and continuou
ling pre-emptively/continuously if some
> buffer
> >> was not processed within X seconds.
> >> 2. Start spilling only once the checkpoint starts (this is the exact
> >> proposal from the current FLIP-76).
> >> 3. Initially we want to spill to a Flink’s FileSystem (for example S3),
link's job data processing rate is 100MB/s, spilling
>> in-flight data will take 3.3 times longer than waiting for the alignment.
>> On the other hand if data processing rate is 10MB/s, overhead of continuous
>> spilling is relatively low.
>> [2] With checkpoints every one minute, with data processing throu
ginal proposal
> and continuous spilling would have to effectively persist all of the data
> anyway.
>
> > On 10 Oct 2019, at 19:51, Yun Tang wrote:
> >
> > Hi Arvid
> >
> > +1 for this future which has been hoped for a long time. End-to-end
ignment.
> > On the other hand if data processing rate is 10MB/s, overhead of
> continuous
> > spilling is relatively low.
> > [2] With checkpoints every one minute, with data processing throughput
> > 30MB/s per node, we would have to persist 1.8GB of data per node betwee
; >
> > Hi Arvid
> >
> > +1 for this future which has been hoped for a long time. End-to-end
> exactly once job could benefit from quicker checkpoint completion.
> >
> >
> > Best
> > Yun Tang
> >
> > From: Y
ld have to effectively persist all of the data
> anyway.
>
> > On 10 Oct 2019, at 19:51, Yun Tang wrote:
> >
> > Hi Arvid
> >
> > +1 for this future which has been hoped for a long time. End-to-end
> exactly once job could benefit from quicker checkpoint comple
>
> Best
> Yun Tang
>
> From: Yun Gao
> Sent: Thursday, October 10, 2019 18:39
> To: dev
> Subject: Re: [DISCUSS] FLIP-76: Unaligned checkpoints
>
>Hi Arvid,
>
>Very thanks for bring up the discussion! From our side unable to
> finish
Hi Arvid
+1 for this future which has been hoped for a long time. End-to-end exactly
once job could benefit from quicker checkpoint completion.
Best
Yun Tang
From: Yun Gao
Sent: Thursday, October 10, 2019 18:39
To: dev
Subject: Re: [DISCUSS] FLIP-76
Hi Arvid,
Very thanks for bring up the discussion! From our side unable to
finish the checkpoint is commonly met for online jobs, therefore +1 from my
side to implement this.
A tiny issue of the FLIP is that the Discussion Thread URL attached
seems to be not right.
Thanks for the FLIP, Arvid.
This is a good improvement for checkpoint under backpressure. Currently, if
a job under backpressure, it almost can't complete the checkpoint. so +1
from my side.
Best,
Congxian
zhijiang 于2019年10月10日周四 上午11:02写道：
> Thanks for writing up this FLIP, Arvid!
>
> Many
Thanks for writing up this FLIP, Arvid!
Many users would expect this feature and also +1 from my side.
Best,
Zhijiang
--
From:Piotr Nowojski
Send Time:2019年10月7日(星期一) 10:13
To:dev
Subject:Re: [DISCUSS] FLIP-76: Unaligned
Hi Arvid,
Thanks for coming up with this FLIP. I think it addresses the issues raised in
the previous mailing list discussion [2].
For the record: +1 from my side to implement this.
Piotrek
> On 30 Sep 2019, at 14:31, Arvid Heise wrote:
>
> Hi Devs,
>
> I would like to start the formal
13 matches
Mail list logo
