BackPressure handling

I did a simulation on session windows ( in 2 modes ) and let it rip for about 12 hours 1. Replay where a kafka topic with retention of 7 days was the source ( earliest ) 2. Start the pipe with kafka source ( latest ) I saw results that differed dramatically. On replay the pipeline stalled after

Re: BackPressure handling

Hi Vishal, your assumptions sound reasonable to me. The community is currently working on a more fine-grained back pressuring with credit-based flow control. It is on the roamap for 1.5 [1]/[2]. I will loop in Nico that might tell you more about the details. Until then I guess you have to imp

Re: BackPressure handling

Could you please point me to any documentation on the "credit-based flow control" approach On Tue, Jan 2, 2018 at 10:35 AM, Timo Walther wrote: > Hi Vishal, > > your assumptions sound reasonable to me. The community is currently > working on a more fine-grained back pressuring with credit-b

Re: BackPressure handling

Hi Vishal, let me already point you towards the JIRA issue for the credit-based flow control: https://issues.apache.org/jira/browse/FLINK-7282 I'll have a look at the rest of this email thread tomorrow... Regards, Nico On 02/01/18 17:52, Vishal Santoshi wrote: > Could you please point me to any

Re: BackPressure handling

Thank you. On Tue, Jan 2, 2018 at 1:31 PM, Nico Kruber wrote: > Hi Vishal, > let me already point you towards the JIRA issue for the credit-based > flow control: https://issues.apache.org/jira/browse/FLINK-7282 > > I'll have a look at the rest of this email thread tomorrow... > > > Regards, > Ni

Re: BackPressure handling

Also note that if I were to start 2 pipelines 1. Working off the head of the topic and thus not prone to the pathological case described above 2. Doing a replay and thus prone to the pathological case described above Than the 2nd pipe will stall the 1st pipeline. This seems to to point to -

Re: BackPressure handling

To add and an interim solution to the issue. I extended the based on the advise "custom source/adapt an existing source" and put in a RateLimiter ( guava ) that effectively put a cap on each kafka consumer ( x times the expected incident rqs ). That solved the issue as in it stabilized the flow

Re: BackPressure handling

Hi, I think your analysis is very thorough and accurate. However, I don't think that https://issues.apache.org/jira/browse/FLINK-7282 will solve this problem. We're dealing with "time back-pressure" here and not traditional processing back-pre

Re: BackPressure handling

Absolutely. But without a a view into a global WM at the source level, the --> would require sources to wait for sources that are "slower in time" --> is not possible for folks creating custom sources on extending existing ones. I would have loved to use a more scientific/data drive approa

Backpressure handling in FileSource APIs - Flink 1.16

Hello Community, Does FileSource APIs for Bulk and Record stream formats handle back pressure by any way like slowing down sending data in piepline further or reading data from source somehow? Or does it give any callback/handle so that any action can be taken? Can you please share details if any?

Re: Backpressure handling in FileSource APIs - Flink 1.16

Hello Community, Can you please share views about the query asked above w.r.t back pressure for FileSource APIs for Bulk and Record stream formats. Planning to use these APIs w.r.t AVRO to Parquet and vice-versa conversion. Rgds, Kamal On Thu, May 18, 2023 at 2:33 PM Kamal Mittal wrote: > Hel

Re: Backpressure handling in FileSource APIs - Flink 1.16

Hi Kamal, If I understand correctly, do you want the source to do some custom actions, such as current limiting, when there is backpressure in the job? Best, Shammon FY On Mon, May 22, 2023 at 2:12 PM Kamal Mittal wrote: > Hello Community, > > Can you please share views about the query asked

Fwd: Backpressure handling in FileSource APIs - Flink 1.16

Added Flink community DL as well. -- Forwarded message - From: Kamal Mittal Date: Tue, May 23, 2023 at 7:57 AM Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16 To: Shammon FY Hello, Yes, want to take some custom actions and also if there is any default

Re: Backpressure handling in FileSource APIs - Flink 1.16

: > Added Flink community DL as well. > > -- Forwarded message - > From: Kamal Mittal > Date: Tue, May 23, 2023 at 7:57 AM > Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16 > To: Shammon FY > > > Hello, > > Yes, want to take some cus

Re: Backpressure handling in FileSource APIs - Flink 1.16

wrote: > >> Added Flink community DL as well. >> >> -- Forwarded message - >> From: Kamal Mittal >> Date: Tue, May 23, 2023 at 7:57 AM >> Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16 >> To: Shammon FY >> >

Re: Backpressure handling in FileSource APIs - Flink 1.16

APIs for Bulk and Record stream formats. >> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa >> conversion. >> >> Rgds, >> Kamal >> >> On Tue, 23 May 2023, 12:26 pm Kamal Mittal, wrote: >> >>> Added Flink community DL as well.

Re: Backpressure handling in FileSource APIs - Flink 1.16

>>> Rgds, >>> Kamal >>> >>> On Tue, 23 May 2023, 12:26 pm Kamal Mittal, wrote: >>> >>>> Added Flink community DL as well. >>>> >>>> -- Forwarded message - >>>> From: Kamal Mittal >&

Re: Backpressure handling in FileSource APIs - Flink 1.16

asked above w.r.t back >>>> pressure for FileSource APIs for Bulk and Record stream formats. >>>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa >>>> conversion. >>>> >>>> Rgds, >>>> Kamal >>>> >>>>

Re: Backpressure handling in FileSource APIs - Flink 1.16

/ops/monitoring/back_pressure/ >>>> >>>> On Tue, May 23, 2023 at 9:40 PM Kamal Mittal >>>> wrote: >>>> >>>>> Hello Community, >>>>> >>>>> Can you please share views about the query asked above w.r.t back &g