Hi Kamal, The network buffer will be full for specific `FileSource` when the job has back pressure which will block the source subtask. You can refer to network buffer [1] for more information.
[1] https://flink.apache.org/2019/06/05/a-deep-dive-into-flinks-network-stack/ Best, Shammon FY On Fri, May 26, 2023 at 7:13 PM Kamal Mittal <kspmm...@gmail.com> wrote: > Hello Shammon, > > Can you please point out the classes where like for "FileSource" slow down > logic is placed? > > Just wanted to understand it more better and try it at my end by running > various perf. runs, also apply changes in my application if any. > > Rgds, > Kamal > > On Thu, May 25, 2023 at 9:16 AM Kamal Mittal <kspmm...@gmail.com> wrote: > >> Hello Shammon, >> >> Can you please point out the classes where like for "FileSource" slow >> down logic is placed? >> >> Just wanted to understand it more better and try it at my end by running >> various perf. runs, also apply changes in my application if any. >> >> Rgds, >> Kamal >> >> On Wed, May 24, 2023 at 11:41 AM Kamal Mittal <kspmm...@gmail.com> wrote: >> >>> Thanks Shammon for clarification. >>> >>> On Wed, May 24, 2023 at 11:01 AM Shammon FY <zjur...@gmail.com> wrote: >>> >>>> Hi Kamal, >>>> >>>> The source will slow down when there is backpressure in the flink job, >>>> you can refer to docs [1] and [2] to get more detailed information about >>>> backpressure mechanism. >>>> >>>> Currently there's no API or Callback in source for users to do some >>>> customized operations for backpressure, but users can collect the metrics >>>> of the job and analysis, for example, the metrics in [1] and [3]. I hope >>>> this can help you. >>>> >>>> [1] >>>> https://flink.apache.org/2021/07/07/how-to-identify-the-source-of-backpressure/#:~:text=Backpressure%20is%20an%20indicator%20that,the%20queues%20before%20being%20processed >>>> . >>>> [2] >>>> https://www.alibabacloud.com/blog/analysis-of-network-flow-control-and-back-pressure-flink-advanced-tutorials_596632 >>>> [3] >>>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/monitoring/back_pressure/ >>>> >>>> On Tue, May 23, 2023 at 9:40 PM Kamal Mittal <kspmm...@gmail.com> >>>> wrote: >>>> >>>>> Hello Community, >>>>> >>>>> Can you please share views about the query asked above w.r.t back >>>>> pressure for FileSource APIs for Bulk and Record stream formats. >>>>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa >>>>> conversion. >>>>> >>>>> Rgds, >>>>> Kamal >>>>> >>>>> On Tue, 23 May 2023, 12:26 pm Kamal Mittal, <kspmm...@gmail.com> >>>>> wrote: >>>>> >>>>>> Added Flink community DL as well. >>>>>> >>>>>> ---------- Forwarded message --------- >>>>>> From: Kamal Mittal <kspmm...@gmail.com> >>>>>> Date: Tue, May 23, 2023 at 7:57 AM >>>>>> Subject: Re: Backpressure handling in FileSource APIs - Flink 1.16 >>>>>> To: Shammon FY <zjur...@gmail.com> >>>>>> >>>>>> >>>>>> Hello, >>>>>> >>>>>> Yes, want to take some custom actions and also if there is any >>>>>> default behavior of slowing down sending data in pipeline further or >>>>>> reading data from source somehow? >>>>>> >>>>>> Rgds, >>>>>> Kamal >>>>>> >>>>>> On Tue, May 23, 2023 at 6:06 AM Shammon FY <zjur...@gmail.com> wrote: >>>>>> >>>>>>> Hi Kamal, >>>>>>> >>>>>>> If I understand correctly, do you want the source to do some custom >>>>>>> actions, such as current limiting, when there is backpressure in the >>>>>>> job? >>>>>>> >>>>>>> Best, >>>>>>> Shammon FY >>>>>>> >>>>>>> >>>>>>> On Mon, May 22, 2023 at 2:12 PM Kamal Mittal <kspmm...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hello Community, >>>>>>>> >>>>>>>> Can you please share views about the query asked above w.r.t back >>>>>>>> pressure for FileSource APIs for Bulk and Record stream formats. >>>>>>>> Planning to use these APIs w.r.t AVRO to Parquet and vice-versa >>>>>>>> conversion. >>>>>>>> >>>>>>>> Rgds, >>>>>>>> Kamal >>>>>>>> >>>>>>>> On Thu, May 18, 2023 at 2:33 PM Kamal Mittal <kspmm...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hello Community, >>>>>>>>> >>>>>>>>> Does FileSource APIs for Bulk and Record stream formats handle >>>>>>>>> back pressure by any way like slowing down sending data in piepline >>>>>>>>> further >>>>>>>>> or reading data from source somehow? >>>>>>>>> Or does it give any callback/handle so that any action can be >>>>>>>>> taken? Can you please share details if any? >>>>>>>>> >>>>>>>>> Rgds, >>>>>>>>> Kamal >>>>>>>>> >>>>>>>>
