Lei Please do not post to both users and dev list. Dropping dev.
Yes these scenarios are based on limitations of a given processor implementation or the nature of a given protocol/mechanism. NiFi enforces back pressure by slowing/stopping scheduling a component and making fact of back pressure available to components. I dont know about this particular processor but it likely needs to be updated so the max amount of data it will return in any one cycle is small enough or controllable enough to avoud what youre seeing. Thanks Joe On Wed, Oct 16, 2019 at 3:06 AM wangl...@geekplus.com.cn < wangl...@geekplus.com.cn> wrote: > CaptureChangeMySQLSelf is from CaptureChangeMySQL and I fix a small bug > about binlog parsing. But i am sure there's no change that will affect the > backpressure mechanism. > > Actually I don't quite understand it. > It is related with the implementation of the processor? In order to use > the backpressure mechanism, the processor must do something? > In my case, what it will happen if the queue continue to grow(for example > to 100G)? Something bad will happen? > > Thanks, > Lei > > > ------------------------------ > wangl...@geekplus.com.cn > > > *From:* josef.zahn...@swisscom.com > *Date:* 2019-10-16 14:17 > *To:* users@nifi.apache.org; d...@nifi.apache.org > *Subject:* Re: NiFi backpressure not work > > Hi Lei > > > > I assume it’s the same as for the ListSFTP. As I don’t know the > CaptureChangeMySQLSelf processor I try to explain it for the ListSFTP. > There with one single execution of the processor you get as much flowfiles > as you have files on your disk. If it finds 1000 it sends out 1000 > flowfiles. If it finds 1’000’000 it sends out 1’000’000 flowfiles. The > processor can’t hold back some flows and knows nothing about your queue, so > it sends out everything in one “batch” per execution – even if it is more > than the queue size. Hope this explains the behavior a bit. > > > > Cheers > > > > > > *From: *"wangl...@geekplus.com.cn" <wangl...@geekplus.com.cn> > *Reply to: *"users@nifi.apache.org" <users@nifi.apache.org> > *Date: *Wednesday, 16 October 2019 at 08:10 > *To: *users <users@nifi.apache.org>, dev <d...@nifi.apache.org> > *Subject: *NiFi backpressure not work > > > > As the picture shows. > > I have set the backpressure object threshold to 10000, but picture shows > there's more than 1,000,000. Why this happens? > > > > > > > > Thanks, Lei > ------------------------------ > > wangl...@geekplus.com.cn > >
