Thanks very much. I got it.
wangl...@geekplus.com.cn From: Joe Witt Date: 2019-10-16 17:24 To: users@nifi.apache.org Subject: Re: Re: NiFi backpressure not work Lei Please do not post to both users and dev list. Dropping dev. Yes these scenarios are based on limitations of a given processor implementation or the nature of a given protocol/mechanism. NiFi enforces back pressure by slowing/stopping scheduling a component and making fact of back pressure available to components. I dont know about this particular processor but it likely needs to be updated so the max amount of data it will return in any one cycle is small enough or controllable enough to avoud what youre seeing. Thanks Joe On Wed, Oct 16, 2019 at 3:06 AM wangl...@geekplus.com.cn <wangl...@geekplus.com.cn> wrote: CaptureChangeMySQLSelf is from CaptureChangeMySQL and I fix a small bug about binlog parsing. But i am sure there's no change that will affect the backpressure mechanism. Actually I don't quite understand it. It is related with the implementation of the processor? In order to use the backpressure mechanism, the processor must do something? In my case, what it will happen if the queue continue to grow(for example to 100G)? Something bad will happen? Thanks, Lei wangl...@geekplus.com.cn From: josef.zahn...@swisscom.com Date: 2019-10-16 14:17 To: users@nifi.apache.org; d...@nifi.apache.org Subject: Re: NiFi backpressure not work Hi Lei I assume it’s the same as for the ListSFTP. As I don’t know the CaptureChangeMySQLSelf processor I try to explain it for the ListSFTP. There with one single execution of the processor you get as much flowfiles as you have files on your disk. If it finds 1000 it sends out 1000 flowfiles. If it finds 1’000’000 it sends out 1’000’000 flowfiles. The processor can’t hold back some flows and knows nothing about your queue, so it sends out everything in one “batch” per execution – even if it is more than the queue size. Hope this explains the behavior a bit. Cheers From: "wangl...@geekplus.com.cn" <wangl...@geekplus.com.cn> Reply to: "users@nifi.apache.org" <users@nifi.apache.org> Date: Wednesday, 16 October 2019 at 08:10 To: users <users@nifi.apache.org>, dev <d...@nifi.apache.org> Subject: NiFi backpressure not work As the picture shows. I have set the backpressure object threshold to 10000, but picture shows there's more than 1,000,000. Why this happens? Thanks, Lei wangl...@geekplus.com.cn
