I would also suggest the ControlRate processor in front of a sensitive step. There's a chance data may accumulate and sent in burst after downtime, so it ensures a predictable outflow.
Andrew On Thu, Apr 27, 2017, 8:03 PM Joe Percivall <joeperciv...@gmail.com> wrote: > Hello Kevin, > > I believe there are two things at play here. The first is the processor > being very fast and processing the FlowFiles before back pressure gets > applied. The second is that in the current distribution, UpdateAttribute > uses an old style of getting higher performance and grabs batches of 100 > with each onTrigger[1]. Since back-pressure gets applied per onTrigger the > UpdateAttribute will process at least 100 FlowFiles before it gets told to > stop processing. > > In the changes for 1.2.0 though I updated it to bring only 1 FlowFile in > per onTrigger. So if you test this on a build of master then you should see > more appropriate back-pressure application. > > [1] > https://github.com/apache/nifi/blob/rel/nifi-1.1.2/nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java#L338 > > Joe > > On Thu, Apr 27, 2017 at 7:21 PM, Kevin Verhoeven < > kevin.verhoe...@ds-iq.com> wrote: > >> Thank you for your help Andy. I think you are correct, the flowfiles are >> very small and the previous Processor is very fast – this might explain >> what is happening. I’ve enclosed screenshots of the connection properties >> and the workflow. In the screenshot I see 400 flowfiles were allowed >> through before back pressure was applied. The back pressure object >> threshold is set to 1. Do you have any recommendations? >> >> >> >> Kevin >> >> >> >> >> >> >> >> *From:* Andy LoPresto [mailto:alopre...@apache.org] >> *Sent:* Thursday, April 27, 2017 4:16 PM >> *To:* users@nifi.apache.org >> *Subject:* Re: Back Pressure Object threshold not honored >> >> >> >> Hi Kevin, >> >> >> >> Sorry to hear you are having this issue. Can you please provide a >> screenshot of the connection properties in the configuration dialog? How >> quickly do those flowfiles get enqueued? I think there’s a chance if they >> are very small & the previous processor is very fast (i.e. >> RouteOnAttribute, SplitText) that it could enqueue a higher number before >> the back pressure check is executed. >> >> >> >> Andy LoPresto >> >> alopre...@apache.org >> >> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* >> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> >> >> On Apr 27, 2017, at 4:07 PM, Kevin Verhoeven <kevin.verhoe...@ds-iq.com> >> wrote: >> >> >> >> I have an odd problem. I set the Back Pressure Object threshold on a link >> between two Processors to 1, but 200 flowfiles are passed to the queue >> before back pressure is honored. I need the back pressure to be set to a >> small number of flowfiles to keep the source from flooding the destination. >> Has anyone come across this problem before? I am running 12 instances of >> NiFi on version 1.1.1 on Ubuntu 14.04. >> >> >> >> Regards, >> >> >> >> Kevin >> >> >> > > > > -- > *Joe Percivall* > e: joeperciv...@gmail.com >
