Hello, Take a look at the options available when right-clicking on a queue... What you described is what NiFi calls back-pressure. You can configured a queue to have an object threshold (# of flow files) or data size threshold (total size of all flow files). When one of these thresholds is reached, NiFi will no longer let the source processor run until the condition goes back under the threshold.
Let us know if you need any more info on this. Thanks, Bryan On Wed, Jun 8, 2016 at 10:40 AM, Shaine Berube < shaine.ber...@perfectsearchcorp.com> wrote: > Hello all, > > I'm kind of new to developing Nifi, though I've been doing some pretty in > depth stuff and some advanced database queries. My question is in > regarding the queues between processor, I want to limit a queue to... say > 2000, how would I go about doing that? Or better yet, how would I tell the > processor generating the queue to only put a max of 2000 files into the > queue? > > Allow me to explain with a scenario: > We are doing data migration from one database to another. > -Processor A is generating a queue consumed by Processor B > -Processor A is taking configuration and generating SQL queries in 1000 > record chunks so that Processor B can insert them into a new database. > Given the size of the source database, Processor A can potentially generate > hundreds of thousands of files. > > Is there a way for Processor A to check it's down stream queue for the > queue size? How would I get Processor A to only put 2000 files into the > queue at any given time, so that Processor A can continue running but wait > for room in the queue? > > Thank you in advance. > > -- > *Shaine Berube* >
