Peter, Does 10k happen to be your swap threshold in nifi.properties by any chance (it defaults to 20k I believe)?
I suspect the behavior you are seeing could be due to the way swapping works, but Mark or others could probably confirm. I found this thread where Mark explained how swapping works with a background thread, and I believe it still works this way: http://apache-nifi.1125220.n5.nabble.com/Nifi-amp-Spark-receiver-performance-configuration-td524.html -Bryan On Tue, Sep 20, 2016 at 10:22 AM, Peter Wicks (pwicks) <pwi...@micron.com> wrote: > I’m using JSONToSQL, followed by PutSQL. I’m using Teradata, which > supports a special JDBC mode called FastLoad, designed for a minimum of > 100,000 rows of data per batch. > > > > What I’m finding is that when PutSQL requests a new batch of FlowFiles > from the queue, which has over 1 million rows in it, with a batch size of > 1000000, it always returns a maximum of 10k. How can I get my obscenely > sized batch request to return all the FlowFile’s I’m asking for? > > > > Thanks, > > Peter >
