Hi Matt, I have set the fetch size to 20000. I tried with small values like 100. Got the same issue with the small sizes. Further I limited the row count fetched by the sql by adding a limit. Then I didn't got the issue even with large fetch sizes.
Thank you. On Mon, 10 May 2021, 8:05 pm Matt Burgess, <[email protected]> wrote: > Vibath, > > What is the "Fetch Size" property set to? It looks like PostgreSQL > will load all results if Fetch Size is set to zero [1]. Try setting it > to 10000 or something like that, whatever doesn't use too much memory > but doesn't slow down the performance too much. > > Regards, > Matt > > [1] https://jdbc.postgresql.org/documentation/head/query.html > > On Mon, May 10, 2021 at 3:41 AM Vibhath Ileperuma > <[email protected]> wrote: > > > > Hi Vijay, > > > > Eventhough that property is set to a postive value, same issue happens. > Seems like NIFI first loads all the rows to RAM and write to multiple files > when that property is set to a non-zero value. > > On the other hand, if this property is set to a very small value, a > large no of flow files can be generated at once. > > > > Thank you. > > Best regards, > > Vibhath > > > > On Mon, 10 May 2021, 9:00 am Vijay Chhipa, <[email protected]> wrote: > >> > >> Hi Vibhath, > >> > >> There is this property on the processor > >> > >> Max Rows Per Flow File > >> > >> > >> Per docs: If the value specified is zero, then all rows are returned > in a single FlowFile. > >> Which seems to be what is happening in your case. > >> > >> > >> > >> On May 9, 2021, at 12:11 PM, Vibhath Ileperuma < > [email protected]> wrote: > >> > >> executeSQLRecord > >> > >> >
