We've had this occur when executing complex queries and/or queries on large tables in ExecuteSQL.
We typically try out some values of Max Rows Per Flow File and Fetch Size (both to the same value) in the range of 1000, 10k, 50k, 100k to make it work without memory issues. Changing the Output Batch Size value may also be needed, but I looked up one example from recent memory and it still had the default of 0. I'd try a value of 1 to test, but the other settings are likely the main cause. Regards, Isha Van: Etienne Jouvin <lapinoujou...@gmail.com> Verzonden: donderdag 4 januari 2024 14:48 Aan: users@nifi.apache.org Onderwerp: Re: Hardware requirement for NIFI instance You don't often get email from lapinoujou...@gmail.com<mailto:lapinoujou...@gmail.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hello. I also think the problem is more about the processor, I guess ExecuteSQL. Should play with batch configuration and commit flag to commit intermediate FlowFile. The out of memory exception makes me believe the full table is retrieved, and if it is huge the FlowFile content is very large. Le jeu. 4 janv. 2024 à 14:37, Pierre Villard <pierre.villard...@gmail.com<mailto:pierre.villard...@gmail.com>> a écrit : It should be memory efficient so I think this is likely a configuration aspect of your processor. Can you share the configuration for all properties? As a side note: if NiFi ran out of memory, you'd always want to restart it because you are never sure what's the state of the JVM after an OOME. Le jeu. 4 janv. 2024 à 17:26, <e-soci...@gmx.fr<mailto:e-soci...@gmx.fr>> a écrit : Hello all, Who could help me to determine the cpu/memory need for nifi instance to fetch the data from Postgresql hosted in google ? We got this error : ==> Error : executesql.error.message Ran out of memory retrieving query results. The procesor ExecuteSQL has this config : Set Auto Commit ==> false driver Jar to use : postgresql-42.7.1.jar Java version : jdk-11.0.19 Table information : rows number : 14958836 fields number : 20 Linux Rocky8 Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Thread(s) per core: 2 Core(s) per socket: 1 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel BIOS Vendor ID: Google CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) CPU @ 2.80GHz Stepping: 7 CPU MHz: 2800.286 BogoMIPS: 5600.57 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 33792K NUMA node0 CPU(s): 0,1 Memory : 8GB Thanks for you helps Minh
