I managed to find the culprit.. it was the init script that I was using that was doing something weird.
I added MAX_FD=50000 to my nifi-env.sh file and everything seems to be fine now thanks ________________________________ From: Abdou B <abour...@gmail.com> Sent: Wednesday, September 18, 2019 10:42 AM To: users@nifi.apache.org <users@nifi.apache.org> Subject: Re: Too many open files Hello, It seems to me that for some distribution, you should modify those values in the Cluster management tool. For example in HDF, with Ambari, you should change the parameter : nifi_user_nofile_limit. for the change to take effect. Best regards Abdou Le mer. 18 sept. 2019 à 16:34, Jean-Sebastien Vachon <jsvac...@brizodata.com<mailto:jsvac...@brizodata.com>> a écrit : Does not seem to help... processes are still limited to 4096 fds ________________________________ From: Jean-Sebastien Vachon <jsvac...@brizodata.com<mailto:jsvac...@brizodata.com>> Sent: Wednesday, September 18, 2019 10:31 AM To: users@nifi.apache.org<mailto:users@nifi.apache.org> <users@nifi.apache.org<mailto:users@nifi.apache.org>> Subject: Re: Too many open files Oups.. just saw the following: Your distribution may require an edit to /etc/security/limits.d/90-nproc.conf by adding: * soft nproc 10000 I will try this ________________________________ From: Jean-Sebastien Vachon <jsvac...@brizodata.com<mailto:jsvac...@brizodata.com>> Sent: Wednesday, September 18, 2019 10:30 AM To: users@nifi.apache.org<mailto:users@nifi.apache.org> <users@nifi.apache.org<mailto:users@nifi.apache.org>> Subject: Too many open files Hi all, I've started to see "Too many open files" error messages in Nifi. I checked https://nifi.apache.org/quickstart.html to see the recommended values to fix this and made the required changes to /etc/security/limits.conf, exited my shell and restarted Nifi. When I check the limits of the Java processes I can still see the limits to be at 4096 I've added the following to /etc/security/limits.conf * hard nofile 50000 * soft nofile 50000 but the processes show this: cat /proc/26861/limits ... Max open files 4096 4096 files ... Any idea where this 4096 comes from? I tried grepping in the init scripts, nifi configuration and nifi-env.sh but could not find this anywhere thanks
