Well, I learned something I thought I'd share: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices states that you should increase the number of file handles. The apache guide says to do this, you change limits.conf
Nonetheless, my nifi would run out of file handles occasionally, no matter what I did. I prefer my nifi to startup on bootup, so I had a system service file so it would autostart and run as my user. As of now, Here is the NEW contents of the file [Unit] Description=Apache NiFi After=network.target [Service] Type=forking User=<myuser> Group=<my group> ExecStart=/opt/nifi/bin/nifi.sh start ExecStop=/opt/nifi/bin/nifi.sh stop # This took me YEARS to figure out. Check out # https://unix.stackexchange.com/questions/345595/how-to-set-ulimits-on-service-with-systemd LimitNOFILE=500000 [Install] WantedBy=multi-user.target See that LimitNOFILE line? Thanks to that stack exchange article, I was finally able to see why my filehandle setting weren't working ... limits.conf doesn't apply when you have a systemd service Guess you learn something new every day. One question remains: Is 500K a good number? The OS is configured so that fs.file-max is set to 6815744 file descriptors. The only other thing this guy runs is backups (and virus scans). Otherwise its dedicated to run flows. I thought 500K would be acceptable for my use case. It might be overkill; a random lsof | wc -l when nifi was up and running but not processing yielded 300K file descriptors in use. Anybody have an opinion? Geoff Greene Associate Technical Fellow / Senior Software Ninjaneer
