Thanks Joe. It will definitely have to be something I do periodically, unfortunately. Calling nifi.sh dump right now fails since it can’t create any new threads…
NiFi actually came back up for a little while shortly after this email, maybe some threads closed out. But I didn’t take the opportunity to do a dump or a restart… and now it’s all locked up again. I’ll run a dump after a couple days run time and see if I can figure it out. We have a custom processor for working with Teradata, which uses a vendor provided JDBC driver. It might be related. Thanks, Peter From: Joe Witt [mailto:joe.w...@gmail.com] Sent: Wednesday, November 15, 2017 11:49 To: users@nifi.apache.org Subject: [EXT] Re: java.lang.OutOfMemoryError: unable to create new native thread You’ll want to get thread dumps during the life of nifi to figure out the pattern of what is leaking threads. Often it will be around some tcp socket handling thread in something like sftp for example. Can be a config issue or a bug. Thanks Joe On Tue, Nov 14, 2017 at 8:32 PM Peter Wicks (pwicks) <pwi...@micron.com<mailto:pwi...@micron.com>> wrote: I’ve been getting the error: 2017-11-15 09:22:24,959 ERROR [NiFi Web Server-566674] org.apache.nifi.NiFi java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at org.eclipse.jetty.util.thread.QueuedThreadPool.startThreads(QueuedThreadPool.java:476) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$200(QueuedThreadPool.java:49) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:657) at java.lang.Thread.run(Thread.java:745) My research online says this isn’t an out of memory error, but an out of resources error. The system can’t support making new threads. This really sucks, because I can’t su to my service account, because that requires a new thread… and if I have a bash session still open I can’t kill NiFi because that requires creating a new thread… last time this happened my Unix admin had to restart my server… The first time this happened we changed the limits for our service account, here is my ulimit statement. # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 96297 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 50000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 50000 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited It’s been about two weeks, and now the issue has come up again. Is this an actual hardware limitation? If I run “ps huH p <PID>| wc -l” I can get a thread count, right now it’s about 9900 threads. Thanks, Peter
