Evan, Thanks for sharing that diagnosing technique. While ideally we would have other controls to prevent excess CPU usage, this seems like a useful tool which could be automated using NiPyAPI [1] to perform a “bisect” command. I’ve used this for git commit searching as well as side-effect unit test identification.
[1] https://nipyapi.readthedocs.io/en/latest/readme.html <https://nipyapi.readthedocs.io/en/latest/readme.html> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 15, 2019, at 1:40 PM, Evan Reynolds <e...@usermind.com> wrote: > > I have found two issues that can cause high CPU when idle (high being about > 200% CPU when idle.) I haven’t verified these with 1.9.2, but it doesn’t hurt > to tell you. > > If you are using ports, make sure each input port is connected. If you have a > one that isn’t connected, that can bring your CPU to 200% and stay there. > > If you have any processors that are set to run on the primary node of a > cluster, that can also take your CPU to 200%. I know RouteOnAttribute will do > that (again, haven’t tested 1.9.2, but it was a problem for me for a bit!) > The fix – either don’t run it on the primary node, or else set the run > schedule to 5 seconds or something instead of 0. > > To find out if this is the case – well, this is what I did. It worked, and > wasn’t that hard, though isn’t exactly elegant. > > Back up your flowfile (flow.xml.gz) > Stop all your processors and see what your CPU does > Start half of them and watch your CPU – basically do a binary search. If your > CPU stays reasonable, then whatever group you started is fine. If not, then > start stopping things until your CPU becomes reasonable. > Eventually you’ll find a processor that spikes your CPU when you start it and > then you can figure out what to do about that processor. Record which > processor it is and how you altered it to bring CPU down. > Repeat, as there may be more than one processor spiking CPU. > Stop NiFi and restore your flowfile by copying it back in place – since you > were running around stopping things, this just makes sure everything is > correctly back to where it should be > > Then use the list of processors and fixes to make changes. > > --------------------------------------------------------------- > > Evan Reynolds > e...@usermind.com <mailto:e...@usermind.com> > > > From: Jon Logan <jmlo...@buffalo.edu> > Reply-To: "users@nifi.apache.org" <users@nifi.apache.org> > Date: Sunday, October 13, 2019 at 6:12 PM > To: "users@nifi.apache.org" <users@nifi.apache.org> > Subject: Re: High CPU consumption > > That isn't logback, that lists all jars on your classpath, the first of which > happens to be logback. > > If you send a SIGKILL3 (you can send it in HTOP) it will dump the thread > stacks to stdout (probably the bootstrap log)...but that's just for one > instant, and may or may not be helpful. > > On Sun, Oct 13, 2019 at 8:58 PM Luis Carmona <lcarm...@openpartner.cl > <mailto:lcarm...@openpartner.cl>> wrote: > hi Aldrin, > > thanks a lot, by now I'm trying to learn how to make the profiling you > mentioned. > > One more question: Is it normal that the father java process has very low > consumption while the child process related to logback jar is the one that is > eating up all the CPU ? > Please take a look at the attached image. > > Thanks, > > LC > > From: "Aldrin Piri" <aldrinp...@gmail.com <mailto:aldrinp...@gmail.com>> > To: "users" <users@nifi.apache.org <mailto:users@nifi.apache.org>> > Sent: Sunday, October 13, 2019 9:30:47 PM > Subject: Re: High CPU consumption > > Luis, please feel free to give us some information on your flow so we can > help you track down problematic areas as well. > > On Sun, Oct 13, 2019 at 3:56 PM Jon Logan <jmlo...@buffalo.edu > <mailto:jmlo...@buffalo.edu>> wrote: > You should put a profiler on it to be sure. > Just because your processors aren't processing data doesn't mean they are > idle though -- many have to poll for new data, especially sources -- ex. > connecting to Kafka, etc, will itself consume some CPU. > > But again, you should attach a profiler before participating in a wild goose > chase of performance issues. > > On Sun, Oct 13, 2019 at 12:20 PM Luis Carmona <lcarm...@openpartner.cl > <mailto:lcarm...@openpartner.cl>> wrote: > HI, > > I've struggling to reduce my nifi installation CPU consumption. Even in idle > state, all processors running but no data flowing, it is beyond 60% CPU > consumption, with peaks of 200%. > > What I've done so far > - Read and followed every instruction/post about tuning NIFI I've found > googling. > - Verify scheduling is 1s for most consuming processors: http processors, > wait/notify, jolt, etc. > - Scratch my head... > > But nothing seems to have a major effect on the issue. > > Can anyone give me some precise directions or tips about how to solve this > please ? > Is this the regular situation, I mean this is the minimun NIFI consumption. > > The server is configure with 4 CPU's, 8 GB RAM - 4 of them dedicated to heap > at bootstrap.conf. > > Thanks in advance. > > LC
