Determine which Nifi processor eating CPU usage
Hello, We have a nifi flow with many custom processors (and many processor groups). We suspect that one or more processors are eating up CPU usage, so we're wondering if there's an easy way to tell which processor has a heavy load on the CPU. There are tables to see processors in order of number of flow files or bytes in/out, etc, but not based on CPU usage. In fact, I can't find a way to see a table of all processors that have active threads. All that we know is that the top command has nifi running at 800%, and we're doing trial and error by turning off processors until we hit the one that makes CPU utilization go down. We did see an earlier post about processors that poll can be eating up CPU cycles, but that doesn't seem to be the case here. Once in the past we had a custom processor with a bug that caused it to eat CPU cycles, but we discovered the issue not through Nifi but because we happened to be examining the code. Thank you! -Elli
Re: Determine which Nifi processor eating CPU usage
Elli, Unfortunately, Java doesn't provide fine-grained details about which threads are using how much CPU, etc. However, the way that we typically tackle this is to go to the Summary view in NiFi (the table that you are talking about here). There are a couple of points of interest here. If you sort the table by Run Status, then you will notice that the Run Status is "Running" for processors that are scheduled to run but have no threads. For Processors that currently have active threads, you will see "Running (1)" for 1 active thread, "Running (3)" for 3 active threads, etc. So sorting by that column will help you determine which Processors are currently running. Likely more of interest, though, is the "Tasks / Time" column. If you click on that column so that it is sorted by which Processors have used the most time (generally that will require 3 clicks to sort by that), then you can understand which Processors are running the most. It is not necessarily using the most CPU, but that's a pretty good indicator. Thanks -Mark > On Oct 26, 2015, at 10:57 AM, Elli Schwarz wrote: > > Hello, > > We have a nifi flow with many custom processors (and many processor groups). > We suspect that one or more processors are eating up CPU usage, so we're > wondering if there's an easy way to tell which processor has a heavy load on > the CPU. There are tables to see processors in order of number of flow files > or bytes in/out, etc, but not based on CPU usage. In fact, I can't find a way > to see a table of all processors that have active threads. All that we know > is that the top command has nifi running at 800%, and we're doing trial and > error by turning off processors until we hit the one that makes CPU > utilization go down. > > We did see an earlier post about processors that poll can be eating up CPU > cycles, but that doesn't seem to be the case here. Once in the past we had a > custom processor with a bug that caused it to eat CPU cycles, but we > discovered the issue not through Nifi but because we happened to be examining > the code. > > Thank you! > > -Elli >
Re: Determine which Nifi processor eating CPU usage
Unfortunately I can’t seem to even see how it would be possible for NiFi to
tell you that since your custom Processors are running within the same JVM as
NiFi.
Having said that the 800% tells me that you probably have some processor with
custom thread pool where each thread is spinning in the loop with a lot of
misses on the functionality it expects to perform.
For example:
while (true) {
if (someCondition){
// do something
}
}
The above will definitely eat 100% of your CPU if ’someCondition’ never happens
and if you have something like this running in multiple threads on 8 cores
there is your 800%.
That could be your code or some library you are using.
There is also a slight chance that your the code executed by multiple threads
is actually doing something very CPU intensive
Hope that helps
Oleg
On Oct 26, 2015, at 10:57 AM, Elli Schwarz
mailto:eliezer_schw...@yahoo.com>> wrote:
Hello,
We have a nifi flow with many custom processors (and many processor groups). We
suspect that one or more processors are eating up CPU usage, so we're wondering
if there's an easy way to tell which processor has a heavy load on the CPU.
There are tables to see processors in order of number of flow files or bytes
in/out, etc, but not based on CPU usage. In fact, I can't find a way to see a
table of all processors that have active threads. All that we know is that the
top command has nifi running at 800%, and we're doing trial and error by
turning off processors until we hit the one that makes CPU utilization go down.
We did see an earlier post about processors that poll can be eating up CPU
cycles, but that doesn't seem to be the case here. Once in the past we had a
custom processor with a bug that caused it to eat CPU cycles, but we discovered
the issue not through Nifi but because we happened to be examining the code.
Thank you!
-Elli
Re: Determine which Nifi processor eating CPU usage
Elli,
In the majority of cases what Mark suggested will help you pinpoint
the offending processors.
If it is not clear which are the offending processes/extension then it
can require some fairly involved digging. For such things I have
found the following articles provide good examples of how to
potentially work through it:
https://blogs.oracle.com/jiechen/entry/analysis_against_jvm_thread_dump
https://blogs.manageengine.com/application-performance-2/appmanager/2011/02/09/identify-java-code-consuming-high-cpu-in-linux-linking-jvm-thread-and-linux-pid.html
And for those really special times you can also, assuming linux, run
'perf top'. This command is *amazing* albeit reveals very low level
details which aren't always easy to correlate to higher level user
code.
Thanks
Joe
On Mon, Oct 26, 2015 at 11:14 AM, Oleg Zhurakousky
wrote:
> Unfortunately I can’t seem to even see how it would be possible for NiFi to
> tell you that since your custom Processors are running within the same JVM
> as NiFi.
> Having said that the 800% tells me that you probably have some processor
> with custom thread pool where each thread is spinning in the loop with a lot
> of misses on the functionality it expects to perform.
> For example:
> while (true) {
> if (someCondition){
> // do something
> }
> }
> The above will definitely eat 100% of your CPU if ’someCondition’ never
> happens and if you have something like this running in multiple threads on 8
> cores there is your 800%.
> That could be your code or some library you are using.
>
> There is also a slight chance that your the code executed by multiple
> threads is actually doing something very CPU intensive
>
> Hope that helps
>
> Oleg
>
>
> On Oct 26, 2015, at 10:57 AM, Elli Schwarz
> wrote:
>
> Hello,
>
> We have a nifi flow with many custom processors (and many processor groups).
> We suspect that one or more processors are eating up CPU usage, so we're
> wondering if there's an easy way to tell which processor has a heavy load on
> the CPU. There are tables to see processors in order of number of flow files
> or bytes in/out, etc, but not based on CPU usage. In fact, I can't find a
> way to see a table of all processors that have active threads. All that we
> know is that the top command has nifi running at 800%, and we're doing trial
> and error by turning off processors until we hit the one that makes CPU
> utilization go down.
>
> We did see an earlier post about processors that poll can be eating up CPU
> cycles, but that doesn't seem to be the case here. Once in the past we had a
> custom processor with a bug that caused it to eat CPU cycles, but we
> discovered the issue not through Nifi but because we happened to be
> examining the code.
>
> Thank you!
>
> -Elli
>
>
Re: Determine which Nifi processor eating CPU usage
This brings up NiFi best practices question: whether a developer can use spawn
threads in processors or controller services?
If there are any guidelines , it would be nice to document in developer guide.
Sumanth
Sent from my iPad
> On Oct 26, 2015, at 8:14 AM, Oleg Zhurakousky
> wrote:
>
> Unfortunately I can’t seem to even see how it would be possible for NiFi to
> tell you that since your custom Processors are running within the same JVM as
> NiFi.
> Having said that the 800% tells me that you probably have some processor with
> custom thread pool where each thread is spinning in the loop with a lot of
> misses on the functionality it expects to perform.
> For example:
> while (true) {
> if (someCondition){
> // do something
> }
> }
> The above will definitely eat 100% of your CPU if ’someCondition’ never
> happens and if you have something like this running in multiple threads on 8
> cores there is your 800%.
> That could be your code or some library you are using.
>
> There is also a slight chance that your the code executed by multiple threads
> is actually doing something very CPU intensive
>
> Hope that helps
>
> Oleg
>
>
>> On Oct 26, 2015, at 10:57 AM, Elli Schwarz wrote:
>>
>> Hello,
>>
>> We have a nifi flow with many custom processors (and many processor groups).
>> We suspect that one or more processors are eating up CPU usage, so we're
>> wondering if there's an easy way to tell which processor has a heavy load on
>> the CPU. There are tables to see processors in order of number of flow files
>> or bytes in/out, etc, but not based on CPU usage. In fact, I can't find a
>> way to see a table of all processors that have active threads. All that we
>> know is that the top command has nifi running at 800%, and we're doing trial
>> and error by turning off processors until we hit the one that makes CPU
>> utilization go down.
>>
>> We did see an earlier post about processors that poll can be eating up CPU
>> cycles, but that doesn't seem to be the case here. Once in the past we had a
>> custom processor with a bug that caused it to eat CPU cycles, but we
>> discovered the issue not through Nifi but because we happened to be
>> examining the code.
>>
>> Thank you!
>>
>> -Elli
>
