Re: High CPU usage in FileSystemRepository.java

[Oleg Zhurakousky]

Adam, thanks for doing all the research and pointing out where the problem is. 
It is definitely a bug.

Joe I’ve raised the https://issues.apache.org/jira/browse/NIFI-1200

Cheers
Oleg

On Nov 19, 2015, at 1:18 PM, Joe Percivall 
> wrote:

Hello Adam,


Are you still seeing high cpu usage?

Sorry no has gotten back to you sooner, we are all working very hard to get 
0.4.0 out.

Joe

- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com




On Friday, November 13, 2015 4:10 PM, Adam Lamar  wrote:



Mark,

For this development system, I'm running the packaged OpenJDK from
Ubuntu 14.04:

$ java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-0ubuntu1.14.04.1)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

Interestingly, I tried another system running the Oracle JDK (same
version) and didn't see the same issue. Though it does seem to benefit
from the additional sleep in that loop, but just barely (maybe 1% CPU
difference - I could be making that up).

I hadn't uncommented those values, but I tried with no noticeable
difference on the OpenJDK system.

Hope that helps,
Adam


On 11/13/15 5:28 AM, Mark Payne wrote:
Adam,

What version of Java are you running?

Do you have the following lines from conf/bootstrap.conf uncommented, or are 
they still commented out?

java.arg.7=-XX:ReservedCodeCacheSize=256m
java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m
java.arg.9=-XX:+UseCodeCacheFlushing
java.arg.11=-XX:PermSize=128M
java.arg.12=-XX:MaxPermSize=128M

Thanks
-Mark


On Nov 13, 2015, at 12:28 AM, Joe Witt  wrote:

sorry - i see now :-)

Thanks for the analysis.  Will dig in.

Joe

On Fri, Nov 13, 2015 at 12:28 AM, Joe Witt  wrote:
Adam,

Are you on a recent master build?

Thanks
Joe

On Fri, Nov 13, 2015 at 12:27 AM, Adam Lamar  wrote:
Hi everybody!

I'm following up from my previous thread about high CPU usage in GetSQS. I
ran into high CPU usage while developing a patch for that processor, and
while investigating with "top", I noticed one NiFi thread in particular
showed high CPU usage, even after turning off all processors and restarting
NiFi.

A jstack showed this thread was busy at FileSystemRepository.java line 1287
[1]. Since that is a continue statement, it suggests that the thread was
churning in the surrounding for loop.

I didn't debug any further, but I did add a sleep statement just before the
continue, and CPU usage dropped wildly, settling around 2-4%.

I hope this is useful information, and I'm happy to debug further if needed.

Cheers,
Adam

[1]
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L1287

Re: High CPU usage in FileSystemRepository.java

[Joe Percivall]

Hello Adam,


Are you still seeing high cpu usage?

Sorry no has gotten back to you sooner, we are all working very hard to get 
0.4.0 out. 

Joe

- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com




On Friday, November 13, 2015 4:10 PM, Adam Lamar  wrote:



Mark,

For this development system, I'm running the packaged OpenJDK from 
Ubuntu 14.04:

$ java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-0ubuntu1.14.04.1)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

Interestingly, I tried another system running the Oracle JDK (same 
version) and didn't see the same issue. Though it does seem to benefit 
from the additional sleep in that loop, but just barely (maybe 1% CPU 
difference - I could be making that up).

I hadn't uncommented those values, but I tried with no noticeable 
difference on the OpenJDK system.

Hope that helps,
Adam


On 11/13/15 5:28 AM, Mark Payne wrote:
> Adam,
>
> What version of Java are you running?
>
> Do you have the following lines from conf/bootstrap.conf uncommented, or are 
> they still commented out?
>
> java.arg.7=-XX:ReservedCodeCacheSize=256m
> java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m
> java.arg.9=-XX:+UseCodeCacheFlushing
> java.arg.11=-XX:PermSize=128M
> java.arg.12=-XX:MaxPermSize=128M
>
> Thanks
> -Mark
>
>
>> On Nov 13, 2015, at 12:28 AM, Joe Witt  wrote:
>>
>> sorry - i see now :-)
>>
>> Thanks for the analysis.  Will dig in.
>>
>> Joe
>>
>> On Fri, Nov 13, 2015 at 12:28 AM, Joe Witt  wrote:
>>> Adam,
>>>
>>> Are you on a recent master build?
>>>
>>> Thanks
>>> Joe
>>>
>>> On Fri, Nov 13, 2015 at 12:27 AM, Adam Lamar  wrote:
 Hi everybody!

 I'm following up from my previous thread about high CPU usage in GetSQS. I
 ran into high CPU usage while developing a patch for that processor, and
 while investigating with "top", I noticed one NiFi thread in particular
 showed high CPU usage, even after turning off all processors and restarting
 NiFi.

 A jstack showed this thread was busy at FileSystemRepository.java line 1287
 [1]. Since that is a continue statement, it suggests that the thread was
 churning in the surrounding for loop.

 I didn't debug any further, but I did add a sleep statement just before the
 continue, and CPU usage dropped wildly, settling around 2-4%.

 I hope this is useful information, and I'm happy to debug further if 
 needed.

 Cheers,
 Adam

 [1]
 https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/FileSystemRepository.java#L1287

Re: queued files

[Bryan Bende]

Charlie,

The behavior you described usually means that the processor encountered an
unexpected error which was thrown back to the framework which rolls back
the processing of that flow file and leaves it in the queue, as opposed to
an error it expected where it would usually route to a failure relationship.

Is the id that you see in the bulletin a uuid?

There should still be some provenance events for this FlowFile from the
previous points in the flow. If it looks like the uuid of the FlowFile,
that should be searchable from provenance using the search button on the
right. Let us know if we can help more.

-Bryan

On Thu, Nov 19, 2015 at 9:10 PM, Charlie Frasure 
wrote:

> I have a question on troubleshooting a flow.  I've built a flow with no
> exception routing, just trying to process the expected values first.  When
> a file exposes a problem with the logic in my flow, it queues up prior to
> the flow that is raising the bulletin.
>
> In the bulletin, I can see an id, but can't tell which file it is.  Data
> provenance doesn't seem to help as it passed the flow on the last
> processor, but hasn't been logged (to my knowledge) on the next one.
>
> Is there a way to match the bulletin back to a file without creating a
> route for failed files?
>

Re: queued files

[Joe Witt]

Charlie,

The fact that this is confusing is something we agree should be more
clear and we will improve.  We're tackling it based on what is
mentioned here [1].

[1] 
https://cwiki.apache.org/confluence/display/NIFI/Interactive+Queue+Management

Thanks
Joe

On Thu, Nov 19, 2015 at 10:30 PM, Corey Flowers  wrote:
> These guys are right. The file to look in for the uuid is the nifi-app.log.
> Also if you wanted to see what the processor itself was doing, you could
> right click on the processor, get its uuid and while it is running, run
> (assuming it is on Linux):
>
> tail -F nifi-app.log | grep uuid
>
> This will just scroll the logs for that specific processor and will show you
> what it is doing. It should also tell you specific file names and uuids of
> the failing files.
>
> Hope that helps! Have a great night and good luck!
>
> Sent from my iPhone
>
> On Nov 19, 2015, at 9:27 PM, Juan Sequeiros  wrote:
>
> You can also check the NiFi logs for a searchable id or for what the
> previous processor ID produced to help search provenance.
>
> On Nov 19, 2015 21:22, "Bryan Bende"  wrote:
>>
>> Charlie,
>>
>> The behavior you described usually means that the processor encountered an
>> unexpected error which was thrown back to the framework which rolls back the
>> processing of that flow file and leaves it in the queue, as opposed to an
>> error it expected where it would usually route to a failure relationship.
>>
>> Is the id that you see in the bulletin a uuid?
>>
>> There should still be some provenance events for this FlowFile from the
>> previous points in the flow. If it looks like the uuid of the FlowFile, that
>> should be searchable from provenance using the search button on the right.
>> Let us know if we can help more.
>>
>> -Bryan
>>
>> On Thu, Nov 19, 2015 at 9:10 PM, Charlie Frasure
>>  wrote:
>>>
>>> I have a question on troubleshooting a flow.  I've built a flow with no
>>> exception routing, just trying to process the expected values first.  When a
>>> file exposes a problem with the logic in my flow, it queues up prior to the
>>> flow that is raising the bulletin.
>>>
>>> In the bulletin, I can see an id, but can't tell which file it is.  Data
>>> provenance doesn't seem to help as it passed the flow on the last processor,
>>> but hasn't been logged (to my knowledge) on the next one.
>>>
>>> Is there a way to match the bulletin back to a file without creating a
>>> route for failed files?
>>
>>
>

queued files

[Charlie Frasure]

I have a question on troubleshooting a flow.  I've built a flow with no
exception routing, just trying to process the expected values first.  When
a file exposes a problem with the logic in my flow, it queues up prior to
the flow that is raising the bulletin.

In the bulletin, I can see an id, but can't tell which file it is.  Data
provenance doesn't seem to help as it passed the flow on the last
processor, but hasn't been logged (to my knowledge) on the next one.

Is there a way to match the bulletin back to a file without creating a
route for failed files?

Re: queued files

[Juan Sequeiros]

You can also check the NiFi logs for a searchable id or for what the
previous processor ID produced to help search provenance.
On Nov 19, 2015 21:22, "Bryan Bende"  wrote:

> Charlie,
>
> The behavior you described usually means that the processor encountered an
> unexpected error which was thrown back to the framework which rolls back
> the processing of that flow file and leaves it in the queue, as opposed to
> an error it expected where it would usually route to a failure relationship.
>
> Is the id that you see in the bulletin a uuid?
>
> There should still be some provenance events for this FlowFile from the
> previous points in the flow. If it looks like the uuid of the FlowFile,
> that should be searchable from provenance using the search button on the
> right. Let us know if we can help more.
>
> -Bryan
>
> On Thu, Nov 19, 2015 at 9:10 PM, Charlie Frasure  > wrote:
>
>> I have a question on troubleshooting a flow.  I've built a flow with no
>> exception routing, just trying to process the expected values first.  When
>> a file exposes a problem with the logic in my flow, it queues up prior to
>> the flow that is raising the bulletin.
>>
>> In the bulletin, I can see an id, but can't tell which file it is.  Data
>> provenance doesn't seem to help as it passed the flow on the last
>> processor, but hasn't been logged (to my knowledge) on the next one.
>>
>> Is there a way to match the bulletin back to a file without creating a
>> route for failed files?
>>
>
>

Re: queued files

[Corey Flowers]

These guys are right. The file to look in for the uuid is the nifi-app.log.
Also if you wanted to see what the processor itself was doing, you could
right click on the processor, get its uuid and while it is running, run
(assuming it is on Linux):

tail -F nifi-app.log | grep uuid

This will just scroll the logs for that specific processor and will show
you what it is doing. It should also tell you specific file names and uuids
of the failing files.

Hope that helps! Have a great night and good luck!

Sent from my iPhone

On Nov 19, 2015, at 9:27 PM, Juan Sequeiros  wrote:

You can also check the NiFi logs for a searchable id or for what the
previous processor ID produced to help search provenance.
On Nov 19, 2015 21:22, "Bryan Bende"  wrote:

> Charlie,
>
> The behavior you described usually means that the processor encountered an
> unexpected error which was thrown back to the framework which rolls back
> the processing of that flow file and leaves it in the queue, as opposed to
> an error it expected where it would usually route to a failure relationship.
>
> Is the id that you see in the bulletin a uuid?
>
> There should still be some provenance events for this FlowFile from the
> previous points in the flow. If it looks like the uuid of the FlowFile,
> that should be searchable from provenance using the search button on the
> right. Let us know if we can help more.
>
> -Bryan
>
> On Thu, Nov 19, 2015 at 9:10 PM, Charlie Frasure  > wrote:
>
>> I have a question on troubleshooting a flow.  I've built a flow with no
>> exception routing, just trying to process the expected values first.  When
>> a file exposes a problem with the logic in my flow, it queues up prior to
>> the flow that is raising the bulletin.
>>
>> In the bulletin, I can see an id, but can't tell which file it is.  Data
>> provenance doesn't seem to help as it passed the flow on the last
>> processor, but hasn't been logged (to my knowledge) on the next one.
>>
>> Is there a way to match the bulletin back to a file without creating a
>> route for failed files?
>>
>
>

MergeContent Processor

[Elli Schwarz]

Hello,
I'm a bit confused about the relationship of certain properties of the 
MergeContent processor. Specifically, how do the properties min entries, max 
entries, max bin age, max number of bins interact? If the MergeContent 
processor receives the min number of entries, does it merge without waiting for 
max bin age? Or does max bin age trump the other properties? If max bin age is 
hit before the min number of entries, does the processor wait until it gets the 
min number? Does it merge once it gets to the max bin age, regardless of 
whether or not the max entries has been received? What about min/max group size 
vs. min/max number of entries?

I want to make sure that the processor isn't waiting forever (ie, will send 
after 10 minutes no matter what) if there's only 1 flowfile in the queue. If I 
set max bin age to 10 minutes, and min entries to 10, what does that mean, it 
seems to work the way I expect, which makes me wonder what does the min entries 
property mean if it doesn't seem to be used?
Thank you for any clarifications possible. I looked through the documentation 
for this processor, but it doesn't seem to explain these crucial details, which 
greatly impact my strategy for using this processor properly.
-Elli

Re: MergeContent Processor

[Aldrin Piri]

Elli,

Your understanding of the functionality is correct. There are a couple of
criteria that drive when a bin is "done." In this case, if you establish
the optional maximum properties, these drive in closing out sooner.  That
is if a max age is specified, and any of the bins have gone beyond that
time, they will be closed and transferred out.

Alternatively, a bin is also considered ready if the max age has not yet
elapsed and:

* both minimum size and minimum number of files has been reached and a few
successive attempts to add to the bin (specifically, five) have been
unsuccessful, signaling that it is nearly full or the objects are ill
suited for tighter packing.

* size or number of entries is greater than or equal to their respective,
optionally, specified maximum

Let us know if you have any other questions!

On Thu, Nov 19, 2015 at 8:09 AM, Elli Schwarz 
wrote:

> Hello,
>
> I'm a bit confused about the relationship of certain properties of the
> MergeContent processor. Specifically, how do the properties min entries,
> max entries, max bin age, max number of bins interact? If the MergeContent
> processor receives the min number of entries, does it merge without waiting
> for max bin age? Or does max bin age trump the other properties? If max bin
> age is hit before the min number of entries, does the processor wait until
> it gets the min number? Does it merge once it gets to the max bin age,
> regardless of whether or not the max entries has been received? What about
> min/max group size vs. min/max number of entries?
>
> I want to make sure that the processor isn't waiting forever (ie, will
> send after 10 minutes no matter what) if there's only 1 flowfile in the
> queue. If I set max bin age to 10 minutes, and min entries to 10, what does
> that mean, it seems to work the way I expect, which makes me wonder what
> does the min entries property mean if it doesn't seem to be used?
>
> Thank you for any clarifications possible. I looked through the
> documentation for this processor, but it doesn't seem to explain these
> crucial details, which greatly impact my strategy for using this processor
> properly.
>
> -Elli
>
>