[
https://issues.apache.org/jira/browse/NIFI-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487688#comment-16487688
]
Mark Payne commented on NIFI-5225:
----------------------------------
[~FrederikP] I think you're right - there is certainly room to improve how we
do the whole generateReport() stuff. I was noticing some of this stuff the
other day as well when working on NIFI-5112/NIFI-950. There are quite a few
inefficiencies. EventSumValue, for instance, is holding 15
AtomicLong/AtomicIntegers and a ConcurrentHashMap that get updated with each
session commit and then have to be read each time we call generateReport() -
for each component. That would probably be significantly more efficient to just
use non-thread-safe member variables and synchronize the methods. Atomics are
fast when you only need to update 1-2 of them but when you get to that many,
it'll slow you down a good bit. But I think this is just due to the way that
the system as evolved. We could also avoid calculating these for components we
don't care about, as you mentioned, as well.
With the results that I found during my profiling, this wasn't much of an
issue. However, I also was mostly focusing on small flows, not flows with
1,000+ processors. That said, I think we should probably finish up the
outstanding issues around NIFI-950, etc. and see where that leaves us. We can
then certainly iterate to improve further if necessary.
> Leak in RingBufferEventRepository for frequently updated flows
> --------------------------------------------------------------
>
> Key: NIFI-5225
> URL: https://issues.apache.org/jira/browse/NIFI-5225
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Environment: HDF-3.1.0.0
> Reporter: Frederik Petersen
> Priority: Major
> Labels: performance
>
> We use NiFi's API to change a part of our flow quite frequently. Over the
> past weeks we have noticed that the performance of web requests degrades over
> time and had a very hard time to find out why.
> Today I took a closer look. When using visualvm to sample cpu it already
> stood out that the longer the cluster was running, the more time was spent in
> 'SecondPrecisionEventContainer.generateReport()' during web requests. This
> method is already relied on a lot right after starting the cluster (for big
> flows and process groups). But the time spent in it increases (in our setup)
> the longer the cluster runs. This increases latency of almost every web
> request. Our flow reconfiguration script (calling many NiFi API endpoints)
> went from 2 minutes to 20 minutes run time in a few days.
> Looking at the source code I couldn't quite figure out why the run time
> should increase over time, because the ring buffers always stay the same size
> (301 entries|5 minutes).
> When sampling memory I noticed quite a lot of EventSum instances, more than
> there should have been. So I took a heap dump and ran a MemoryAnalyzer tool.
> The "Leak Suspects" overview gave me the final hint to what was wrong.
> It reported:
> One instance of "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by
> "<system class loader>" occupies 5,649,926,328 (55.74%) bytes. The instance
> is referenced by
> org.apache.nifi.controller.repository.metrics.RingBufferEventRepository @
> 0x7f86c50cda40 , loaded by "org.apache.nifi.nar.NarClassLoader @
> 0x7f86a0000000". The memory is accumulated in one instance of
> "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class
> loader>".
> The issue is:
> When we remove processors, connections, process groups from the flow, their
> data is not removed from the ConcurrentHashMap in RingBufferEventRepository.
> There is a 'purgeTransferEvents' but it only calls an empty 'purgeEvents'
> method on all 'SecondPrecisionEventContainer's in the map.
> This means that the map grows without bounds and every time
> 'reportTransferEvents' is called it iterates over all (meaning more and more
> over time) entries of the map. This increases latency of every web request
> and also a huge amount of memory occupied.
> A rough idea to fix this:
> Remove the entry for each removed component (processor, process group,
> connection, ?...) using their onRemoved Methods in the FlowController
> This should stop the map from growing infinitely for any flow where removals
> of any components happens frequently. Especially when automated.
> Since this is quite urgent for us, I'll try to work on a fix for this and
> provide a pull request if successful.
> Since no-one noticed this before, I guess we are not the typical user of
> NiFi, as we thought it was possible to heavily reconfigure flows using the
> API, but with this performance issue, it's not.
> Please let me know if I can provide any more helpful detail for this problem.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
