Anshu Singh created NIFI-16480:
----------------------------------
Summary: Copy (copyComponents) maps the whole parent process group
and scans the full flow once per parameter, making copy take minutes on large
flows
Key: NIFI-16480
URL: https://issues.apache.org/jira/browse/NIFI-16480
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Affects Versions: 2.12.0, 2.11.0, 2.10.0, 2.9.0, 2.8.0, 2.7.0, 2.6.0
Environment: 3-node NiFi cluster (NiFi 2.6.0 lab, issue also observed
on NiFi 2.8.0), Java 21, 4 vCPU / 16 GB RAM per node, 8 GB heap, external
ZooKeeper, nifi.cluster.node.read.timeout=20 sec. Flow: ~3,000 process groups,
~15,000 processors, parameter contexts with inheritance (one context bound to
every process group, 1,645 parameters reachable).
Labels: performance cluster locking copy-paste
Reporter: Anshu Singh
h3. Summary
Even after the locking problem in NIFI-16479 is fixed, a single copy (Ctrl+C)
is very expensive on large flows. In our lab (3,005 process groups, ~14.7k
processors, 11 parameter contexts with 1,645 parameters reachable from every
group), copying ONE process group took 50-82 s on a 4 vCPU node. With a lighter
flow the same copy takes 2 s. The cost scales with (parameters reachable from
the copied groups) x (size of the whole flow), not with the size of the
selection.
h3. Where the time goes (StandardNiFiServiceFacade.copyComponents, 2.8.0; same
code in 2.12.0 and main)
# {{mapper.mapProcessGroup(processGroup, ...)}} maps the entire *parent* group
of the selection, including all descendants, and only afterwards filters to the
selected components. Copying one small group from the root canvas therefore
maps the whole flow.
# For each selected group,
{{parameterContexts.putAll(mapper.mapParameterContexts(pg, true, ...))}} is
called separately. Each call builds a new map, so a context shared by all
selected groups (common with inheritance) is mapped again for every selected
group.
# For every parameter, {{NiFiRegistryFlowMapper.mapParameter}} calls
{{{}StandardParameterReferenceManager.getReferencedControllerServiceData(context,
name){}}}. That method:
** walks the whole flow tree: {{{}rootGroup.findAllProcessGroups(group ->
group.referencesParameterContext(context)){}}};
** for every referencing group, calls {{getProperties()}} on every processor
and controller service, which resolves every property descriptor
({{{}AbstractComponentNode.getProperties{}}} -> {{{}getPropertyDescriptor{}}}).
So one copy is roughly O(parameters x process groups + parameters x components
x properties).
Thread dump of the copying node (NiFi 2.8.0) shows the time spent here:
{noformat}
at
org.apache.nifi.components.AbstractConfigurableComponent.getSupportedPropertyDescriptor(AbstractConfigurableComponent.java:80)
at
org.apache.nifi.controller.AbstractComponentNode.getProperties(AbstractComponentNode.java:659)
at
org.apache.nifi.parameter.StandardParameterReferenceManager.getReferencedControllerServiceData(StandardParameterReferenceManager.java:125)
at
org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper.mapParameter(NiFiRegistryFlowMapper.java:939)
at
org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper.mapParameterContexts(NiFiRegistryFlowMapper.java:876)
at
org.apache.nifi.web.StandardNiFiServiceFacade.copyComponents(StandardNiFiServiceFacade.java:5361)
{noformat}
h3. Measurements (lab, NiFi 2.6.0, write-lock hold time of copyComponents)
||Flow||Copy||Time||
|~16.5k processors, 1,600 contexts with ~3 parameters each|1 top-level
group|2.0 s|
|same + a shared context with 100 parameters inherited by all groups|1
top-level group|7.4 s (~49 ms per extra parameter)|
|same|all 13 top-level groups|71.2 s|
|3,005 groups, 11 contexts / 1,645 parameters bound to all groups|1 group|50-82
s|
h3. Possible improvements
* Map only the selected components (and their descendants), not the whole
parent group.
* Map each parameter context once per copy request (share one map across the
selected groups).
* In getReferencedControllerServiceData, compute the set of groups referencing
a context once per context instead of once per parameter, and only inspect
properties whose descriptor identifies a controller service. Alternatively,
cache per-context references for the duration of the mapping.
* The same mapping is used by Download Flow Definition and version control, so
they would benefit as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
