[
https://issues.apache.org/jira/browse/NIFI-16480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130166#comment-18130166
]
Anshu Singh commented on NIFI-16480:
------------------------------------
Opening this for discussion alongside NIFI-16479. The locking fix there stops a
copy from blocking other users, but the copy itself still takes 50-80 s of CPU
on large flows with shared parameter contexts.
Thoughts on approach:
1. Map only the selected components instead of the whole parent group.
2. Map each parameter context once per copy request.
3. Compute the set of groups referencing a context once per context, rather
than once per parameter, in getReferencedControllerServiceData.
Would committers prefer these as one change or as separate PRs? Since Download
Flow Definition and version control use the same mapping code, is there an
existing ticket I should align with? I am happy to contribute and to share
measurements.
cc: [~markap14] [~mcgilman] [~exceptionfactory]
> Copy (copyComponents) maps the whole parent process group and scans the full
> flow once per parameter, making copy take minutes on large flows
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16480
> URL: https://issues.apache.org/jira/browse/NIFI-16480
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 2.6.0, 2.7.0, 2.8.0, 2.9.0, 2.10.0, 2.11.0, 2.12.0
> Environment: 3-node NiFi cluster (NiFi 2.6.0 lab, issue also observed
> on NiFi 2.8.0), Java 21, 4 vCPU / 16 GB RAM per node, 8 GB heap, external
> ZooKeeper, nifi.cluster.node.read.timeout=20 sec. Flow: ~3,000 process
> groups, ~15,000 processors, parameter contexts with inheritance (one context
> bound to every process group, 1,645 parameters reachable).
> Labels: performance cluster locking copy-paste
> Reporter: Anshu Singh
> Priority: Major
> Labels: ParameterContext, copypaste, performance
>
> h3. Summary
> Even after the locking problem in NIFI-16479 is fixed, a single copy (Ctrl+C)
> is very expensive on large flows. In our lab (3,005 process groups, ~14.7k
> processors, 11 parameter contexts with 1,645 parameters reachable from every
> group), copying ONE process group took 50-82 s on a 4 vCPU node. With a
> lighter flow the same copy takes 2 s. The cost scales with (parameters
> reachable from the copied groups) x (size of the whole flow), not with the
> size of the selection.
> h3. Where the time goes (StandardNiFiServiceFacade.copyComponents, 2.8.0;
> same code in 2.12.0 and main)
> # {{mapper.mapProcessGroup(processGroup, ...)}} maps the entire *parent*
> group of the selection, including all descendants, and only afterwards
> filters to the selected components. Copying one small group from the root
> canvas therefore maps the whole flow.
> # For each selected group,
> {{parameterContexts.putAll(mapper.mapParameterContexts(pg, true, ...))}} is
> called separately. Each call builds a new map, so a context shared by all
> selected groups (common with inheritance) is mapped again for every selected
> group.
> # For every parameter, {{NiFiRegistryFlowMapper.mapParameter}} calls
> {{{}StandardParameterReferenceManager.getReferencedControllerServiceData(context,
> name){}}}. That method:
> ** walks the whole flow tree: {{{}rootGroup.findAllProcessGroups(group ->
> group.referencesParameterContext(context)){}}};
> ** for every referencing group, calls {{getProperties()}} on every processor
> and controller service, which resolves every property descriptor
> ({{{}AbstractComponentNode.getProperties{}}} ->
> {{{}getPropertyDescriptor{}}}).
> So one copy is roughly O(parameters x process groups + parameters x
> components x properties).
> Thread dump of the copying node (NiFi 2.8.0) shows the time spent here:
> {noformat}
> at
> org.apache.nifi.components.AbstractConfigurableComponent.getSupportedPropertyDescriptor(AbstractConfigurableComponent.java:80)
> at
> org.apache.nifi.controller.AbstractComponentNode.getProperties(AbstractComponentNode.java:659)
> at
> org.apache.nifi.parameter.StandardParameterReferenceManager.getReferencedControllerServiceData(StandardParameterReferenceManager.java:125)
> at
> org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper.mapParameter(NiFiRegistryFlowMapper.java:939)
> at
> org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper.mapParameterContexts(NiFiRegistryFlowMapper.java:876)
> at
> org.apache.nifi.web.StandardNiFiServiceFacade.copyComponents(StandardNiFiServiceFacade.java:5361)
> {noformat}
> h3. Measurements (lab, NiFi 2.6.0, write-lock hold time of copyComponents)
> ||Flow||Copy||Time||
> |~16.5k processors, 1,600 contexts with ~3 parameters each|1 top-level
> group|2.0 s|
> |same + a shared context with 100 parameters inherited by all groups|1
> top-level group|7.4 s (~49 ms per extra parameter)|
> |same|all 13 top-level groups|71.2 s|
> |3,005 groups, 11 contexts / 1,645 parameters bound to all groups|1
> group|50-82 s|
> h3. Possible improvements
> * Map only the selected components (and their descendants), not the whole
> parent group.
> * Map each parameter context once per copy request (share one map across the
> selected groups).
> * In getReferencedControllerServiceData, compute the set of groups
> referencing a context once per context instead of once per parameter, and
> only inspect properties whose descriptor identifies a controller service.
> Alternatively, cache per-context references for the duration of the mapping.
> * The same mapping is used by Download Flow Definition and version control,
> so they would benefit as well.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
