Sheng Zhang created NIFI-16570:
----------------------------------
Summary: Updating a Parameter Context with many parameters and
many bound Process Groups is slow: building the Parameter Context DTO walks the
whole Process Group tree twice per parameter
Key: NIFI-16570
URL: https://issues.apache.org/jira/browse/NIFI-16570
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Affects Versions: 2.5.0
Reporter: Sheng Zhang
Updating a Parameter Context that has a large number of parameters and is bound
to a large number of Process Groups takes tens of seconds, even when almost
nothing changes.
`DtoFactory#createParameterContextDto` builds the DTO as follows:
1. For every parameter it calls `createParameterEntity`.
2. `createParameterEntity` calls
`ParameterReferenceManager#getProcessorsReferencing` and
`#getControllerServicesReferencing` for that parameter.
3. In `StandardParameterReferenceManager`, each of those calls runs
`rootGroup.findAllProcessGroups(group ->
group.referencesParameterContext(parameterContext))`, i.e. a full walk of the
Process Group tree, and then scans the components of every matching group.
That is 2 tree walks per parameter, on top of one more walk in
`getProcessGroupsBound`. The cost is O(parameters x process groups).
The DTO is built several times during one update request:
- for the response of `PUT /parameter-contexts/\{id}`;
- when `ParameterUpdateManager` re-reads the context after replication;
- on every poll of `GET /parameter-contexts/\{id}/update-requests/\{requestId}`.
### Measurements
Observed on a 2.5.0-based build, on a Parameter Context with 438 parameters
bound to 1426 Process Groups:
- About 18 ms per tree walk, so 438 x 2 x 18 ms is roughly 16 s per DTO build.
- The end-to-end update (PUT until the update request completes) took 33.9 s.
Measured numbers came from temporary timing logs on our fork. They are not part
of any patch.
### Proposed change
1. Add
`ParameterReferenceManager#getComponentsReferencingByParameterName(ParameterContext)`,
returning `Map<String, Set<ComponentNode>>`.
`StandardParameterReferenceManager` builds it in a single pass over the bound
Process Groups. A default implementation can return null so that other
implementations fall back to the current per-parameter lookup.
2. `DtoFactory#createParameterContextDto` builds the index once per call and
uses it for every parameter, instead of one lookup per parameter.
The output is identical to today and no API contract changes.
### Result (our fork)
On the same context, `createParameterContextDto` dropped from about 16 s to
42-122 ms, and the end-to-end update from 33.9 s to 0.92 s. These numbers were
measured on a build that also skipped building the 1426 bound Process Group
entities in the update response. That step takes about 69 ms per DTO build, so
it is not what makes the difference; the index is.
## Notes
I searched JIRA and apache/nifi issues and PRs and found nothing about this.
NIFI-7536 (timeouts while updating a Parameter Context) is related only in
symptom; it addressed the stop/replication wait, not DTO building.
I can open a PR with the index change and a unit test.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
