[
https://issues.apache.org/jira/browse/NIFI-16474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130169#comment-18130169
]
Marc Pellmann commented on NIFI-16474:
--------------------------------------
To make this even clearer – the problem arises when you have versioned many
process groups within other versioned groups, and these are then frequently
used within them.
But that is precisely the approach advocated by NiFi if you wish to reuse
process groups.
One example of this is process groups that perform business-specific logging to
an external system and are used in many places.
And if another layer is added – that is, if another reused group contains many
of these logging process groups – then the number of REST requests to Bitbucket
skyrockets.
> Nested versioned flows are fetched from the Flow Registry an exponential
> number of times when loading a flow's contents
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16474
> URL: https://issues.apache.org/jira/browse/NIFI-16474
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 2.12.0
> Reporter: Marc Pellmann
> Priority: Major
>
> When NiFi loads a versioned flow together with its nested versioned flows,
> each nested flow is fetched from the Flow Registry Client many times instead
> of once. This happens through
> StandardFlowRegistryClientNode.getFlowContents(..., fetchRemoteFlows = true).
> The number of fetches grows exponentially with nesting depth.
>
> This affects every operation that resolves the full snapshot, for example:
> * GET /process-groups/\{id}/local-modifications
> (StandardNiFiServiceFacade.getLocalModifications)
> * putting an existing process group under version control (PUT
> /process-groups/\{id} with versionControlInformation)
> * importing a versioned flow into the canvas
>
> The Git-based registry clients (GitHub, GitLab, Bitbucket, Azure DevOps) talk
> to a hosted REST API. With them, a single request takes minutes and issues
> hundreds of API calls.
>
> *Cause 1: duplicated call in getFlowContents*
> {code:java}
> for (final VersionedProcessGroup child : contents.getProcessGroups()) {
> final Map<String, VersionedParameterContext> childParameterContexts =
> populateVersionedContentsRecursively(context, child, snapshotContainer);
>
> for (final Map.Entry<String, VersionedParameterContext>
> childParameterContext : childParameterContexts.entrySet()) {
>
> flowSnapshot.getParameterContexts().putIfAbsent(childParameterContext.getKey(),
> childParameterContext.getValue());
> }
> populateVersionedContentsRecursively(context, child, snapshotContainer);
> // <-- second, identical call
> } {code}
> populateVersionedContentsRecursively is called twice for every child. The
> result of the second call is discarded. The line was introduced with
> NIFI-11464.
>
> *Cause 2: nested flows are resolved twice per level*
> For a group that has VersionedFlowCoordinates,
> populateVersionedContentsRecursively calls fetchFlowContents(context,
> coordinates, true). That goes through clientNode.getFlowContents(...,
> fetchRemoteFlows = true), which already resolves every nested flow below that
> group. The group's contents are then copied over, and the method recurses
> into group.getProcessGroups() again. Those children still carry their
> coordinates, so every nested flow below is fetched again.
> Combining both causes, reading the code suggests that a flow at nesting depth
> d is fetched on the order of 2·3^(d−1) times. If the same flow occurs several
> times in the tree, that number is multiplied again. Nested contents are also
> added to the FlowSnapshotContainer repeatedly.
>
> *Cause 3: every fetch also lists the file's commit history (Git-based
> clients)*
> AbstractGitFlowRegistryClient.getFlowContents calls getLatestVersion on every
> fetch, only to set RegisteredFlowSnapshot.isLatest. That lists the commit
> history of the snapshot file (GitRepositoryClient.getCommits), which may span
> several pages. Each fetch therefore costs at least two REST calls: content
> plus history.
>
> *Observed*
> A flow with three levels of nesting, where one nested flow is used three
> times, on Bitbucket Cloud. A single GET
> /process-groups/\{id}/local-modifications ran for more than 4.5 minutes.
> During that time it fetched individual nested flows dozens of times (e.g.
> Nested.json 44 times, still running). One request thread produced more than
> 1,200 Bitbucket client log entries. Earlier, putting the same group under
> version control fetched the deepest nested flows 108 times each and took ~35
> s.
>
> *Impact*
> * getLocalModifications holds the flow read lock
> (NiFiServiceFacadeLock.proceedWithReadLock) for the whole time. Every write
> request waits behind it, e.g. deleting the very process group.
> * Synchronous REST calls such as starting version control and importing a
> flow can outlast client read timeouts. The operation still completes on the
> server, and a client retry then fails with 409 ("process group is already
> under version control") or creates a duplicate group.
> * The hosted Git APIs see hundreds of calls per user action, which risks
> rate limiting.
>
> *Steps to reproduce*
> # Configure a Git-based Flow Registry Client, e.g. the Bitbucket or GitHub
> client.
> # Version flows B and C. Version flow A so that it contains B, and B
> contains C. Optionally reference B more than once inside A.
> # Import A into the canvas, or call GET
> /nifi-api/process-groups/\{id}/local-modifications on it.
> # Enable DEBUG logging for the registry client. Each nested snapshot file
> is requested many times, and the count grows with depth.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
