Marc Pellmann created NIFI-16474:
------------------------------------
Summary: Nested versioned flows are fetched from the Flow Registry
an exponential number of times when loading a flow's contents
Key: NIFI-16474
URL: https://issues.apache.org/jira/browse/NIFI-16474
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 2.12.0
Reporter: Marc Pellmann
When NiFi loads a versioned flow together with its nested versioned flows, each
nested flow is fetched from the Flow Registry Client many times instead of
once. This happens through StandardFlowRegistryClientNode.getFlowContents(...,
fetchRemoteFlows = true). The number of fetches grows exponentially with
nesting depth.
This affects every operation that resolves the full snapshot, for example:
* GET /process-groups/\{id}/local-modifications
(StandardNiFiServiceFacade.getLocalModifications)
* putting an existing process group under version control (PUT
/process-groups/\{id} with versionControlInformation)
* importing a versioned flow into the canvas
The Git-based registry clients (GitHub, GitLab, Bitbucket, Azure DevOps) talk
to a hosted REST API. With them, a single request takes minutes and issues
hundreds of API calls.
*Cause 1: duplicated call in getFlowContents*
{code:java}
for (final VersionedProcessGroup child : contents.getProcessGroups()) {
final Map<String, VersionedParameterContext> childParameterContexts =
populateVersionedContentsRecursively(context, child, snapshotContainer);
for (final Map.Entry<String, VersionedParameterContext>
childParameterContext : childParameterContexts.entrySet()) {
flowSnapshot.getParameterContexts().putIfAbsent(childParameterContext.getKey(),
childParameterContext.getValue());
}
populateVersionedContentsRecursively(context, child, snapshotContainer);
// <-- second, identical call
} {code}
populateVersionedContentsRecursively is called twice for every child. The
result of the second call is discarded. The line was introduced with NIFI-11464.
*Cause 2: nested flows are resolved twice per level*
For a group that has VersionedFlowCoordinates,
populateVersionedContentsRecursively calls fetchFlowContents(context,
coordinates, true). That goes through clientNode.getFlowContents(...,
fetchRemoteFlows = true), which already resolves every nested flow below that
group. The group's contents are then copied over, and the method recurses into
group.getProcessGroups() again. Those children still carry their coordinates,
so every nested flow below is fetched again.
Combining both causes, reading the code suggests that a flow at nesting depth d
is fetched on the order of 2·3^(d−1) times. If the same flow occurs several
times in the tree, that number is multiplied again. Nested contents are also
added to the FlowSnapshotContainer repeatedly.
*Cause 3: every fetch also lists the file's commit history (Git-based clients)*
AbstractGitFlowRegistryClient.getFlowContents calls getLatestVersion on every
fetch, only to set RegisteredFlowSnapshot.isLatest. That lists the commit
history of the snapshot file (GitRepositoryClient.getCommits), which may span
several pages. Each fetch therefore costs at least two REST calls: content plus
history.
*Observed*
A flow with three levels of nesting, where one nested flow is used three times,
on Bitbucket Cloud. A single GET /process-groups/\{id}/local-modifications ran
for more than 4.5 minutes. During that time it fetched individual nested flows
dozens of times (e.g. Nested.json 44 times, still running). One request thread
produced more than 1,200 Bitbucket client log entries. Earlier, putting the
same group under version control fetched the deepest nested flows 108 times
each and took ~35 s.
*Impact*
* getLocalModifications holds the flow read lock
(NiFiServiceFacadeLock.proceedWithReadLock) for the whole time. Every write
request waits behind it, e.g. deleting the very process group.
* Synchronous REST calls such as starting version control and importing a flow
can outlast client read timeouts. The operation still completes on the server,
and a client retry then fails with 409 ("process group is already under version
control") or creates a duplicate group.
* The hosted Git APIs see hundreds of calls per user action, which risks rate
limiting.
*Steps to reproduce*
# Configure a Git-based Flow Registry Client, e.g. the Bitbucket or GitHub
client.
# Version flows B and C. Version flow A so that it contains B, and B contains
C. Optionally reference B more than once inside A.
# Import A into the canvas, or call GET
/nifi-api/process-groups/\{id}/local-modifications on it.
# Enable DEBUG logging for the registry client. Each nested snapshot file is
requested many times, and the count grows with depth.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
