Xinyu Wang created NIFI-15889:
---------------------------------
Summary: Cache Parameter Context name mapping to fix O(N_PG ×
N_PC^2) startup synchronization
Key: NIFI-15889
URL: https://issues.apache.org/jira/browse/NIFI-15889
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Affects Versions: 2.5.0
Reporter: Xinyu Wang
On NiFi startup, VersionedFlowSynchronizer synchronizes all child Process
Groups, and for each PG it iterates the versioned Parameter Contexts and calls
StandardParameterContextManager.getParameterContextNameMapping(). That method
rebuilds the entire name→PC HashMap from a stream collect on every call, giving
the synchronization an effective complexity of:
O(N_PG × N_PC^2)
For flows with thousands of Parameter Contexts the rebuild dominates startup
time and turns a minute-scale boot into an hour-scale one.
Affected code
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/parameter/StandardParameterContextManager.java
@Override
public Map<String, ParameterContext> getParameterContextNameMapping() {
return parameterContexts.values().stream()
.collect(Collectors.toMap(ParameterContext::getName,
Function.identity()));
}
Hot caller:
StandardVersionedComponentSynchronizer#createParameterContextWithoutReferences,
which is invoked from addProcessGroup for every child PG. The same name mapping
is rebuilt N_PG times even though the underlying parameterContexts map barely
changes between calls.
Proposed fix
Maintain a name index incrementally and cache an unmodifiable view; reads
become O(1), and the unique-name check at insert becomes O(1) too.
private final Map<String, ParameterContext> parameterContexts = new
HashMap<>();
private final Map<String, ParameterContext> nameIndex = new HashMap<>();
private volatile Map<String, ParameterContext> cachedNameMapping =
Collections.emptyMap();
@Override
public synchronized void addParameterContext(final ParameterContext pc) {
Objects.requireNonNull(pc);
final ParameterContext existingById =
parameterContexts.get(pc.getIdentifier());
if (existingById != null && !(existingById instanceof
ReferenceOnlyParameterContext)) {
throw new IllegalStateException("Cannot add Parameter Context because
another Parameter Context already exists with the same ID");
}
final ParameterContext existingByName = nameIndex.get(pc.getName());
if (existingByName != null && existingByName != existingById) {
throw new IllegalStateException("Cannot add Parameter Context because
another Parameter Context already exists with the name '" + pc + "'");
}
// When replacing a ReferenceOnlyParameterContext placeholder, its name
// differs from the real PC name, so its stale entry must be removed
// from nameIndex first.
if (existingById instanceof ReferenceOnlyParameterContext) {
nameIndex.remove(existingById.getName(), existingById);
}
parameterContexts.put(pc.getIdentifier(), pc);
nameIndex.put(pc.getName(), pc);
cachedNameMapping = Collections.unmodifiableMap(new HashMap<>(nameIndex));
}
@Override
public synchronized ParameterContext removeParameterContext(final String id) {
Objects.requireNonNull(id);
final ParameterContext removed = parameterContexts.remove(id);
if (removed != null) {
nameIndex.remove(removed.getName(), removed);
cachedNameMapping = Collections.unmodifiableMap(new
HashMap<>(nameIndex));
}
return removed;
}
@Override
public Map<String, ParameterContext> getParameterContextNameMapping() {
return cachedNameMapping;
}
Single file, ~25 lines. Complexity changes from O(N_PG × N_PC^2) to O(N_PG ×
N_PC).
Subtle case: ReferenceOnlyParameterContext placeholder
ReferenceOnlyParameterContext is used to break the chicken-and-egg cycle when
Parameter Contexts inherit from each other. Its name is "Reference-Only
Parameter Context [<id>]", which is different from the real PC's name. When
addParameterContext is called to replace a placeholder with the real PC, the
placeholder's old name must be explicitly removed from nameIndex; otherwise
VersionedFlowSynchronizer.removeMissingParameterContexts walks the cached name
map, sees "Reference-Only Parameter Context [...]" not in the proposed name
list, looks up its ID, and calls removeParameterContext — deleting the real PC.
Backwards compatibility
- Public method signature is unchanged.
- Returned map is now an unmodifiable view — callers that mutated it would
already have been buggy.
- All mutators remain synchronized.
- Existing unique-name and unique-ID invariants are preserved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
