Bob Paulin created NIP-36:
-----------------------------
Summary: Add immutable option to ConnectorPropertyDescriptor
Key: NIP-36
URL: https://issues.apache.org/jira/browse/NIP-36
Project: NiFi Improvement Proposal
Issue Type: Improvement
Reporter: Bob Paulin
h2. Motivation
Some Connector configuration properties describe decisions that cannot be
safely changed after the Connector has started producing data. Typical examples:
* The output format written to a destination. Once objects have been written
to an S3 bucket as Parquet, silently switching the property to Avro produces a
bucket whose contents can no longer be read by a single reader.
* The physical destination identifier. The S3 bucket, Kafka topic, or
object-store prefix that the Connector writes into. Changing this after data
has been written silently forks the destination and leaves the previously
written data orphaned.
* The identity used to track source progress. The Kafka consumer group id or
source offset-tracking key. Changing this after commit either re-reads
already-processed data or leaves committed offsets behind, in both cases
producing duplicates or gaps.
Today there is no first-class way for a Connector author to declare that such a
property is frozen after the Connector has started. The intent lives only in ad
hoc validation code inside each Connector, is not exposed to the REST API, is
not visible in the extension manifest, and is not visible in the UI. Users can
freely edit these fields in the configuration dialog and see a generic error
only when they attempt to apply the change, if the Connector happens to defend
it at all.
This proposal makes _"once the Connector has been started at least once, this
property is frozen at its current value"_ a first-class descriptor attribute,
enforced by the framework, surfaced through the REST API and extension
manifest, and rendered as read-only in the UI so the user never has the
opportunity to make an unappliable change in the first place.
h2. Scope
Out of Scope:
Immutable Properties for Processors or Controller Services are not part of the
scope of this NIP. Processors or Controller Services have done without this
feature and have implemented ways to handle property value changes within the
component code. Connectors integrate an entire flow worth of components
together so trying to force connectors to handle any possible connector
property change may not always be possible.
{{nifi-api}} additions:
* {{ConnectorPropertyDescriptor.isImmutable()}} and
{{{}ConnectorPropertyDescriptor.Builder.immutable(boolean){}}}, defaulting to
{{{}false{}}}. {{Builder.from(other)}} copies the flag.
* {{VersionedConnector.hasBeenStarted}} ({{{}boolean{}}}) with getter/setter,
serialized alongside the Connector's other flow-level metadata.
* {{XmlConnectorDocumentationWriter.writeConnectorProperty(...)}} emits a new
{{<immutable>}} element so the extension manifest exposes the flag.
Framework additions:
* New methods on {{{}org.apache.nifi.components.connector.ConnectorNode{}}}:
{{boolean hasBeenStarted()}} and {{{}void setHasBeenStarted(boolean){}}}.
* {{StandardConnectorNode}} sets {{hasBeenStarted = true}} after every
successful transition to {{{}RUNNING{}}}, and rejects changes to any immutable
property in {{setConfiguration(...)}} and {{replaceWorkingConfiguration(...)}}
once the flag is set. Comparison is done on the raw {{ConnectorValueReference}}
instances so asset and secret references are compared by identity, not by their
resolved values.
* {{VersionedComponentFlowMapper.mapConnector(...)}} persists the flag;
{{StandardConnectorRepository.createConnectorNode(VersionedConnector)}}
restores it via {{{}ConnectorNode.setHasBeenStarted(...){}}}. The flag survives
restarts, flow imports, and Connector-version upgrades.
Model / DTO / manifest plumbing:
* {{ConnectorPropertyDescriptorDTO}} (nifi-client-dto) gains an {{immutable}}
field, populated by
{{{}DtoFactory.createConnectorPropertyDescriptorDto(...){}}}.
* {{ConnectorDTO}} (nifi-client-dto) gains a {{hasBeenStarted}} field,
populated by {{DtoFactory}} from {{{}ConnectorNode.hasBeenStarted(){}}}, so
that any client can determine whether a Connector's immutable properties are
currently locked.
* {{org.apache.nifi.extension.manifest.ConnectorProperty}} and
{{org.apache.nifi.c2.protocol.component.api.ConnectorPropertyDescriptor}} gain
the same {{immutable}} field;
{{StandardRuntimeManifestBuilder.getConnectorPropertyDescriptor(...)}} copies
it through.
UI additions ({{{}nifi-frontend{}}}):
* The Connector property-editor components consume {{descriptor.immutable}}
together with {{{}connector.hasBeenStarted{}}}. When both are true, the
corresponding form control is rendered as disabled and no editable value
control is offered.
* A locked visual indicator (padlock icon plus tooltip text) is displayed next
to the disabled control, explaining that the property was fixed at the value
from the Connector's first successful start and cannot be changed..
h2. Description
Descriptor contract.
A property built with {{.immutable(true)}} behaves identically to a
non-immutable property until the Connector transitions to {{RUNNING}} for the
first time. Attempts to change it before that point succeed, which preserves
normal initial configuration, verification, and staged flow deployment.
Immutability begins the moment {{StandardConnectorNode}} sets
{{{}hasBeenStarted = true{}}}, and from then on the value locked in is whatever
was in the active configuration at the point of transition.
Enforcement point.
The check lives in a single private helper —
{{StandardConnectorNode.verifyImmutablePropertiesUnchanged(stepName, incoming,
replace)}} — called from the two mutation entry points a user request can reach:
* {{setConfiguration(String, StepConfiguration)}} — the merge path used by
REST configuration updates and by {{{}ConnectorTestRunner.configure(...){}}}.
* {{replaceWorkingConfiguration(String, StepConfiguration)}} — the whole-step
replacement path used by configuration providers.
Restore paths (flow deserialization, snapshot inheritance,
{{{}recreateWorkingFlowContext{}}}) deliberately bypass the check because they
must be able to reconstruct the Connector's existing state without the
framework second-guessing itself. A violating change throws
{{FlowUpdateException}} with a message that names the offending property, the
step, and the Connector.
Persistence.
{{hasBeenStarted}} is stored on {{{}VersionedConnector{}}}, not on the
descriptor. On save, {{VersionedComponentFlowMapper}} reads
{{{}connectorNode.hasBeenStarted(){}}}. On load,
{{StandardConnectorRepository}} reads the field off the {{VersionedConnector}}
and calls {{setHasBeenStarted(true)}} before the Connector is exposed to any
user request, so enforcement is active immediately after restart. Because the
flag lives on the Connector and not on the descriptor, values locked under
Connector v1 remain locked under Connector v2 even if v2 changes the descriptor
set: whether a property is immutable is decided by the current v2 descriptors,
and _"the Connector has been started"_ is decided by the persisted flag.
Backwards compatibility.
{{isImmutable()}} defaults to {{{}false{}}}, so existing Connectors are
unaffected. {{VersionedConnector.hasBeenStarted}} defaults to {{false}} in the
JSON representation, so flows saved by older frameworks come back unlocked,
which is the same behaviour as today. The new DTO / manifest / c2 fields are
additive, and building an older Connector against the new {{nifi-api}}
continues to compile because both new descriptor methods have default
implementations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
