[
https://issues.apache.org/jira/browse/NIP-36?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Paulin updated NIP-36:
--------------------------
Description:
h2. Motivation
Some Connector configuration properties describe decisions that cannot be
safely changed after the Connector has started producing data. Typical examples:
* The output format written to a destination. Once objects have been written
to an S3 bucket as Parquet, silently switching the property to Avro produces a
bucket whose contents can no longer be read by a single reader.
* The physical destination identifier. The S3 bucket, Kafka topic, or
object-store prefix that the Connector writes into. Changing this after data
has been written silently forks the destination and leaves the previously
written data orphaned.
* The identity used to track source progress. The Kafka consumer group id or
source offset-tracking key. Changing this after commit either re-reads
already-processed data or leaves committed offsets behind, in both cases
producing duplicates or gaps.
Today there is no first-class way for a Connector author to protect such a
property. The intent lives only in ad hoc validation code inside each
Connector, is not exposed to the REST API, is not visible in the extension
manifest, and is not visible in the UI. Users can freely edit these fields in
the configuration dialog and see a generic error only when they attempt to
apply the change, if the Connector happens to defend it at all.
These decisions are not permanent, though. A user may deliberately move to a
new bucket, switch formats, or replay a topic from the beginning. Today the
only safe way to do that is to delete the Connector and create a new one,
losing the rest of its configuration and giving the Connector no chance to
clean up what the old value left behind.
h2. Scope
Out of scope:
Lockable properties for Processors or Controller Services are not part of the
scope of this NIP. Processors and Controller Services have done without this
feature and have implemented ways to handle property value changes within the
component code. Connectors integrate an entire flow worth of components
together, so forcing Connectors to handle any possible Connector property
change may not always be possible.
{{nifi-api}} additions:
* {{ConnectorPropertyDescriptor.isLockable()}} and
{{{}ConnectorPropertyDescriptor.Builder.lockable(boolean){}}}, defaulting to
{{{}false{}}}. {{Builder.from(other)}} copies the flag.
* {{VersionedConnector.lockedProperties}} with getter and setter.
* {{UnlockContext}} and {{{}Connector.prepareForUnlock(UnlockContext){}}},
with a default implementation.
* {{XmlConnectorDocumentationWriter.writeConnectorProperty(...)}} emits a new
{{<lockable>}} element.
Framework additions:
* {{ConnectorNode}} methods to read and restore the locked set, and
{{{}unlock(...){}}}.
* {{StandardConnectorNode}} records locks, enforces them, and advertises an
{{UNLOCK}} action through {{{}getAvailableActions(){}}}.
* {{VersionedComponentFlowMapper.mapConnector(...)}} persists the locked set,
and {{StandardConnectorRepository.createConnectorNode(VersionedConnector)}}
restores it.
Model, DTO, manifest, and C2 plumbing:
* {{ConnectorPropertyDescriptorDTO}} ({{{}nifi-client-dto{}}}) gains
{{{}lockable{}}}, populated by
{{{}DtoFactory.createConnectorPropertyDescriptorDto(...){}}}.
* {{ConnectorDTO}} ({{{}nifi-client-dto{}}}) gains {{{}lockedProperties{}}},
populated by {{{}DtoFactory{}}}.
* {{org.apache.nifi.extension.manifest.ConnectorProperty}} and
{{org.apache.nifi.c2.protocol.component.api.ConnectorPropertyDescriptor}} gain
{{{}lockable{}}}.
{{StandardRuntimeManifestBuilder.getConnectorPropertyDescriptor(...)}} copies
it through.
* The Connector REST resource gains an unlock endpoint, following the existing
Connector action endpoints.
UI additions ({{{}nifi-frontend{}}}):
* The Connector property editor shows lock status, and offers unlock and reset
controls.
h2. Description
h3. Descriptor contract
{{lockable}} is a static, author-declared strategy on the shared
{{{}ConnectorPropertyDescriptor{}}}. It does not describe the state of any
Connector instance. A lockable property behaves like any other property until
the Connector successfully transitions to {{{}RUNNING{}}}, so initial
configuration, verification, and staged flow deployment are unaffected.
h3. Lock status
Each Connector instance keeps a persisted set of locked property identities,
exposed as {{{}lockedProperties{}}}. An identity is a configuration step name
plus a property name; {{ConfigurationStep.Builder}} already requires property
names to be unique within a step.
A property is LOCKED when its identity is in the set and its current descriptor
is {{{}lockable{}}}. Otherwise it is UNLOCKED.
On every successful transition to {{{}RUNNING{}}}, the framework adds every
property whose current descriptor is {{{}lockable{}}}, locking the value in the
active configuration at that moment. No other state transition adds or removes
locks, including entering or leaving {{{}TROUBLESHOOTING{}}}. Unlock is the
only operation that removes identities.
h3. Enforcement
A single private helper,
{{{}StandardConnectorNode.verifyLockedPropertiesUnchanged(...){}}}, is called
from the two mutation entry points a user request can reach:
* {{{}setConfiguration(String, StepConfiguration){}}}: the merge path used by
REST configuration updates and by {{{}ConnectorTestRunner.configure(...){}}}.
* {{{}replaceWorkingConfiguration(...){}}}: the whole-configuration
replacement path used by configuration providers.
For each LOCKED property, the helper compares the incoming raw
{{ConnectorValueReference}} with the active one, so asset and secret references
are compared by identity, not by resolved value. An equal reference is
accepted. A different reference throws {{FlowUpdateException}} naming the
property, the step, and the Connector.
Restore paths (flow deserialization, snapshot inheritance,
{{{}recreateWorkingFlowContext{}}}) bypass the check because they reconstruct
existing state.
!image-2026-10-16-07-49-01-251.png!
h3. Unlock and reset
{{UNLOCK}} is a framework Connector action alongside existing actions such as
{{START}} and {{{}PURGE_FLOWFILES{}}}. It is allowed when the Connector is
stopped, is not in Troubleshooting, and has at least one LOCKED property.
Otherwise it is listed as disallowed with a reason.
The request carries an optional set of property identities:
* No identities resets the Connector: every LOCKED property is unlocked, and
{{UnlockContext.isReset()}} is true.
* A non-empty set unlocks only those identities. If any identity is not
currently LOCKED, the request fails and nothing is removed.
Before changing the set, the framework calls
{{{}Connector.prepareForUnlock(UnlockContext){}}}. The locked values and the
active flow are still in place. {{UnlockContext}} provides the identities, the
active {{ConnectorValueReference}} for each, the active flow context, and
{{{}isReset(){}}}.
The Connector uses this hook to drop committed offsets, abandon a destination,
or refuse the change. It refuses synchronously by throwing
{{{}FlowUpdateException{}}}, or asynchronously by completing the returned
{{CompletableFuture}} exceptionally. Either keeps every lock. The framework
cancels the future if the caller cancels the request, so delegated work must be
cooperatively cancellable, and the future must not complete while other work
can still mutate the flow. The default implementation returns a completed
future.
When the future completes normally, the framework removes the identities and
persists the Connector. Active values do not change. The user then edits the
properties through the normal configuration path, so validation,
{{{}onConfigurationStepConfigured{}}}, {{{}prepareForUpdate{}}}, and
{{applyUpdate}} all still apply.
The next successful start locks them again.
h3. Persistence and upgrades
{{VersionedComponentFlowMapper}} writes the locked set to
{{{}VersionedConnector{}}}. {{StandardConnectorRepository}} restores it before
the Connector accepts user requests, so enforcement is active immediately after
restart. The set, including the result of a partial unlock, survives restarts,
flow imports, and Connector version upgrades.
Because locks are tracked per property:
* A property that becomes lockable only in a later Connector version stays
editable until the next successful start.
* An identity whose current descriptor is not lockable is ignored. Removing
{{lockable}} in a later version is how an author retires the constraint.
* A renamed property is a new identity and locks on the next successful start.
Old identities are not remapped.
was:
h2. Motivation
Some Connector configuration properties describe decisions that cannot be
safely changed after the Connector has started producing data. Typical examples:
* The output format written to a destination. Once objects have been written
to an S3 bucket as Parquet, silently switching the property to Avro produces a
bucket whose contents can no longer be read by a single reader.
* The physical destination identifier. The S3 bucket, Kafka topic, or
object-store prefix that the Connector writes into. Changing this after data
has been written silently forks the destination and leaves the previously
written data orphaned.
* The identity used to track source progress. The Kafka consumer group id or
source offset-tracking key. Changing this after commit either re-reads
already-processed data or leaves committed offsets behind, in both cases
producing duplicates or gaps.
Today there is no first-class way for a Connector author to declare that such a
property is frozen after the Connector has started. The intent lives only in ad
hoc validation code inside each Connector, is not exposed to the REST API, is
not visible in the extension manifest, and is not visible in the UI. Users can
freely edit these fields in the configuration dialog and see a generic error
only when they attempt to apply the change, if the Connector happens to defend
it at all.
This proposal makes _"once the Connector has been started at least once, this
property is frozen at its current value"_ a first-class descriptor attribute,
enforced by the framework, surfaced through the REST API and extension
manifest, and rendered as read-only in the UI so the user never has the
opportunity to make an unappliable change in the first place.
h2. Scope
Out of Scope:
Immutable Properties for Processors or Controller Services are not part of the
scope of this NIP. Processors or Controller Services have done without this
feature and have implemented ways to handle property value changes within the
component code. Connectors integrate an entire flow worth of components
together so trying to force connectors to handle any possible connector
property change may not always be possible.
{{nifi-api}} additions:
* {{ConnectorPropertyDescriptor.isImmutable()}} and
{{{}ConnectorPropertyDescriptor.Builder.immutable(boolean){}}}, defaulting to
{{{}false{}}}. {{Builder.from(other)}} copies the flag.
* {{VersionedConnector.hasBeenStarted}} ({{{}boolean{}}}) with getter/setter,
serialized alongside the Connector's other flow-level metadata.
* {{XmlConnectorDocumentationWriter.writeConnectorProperty(...)}} emits a new
{{<immutable>}} element so the extension manifest exposes the flag.
Framework additions:
* New methods on {{{}org.apache.nifi.components.connector.ConnectorNode{}}}:
{{boolean hasBeenStarted()}} and {{{}void setHasBeenStarted(boolean){}}}.
* {{StandardConnectorNode}} sets {{hasBeenStarted = true}} after every
successful transition to {{{}RUNNING{}}}, and rejects changes to any immutable
property in {{setConfiguration(...)}} and {{replaceWorkingConfiguration(...)}}
once the flag is set. Comparison is done on the raw {{ConnectorValueReference}}
instances so asset and secret references are compared by identity, not by their
resolved values.
* {{VersionedComponentFlowMapper.mapConnector(...)}} persists the flag;
{{StandardConnectorRepository.createConnectorNode(VersionedConnector)}}
restores it via {{{}ConnectorNode.setHasBeenStarted(...){}}}. The flag survives
restarts, flow imports, and Connector-version upgrades.
Model / DTO / manifest plumbing:
* {{ConnectorPropertyDescriptorDTO}} (nifi-client-dto) gains an {{immutable}}
field, populated by
{{{}DtoFactory.createConnectorPropertyDescriptorDto(...){}}}.
* {{ConnectorDTO}} (nifi-client-dto) gains a {{hasBeenStarted}} field,
populated by {{DtoFactory}} from {{{}ConnectorNode.hasBeenStarted(){}}}, so
that any client can determine whether a Connector's immutable properties are
currently locked.
* {{org.apache.nifi.extension.manifest.ConnectorProperty}} and
{{org.apache.nifi.c2.protocol.component.api.ConnectorPropertyDescriptor}} gain
the same {{immutable}} field;
{{StandardRuntimeManifestBuilder.getConnectorPropertyDescriptor(...)}} copies
it through.
UI additions ({{{}nifi-frontend{}}}):
* The Connector property-editor components consume {{descriptor.immutable}}
together with {{{}connector.hasBeenStarted{}}}. When both are true, the
corresponding form control is rendered as disabled and no editable value
control is offered.
* A locked visual indicator (padlock icon plus tooltip text) is displayed next
to the disabled control, explaining that the property was fixed at the value
from the Connector's first successful start and cannot be changed..
h2. Description
Descriptor contract.
A property built with {{.immutable(true)}} behaves identically to a
non-immutable property until the Connector transitions to {{RUNNING}} for the
first time. Attempts to change it before that point succeed, which preserves
normal initial configuration, verification, and staged flow deployment.
Immutability begins the moment {{StandardConnectorNode}} sets
{{{}hasBeenStarted = true{}}}, and from then on the value locked in is whatever
was in the active configuration at the point of transition.
Enforcement point.
The check lives in a single private helper —
{{StandardConnectorNode.verifyImmutablePropertiesUnchanged(stepName, incoming,
replace)}} — called from the two mutation entry points a user request can reach:
* {{setConfiguration(String, StepConfiguration)}} — the merge path used by
REST configuration updates and by {{{}ConnectorTestRunner.configure(...){}}}.
* {{replaceWorkingConfiguration(String, StepConfiguration)}} — the whole-step
replacement path used by configuration providers.
Restore paths (flow deserialization, snapshot inheritance,
{{{}recreateWorkingFlowContext{}}}) deliberately bypass the check because they
must be able to reconstruct the Connector's existing state without the
framework second-guessing itself. A violating change throws
{{FlowUpdateException}} with a message that names the offending property, the
step, and the Connector.
Persistence.
{{hasBeenStarted}} is stored on {{{}VersionedConnector{}}}, not on the
descriptor. On save, {{VersionedComponentFlowMapper}} reads
{{{}connectorNode.hasBeenStarted(){}}}. On load,
{{StandardConnectorRepository}} reads the field off the {{VersionedConnector}}
and calls {{setHasBeenStarted(true)}} before the Connector is exposed to any
user request, so enforcement is active immediately after restart. Because the
flag lives on the Connector and not on the descriptor, values locked under
Connector v1 remain locked under Connector v2 even if v2 changes the descriptor
set: whether a property is immutable is decided by the current v2 descriptors,
and _"the Connector has been started"_ is decided by the persisted flag.
Backwards compatibility.
{{isImmutable()}} defaults to {{{}false{}}}, so existing Connectors are
unaffected. {{VersionedConnector.hasBeenStarted}} defaults to {{false}} in the
JSON representation, so flows saved by older frameworks come back unlocked,
which is the same behaviour as today. The new DTO / manifest / c2 fields are
additive, and building an older Connector against the new {{nifi-api}}
continues to compile because both new descriptor methods have default
implementations.
> Lockable Connector Properties
> -----------------------------
>
> Key: NIP-36
> URL: https://issues.apache.org/jira/browse/NIP-36
> Project: NiFi Improvement Proposal
> Issue Type: Improvement
> Reporter: Bob Paulin
> Priority: Major
> Attachments: image-2026-10-16-07-49-01-251.png
>
>
> h2. Motivation
> Some Connector configuration properties describe decisions that cannot be
> safely changed after the Connector has started producing data. Typical
> examples:
> * The output format written to a destination. Once objects have been written
> to an S3 bucket as Parquet, silently switching the property to Avro produces
> a bucket whose contents can no longer be read by a single reader.
> * The physical destination identifier. The S3 bucket, Kafka topic, or
> object-store prefix that the Connector writes into. Changing this after data
> has been written silently forks the destination and leaves the previously
> written data orphaned.
> * The identity used to track source progress. The Kafka consumer group id or
> source offset-tracking key. Changing this after commit either re-reads
> already-processed data or leaves committed offsets behind, in both cases
> producing duplicates or gaps.
> Today there is no first-class way for a Connector author to protect such a
> property. The intent lives only in ad hoc validation code inside each
> Connector, is not exposed to the REST API, is not visible in the extension
> manifest, and is not visible in the UI. Users can freely edit these fields in
> the configuration dialog and see a generic error only when they attempt to
> apply the change, if the Connector happens to defend it at all.
> These decisions are not permanent, though. A user may deliberately move to a
> new bucket, switch formats, or replay a topic from the beginning. Today the
> only safe way to do that is to delete the Connector and create a new one,
> losing the rest of its configuration and giving the Connector no chance to
> clean up what the old value left behind.
> h2. Scope
> Out of scope:
> Lockable properties for Processors or Controller Services are not part of the
> scope of this NIP. Processors and Controller Services have done without this
> feature and have implemented ways to handle property value changes within the
> component code. Connectors integrate an entire flow worth of components
> together, so forcing Connectors to handle any possible Connector property
> change may not always be possible.
> {{nifi-api}} additions:
> * {{ConnectorPropertyDescriptor.isLockable()}} and
> {{{}ConnectorPropertyDescriptor.Builder.lockable(boolean){}}}, defaulting to
> {{{}false{}}}. {{Builder.from(other)}} copies the flag.
> * {{VersionedConnector.lockedProperties}} with getter and setter.
> * {{UnlockContext}} and {{{}Connector.prepareForUnlock(UnlockContext){}}},
> with a default implementation.
> * {{XmlConnectorDocumentationWriter.writeConnectorProperty(...)}} emits a
> new {{<lockable>}} element.
> Framework additions:
> * {{ConnectorNode}} methods to read and restore the locked set, and
> {{{}unlock(...){}}}.
> * {{StandardConnectorNode}} records locks, enforces them, and advertises an
> {{UNLOCK}} action through {{{}getAvailableActions(){}}}.
> * {{VersionedComponentFlowMapper.mapConnector(...)}} persists the locked
> set, and
> {{StandardConnectorRepository.createConnectorNode(VersionedConnector)}}
> restores it.
> Model, DTO, manifest, and C2 plumbing:
> * {{ConnectorPropertyDescriptorDTO}} ({{{}nifi-client-dto{}}}) gains
> {{{}lockable{}}}, populated by
> {{{}DtoFactory.createConnectorPropertyDescriptorDto(...){}}}.
> * {{ConnectorDTO}} ({{{}nifi-client-dto{}}}) gains {{{}lockedProperties{}}},
> populated by {{{}DtoFactory{}}}.
> * {{org.apache.nifi.extension.manifest.ConnectorProperty}} and
> {{org.apache.nifi.c2.protocol.component.api.ConnectorPropertyDescriptor}}
> gain {{{}lockable{}}}.
> {{StandardRuntimeManifestBuilder.getConnectorPropertyDescriptor(...)}} copies
> it through.
> * The Connector REST resource gains an unlock endpoint, following the
> existing Connector action endpoints.
> UI additions ({{{}nifi-frontend{}}}):
> * The Connector property editor shows lock status, and offers unlock and
> reset controls.
> h2. Description
> h3. Descriptor contract
> {{lockable}} is a static, author-declared strategy on the shared
> {{{}ConnectorPropertyDescriptor{}}}. It does not describe the state of any
> Connector instance. A lockable property behaves like any other property until
> the Connector successfully transitions to {{{}RUNNING{}}}, so initial
> configuration, verification, and staged flow deployment are unaffected.
> h3. Lock status
> Each Connector instance keeps a persisted set of locked property identities,
> exposed as {{{}lockedProperties{}}}. An identity is a configuration step name
> plus a property name; {{ConfigurationStep.Builder}} already requires property
> names to be unique within a step.
> A property is LOCKED when its identity is in the set and its current
> descriptor is {{{}lockable{}}}. Otherwise it is UNLOCKED.
> On every successful transition to {{{}RUNNING{}}}, the framework adds every
> property whose current descriptor is {{{}lockable{}}}, locking the value in
> the active configuration at that moment. No other state transition adds or
> removes locks, including entering or leaving {{{}TROUBLESHOOTING{}}}. Unlock
> is the only operation that removes identities.
> h3. Enforcement
> A single private helper,
> {{{}StandardConnectorNode.verifyLockedPropertiesUnchanged(...){}}}, is called
> from the two mutation entry points a user request can reach:
> * {{{}setConfiguration(String, StepConfiguration){}}}: the merge path used
> by REST configuration updates and by
> {{{}ConnectorTestRunner.configure(...){}}}.
> * {{{}replaceWorkingConfiguration(...){}}}: the whole-configuration
> replacement path used by configuration providers.
> For each LOCKED property, the helper compares the incoming raw
> {{ConnectorValueReference}} with the active one, so asset and secret
> references are compared by identity, not by resolved value. An equal
> reference is accepted. A different reference throws {{FlowUpdateException}}
> naming the property, the step, and the Connector.
> Restore paths (flow deserialization, snapshot inheritance,
> {{{}recreateWorkingFlowContext{}}}) bypass the check because they reconstruct
> existing state.
> !image-2026-10-16-07-49-01-251.png!
> h3. Unlock and reset
> {{UNLOCK}} is a framework Connector action alongside existing actions such as
> {{START}} and {{{}PURGE_FLOWFILES{}}}. It is allowed when the Connector is
> stopped, is not in Troubleshooting, and has at least one LOCKED property.
> Otherwise it is listed as disallowed with a reason.
> The request carries an optional set of property identities:
> * No identities resets the Connector: every LOCKED property is unlocked, and
> {{UnlockContext.isReset()}} is true.
> * A non-empty set unlocks only those identities. If any identity is not
> currently LOCKED, the request fails and nothing is removed.
> Before changing the set, the framework calls
> {{{}Connector.prepareForUnlock(UnlockContext){}}}. The locked values and the
> active flow are still in place. {{UnlockContext}} provides the identities,
> the active {{ConnectorValueReference}} for each, the active flow context, and
> {{{}isReset(){}}}.
> The Connector uses this hook to drop committed offsets, abandon a
> destination, or refuse the change. It refuses synchronously by throwing
> {{{}FlowUpdateException{}}}, or asynchronously by completing the returned
> {{CompletableFuture}} exceptionally. Either keeps every lock. The framework
> cancels the future if the caller cancels the request, so delegated work must
> be cooperatively cancellable, and the future must not complete while other
> work can still mutate the flow. The default implementation returns a
> completed future.
> When the future completes normally, the framework removes the identities and
> persists the Connector. Active values do not change. The user then edits the
> properties through the normal configuration path, so validation,
> {{{}onConfigurationStepConfigured{}}}, {{{}prepareForUpdate{}}}, and
> {{applyUpdate}} all still apply.
>
> The next successful start locks them again.
>
> h3. Persistence and upgrades
> {{VersionedComponentFlowMapper}} writes the locked set to
> {{{}VersionedConnector{}}}. {{StandardConnectorRepository}} restores it
> before the Connector accepts user requests, so enforcement is active
> immediately after restart. The set, including the result of a partial unlock,
> survives restarts, flow imports, and Connector version upgrades.
> Because locks are tracked per property:
> * A property that becomes lockable only in a later Connector version stays
> editable until the next successful start.
> * An identity whose current descriptor is not lockable is ignored. Removing
> {{lockable}} in a later version is how an author retires the constraint.
> * A renamed property is a new identity and locks on the next successful
> start. Old identities are not remapped.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
