[
https://issues.apache.org/jira/browse/NIFI-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alaksiej Ščarbaty updated NIFI-16310:
-------------------------------------
Description:
h2. Problem
A Controller Service created by a component's {{migrateProperties}} is removed
and recreated whenever the version of a versioned Process Group changes. During
that the service's state is lost.
A stateless service is unaffected by that. Whereas for stateful service it
causes a silent state loss: the replacement service is correctly configured and
enabled, but it holds no state from the removed controller service.
_Related jiras:_ NIFI-14995 and NIFI-16149 stopped these services being
reported as local modifications when a flow is compared with its registered
version. That handling does not extend to applying a version change.
h2. Root cause
{{StandardControllerServiceFactory.create}} does not assign a versioned
component id to the service it creates.
{{StandardVersionedComponentSynchronizer}} generates a new versioned id from
the service's instance identifier for any controller services without a
versioned id. Then it removes every local controller service whose version id
is absent from the proposed snapshot. There is no exception for controller
services created in {{{}migrateProperties{}}}.
h2. Proposed change
The case to support first: a Process Group is upgraded to a version that
declares the Controller Service which was already created locally in
{{{}migrateProperties{}}}. The service must survive with its current state, and
the flow must end up with one controller service, not two.
# Synchronization must not delete a Controller Service that was created in
{{{}migrateProperties{}}}.
# This local Controller Service must be set with a versioned component id
assigned to that Controller Service in the versioned process group.
The matching of Controller Services should go through the component that
references them: if a component's Controller Service property points at a
migration-created service now, and the same property in the new version points
at a declared service of the same type in the same group, they must be treated
as the same service.
After matching the created Controller Service looks identical to its
counterpart in the versioned process group, and it should carry no signs of
being created in {{migrateProperties.}}
h3. Other matching strategies
Neither the identifier nor the versioned component id can be used to match
these two services.
Matching by class and property values work well only when controller service
created by {{migrateProperties}} is identical to a service in the versioned
group (see edge cases for properties drift). This matching strategy also won't
work if there are 2 identical controller services in the versioned group.
h2. Edge cases
This solution creates a lot of edge cases, most of which are caused by
Controller Service implementation/properties drift.
Normally a versioned process group author must ensure there is no properties
drift between a Controller Service created in {{migrateProperties}} and the
Service from versioned process group.
For each edge case I provide synchronization behavior.
*New version does not declare the service.* Keep the local service,
unversioned. Currently it is removed and then recreated empty by calling
{{{}migrateProperties{}}}.
*Declared service has different property values* {*}than the local
service{*}{*}.{*} The local service becomes versioned, then new properties are
applied to it.
*A single declared service is present instead of several local services.* The
first controller service becomes versioned, the rest are kept as local services.
The following cases are related to Controller Services drift when versioned
Controller Services don't match local Controller Services created in
{{{}migrateProperties{}}}.
If that happens, we create the declared services from the versioned group,
which are used by the components. Local Controller Services are kept and not
removed. The drift has to be resolved manually.
# *New version declares a service of a different implementation type than the
local service.*
# *Declared service is in a different process group from the local service.*
# *A single local service is referenced by several components.*
# *Referencing component is absent from the new version, either deleted or
itself a migration-created service with no versioned component id.*
h2. Reproduction
Reproduced against 2.12.0-SNAPSHOT on main.
System tests:
[https://github.com/awelless/nifi/tree/migration-created-cs-versioning-tests]
{{MigrationCreatedControllerServiceVersioningIT}} uses a processor whose
{{migrateProperties}} creates a stateful Controller SErvice.
* {{testFlowUpgradePreservesMigrationCreatedControllerService}} - fails: the
service is replaced by the one the new version declares;
*
{{testFlowUpgradeAddingUnrelatedProcessorPreservesMigrationCreatedControllerService}}
- fails: the service is recreated under the same id with an empty store;
* -{{testFlowDowngradePreservesMigrationCreatedControllerService}} - fails at
its precondition, because the upgrade it performs first already removes the
service, leaving the downgrade path unverified.- During downgrade we remove a
Controller Service, as it's done for normal Controller Services.
Two further tests pass, showing the NiFi upgrade and a plain NiFi restart are
unaffected.
was:
h2. Problem
A Controller Service created by a component's {{migrateProperties}} is removed
and recreated whenever the version of a versioned Process Group changes. During
that the service's state is lost.
A stateless service is unaffected by that. Whereas for stateful service it
causes a silent state loss: the replacement service is correctly configured and
enabled, but it holds no state from the removed controller service.
_Related jiras:_ NIFI-14995 and NIFI-16149 stopped these services being
reported as local modifications when a flow is compared with its registered
version. That handling does not extend to applying a version change.
h2. Root cause
{{StandardControllerServiceFactory.create}} does not assign a versioned
component id to the service it creates.
{{StandardVersionedComponentSynchronizer}} generates a new versioned id from
the service's instance identifier for any controller services without a
versioned id. Then it removes every local controller service whose version id
is absent from the proposed snapshot. There is no exception for controller
services created in {{{}migrateProperties{}}}.
h2. Proposed change
The case to support first: a Process Group is upgraded to a version that
declares the Controller Service which was already created locally in
{{{}migrateProperties{}}}. The service must survive with its current state, and
the flow must end up with one controller service, not two.
# Synchronization must not delete a Controller Service that was created in
{{{}migrateProperties{}}}.
# This local Controller Service must be set with a versioned component id
assigned to that Controller Service in the versioned process group.
The matching of Controller Services should go through the component that
references them: if a component's Controller Service property points at a
migration-created service now, and the same property in the new version points
at a declared service of the same type in the same group, they must be treated
as the same service.
After matching the created Controller Service looks identical to its
counterpart in the versioned process group, and it should carry no signs of
being created in {{migrateProperties.}}
h3. Other matching strategies
Neither the identifier nor the versioned component id can be used to match
these two services.
Matching by class and property values work well only when controller service
created by {{migrateProperties}} is identical to a service in the versioned
group (see edge cases for properties drift). This matching strategy also won't
work if there are 2 identical controller services in the versioned group.
h2. Edge cases
This solution creates a lot of edge cases, most of which are caused by
Controller Service implementation/properties drift.
Normally a versioned process group author must ensure there is no properties
drift between a Controller Service created in {{migrateProperties}} and the
Service from versioned process group.
For each edge case I provide synchronization behavior.
*New version does not declare the service.* Keep the local service,
unversioned. Currently it is removed and then recreated empty by calling
{{{}migrateProperties{}}}.
*Declared service has different property values* {*}than the local
service{*}{*}.{*} The local service becomes versioned, then new properties are
applied to it.
*A single declared service is present instead of several local services.* The
first controller service becomes versioned, the rest are kept as local services.
The following cases are related to Controller Services drift when versioned
Controller Services don't match local Controller Services created in
{{{}migrateProperties{}}}.
If that happens, we create the declared services from the versioned group,
which are used by the components. Local Controller Services are kept and not
removed. The drift has to be resolved manually.
# *New version declares a service of a different implementation type than the
local service.*
# *Declared service is in a different process group from the local service.*
# *Multiple declared services are present instead of a single local service
which is referenced by several components.*
# *Referencing component is absent from the new version, either deleted or
itself a migration-created service with no versioned component id.*
h2. Reproduction
Reproduced against 2.12.0-SNAPSHOT on main.
System tests:
[https://github.com/awelless/nifi/tree/migration-created-cs-versioning-tests]
{{MigrationCreatedControllerServiceVersioningIT}} uses a processor whose
{{migrateProperties}} creates a stateful Controller SErvice.
* {{testFlowUpgradePreservesMigrationCreatedControllerService}} - fails: the
service is replaced by the one the new version declares;
*
{{testFlowUpgradeAddingUnrelatedProcessorPreservesMigrationCreatedControllerService}}
- fails: the service is recreated under the same id with an empty store;
* -{{testFlowDowngradePreservesMigrationCreatedControllerService}} - fails at
its precondition, because the upgrade it performs first already removes the
service, leaving the downgrade path unverified.- During downgrade we remove a
Controller Service, as it's done for normal Controller Services.
Two further tests pass, showing the NiFi upgrade and a plain NiFi restart are
unaffected.
> A controller service created by migrateProperties is removed on versioned
> group upgrade
> ---------------------------------------------------------------------------------------
>
> Key: NIFI-16310
> URL: https://issues.apache.org/jira/browse/NIFI-16310
> Project: Apache NiFi
> Issue Type: Bug
> Components: Flow Versioning
> Affects Versions: 2.11.0
> Reporter: Alaksiej Ščarbaty
> Assignee: Alaksiej Ščarbaty
> Priority: Major
>
> h2. Problem
> A Controller Service created by a component's {{migrateProperties}} is
> removed and recreated whenever the version of a versioned Process Group
> changes. During that the service's state is lost.
> A stateless service is unaffected by that. Whereas for stateful service it
> causes a silent state loss: the replacement service is correctly configured
> and enabled, but it holds no state from the removed controller service.
> _Related jiras:_ NIFI-14995 and NIFI-16149 stopped these services being
> reported as local modifications when a flow is compared with its registered
> version. That handling does not extend to applying a version change.
> h2. Root cause
> {{StandardControllerServiceFactory.create}} does not assign a versioned
> component id to the service it creates.
> {{StandardVersionedComponentSynchronizer}} generates a new versioned id from
> the service's instance identifier for any controller services without a
> versioned id. Then it removes every local controller service whose version id
> is absent from the proposed snapshot. There is no exception for controller
> services created in {{{}migrateProperties{}}}.
> h2. Proposed change
> The case to support first: a Process Group is upgraded to a version that
> declares the Controller Service which was already created locally in
> {{{}migrateProperties{}}}. The service must survive with its current state,
> and the flow must end up with one controller service, not two.
> # Synchronization must not delete a Controller Service that was created in
> {{{}migrateProperties{}}}.
> # This local Controller Service must be set with a versioned component id
> assigned to that Controller Service in the versioned process group.
> The matching of Controller Services should go through the component that
> references them: if a component's Controller Service property points at a
> migration-created service now, and the same property in the new version
> points at a declared service of the same type in the same group, they must be
> treated as the same service.
> After matching the created Controller Service looks identical to its
> counterpart in the versioned process group, and it should carry no signs of
> being created in {{migrateProperties.}}
> h3. Other matching strategies
> Neither the identifier nor the versioned component id can be used to match
> these two services.
> Matching by class and property values work well only when controller service
> created by {{migrateProperties}} is identical to a service in the versioned
> group (see edge cases for properties drift). This matching strategy also
> won't work if there are 2 identical controller services in the versioned
> group.
> h2. Edge cases
> This solution creates a lot of edge cases, most of which are caused by
> Controller Service implementation/properties drift.
> Normally a versioned process group author must ensure there is no properties
> drift between a Controller Service created in {{migrateProperties}} and the
> Service from versioned process group.
> For each edge case I provide synchronization behavior.
> *New version does not declare the service.* Keep the local service,
> unversioned. Currently it is removed and then recreated empty by calling
> {{{}migrateProperties{}}}.
> *Declared service has different property values* {*}than the local
> service{*}{*}.{*} The local service becomes versioned, then new properties
> are applied to it.
> *A single declared service is present instead of several local services.* The
> first controller service becomes versioned, the rest are kept as local
> services.
> The following cases are related to Controller Services drift when versioned
> Controller Services don't match local Controller Services created in
> {{{}migrateProperties{}}}.
> If that happens, we create the declared services from the versioned group,
> which are used by the components. Local Controller Services are kept and not
> removed. The drift has to be resolved manually.
> # *New version declares a service of a different implementation type than
> the local service.*
> # *Declared service is in a different process group from the local service.*
> # *A single local service is referenced by several components.*
> # *Referencing component is absent from the new version, either deleted or
> itself a migration-created service with no versioned component id.*
> h2. Reproduction
> Reproduced against 2.12.0-SNAPSHOT on main.
> System tests:
> [https://github.com/awelless/nifi/tree/migration-created-cs-versioning-tests]
> {{MigrationCreatedControllerServiceVersioningIT}} uses a processor whose
> {{migrateProperties}} creates a stateful Controller SErvice.
> * {{testFlowUpgradePreservesMigrationCreatedControllerService}} - fails: the
> service is replaced by the one the new version declares;
> *
> {{testFlowUpgradeAddingUnrelatedProcessorPreservesMigrationCreatedControllerService}}
> - fails: the service is recreated under the same id with an empty store;
> * -{{testFlowDowngradePreservesMigrationCreatedControllerService}} - fails
> at its precondition, because the upgrade it performs first already removes
> the service, leaving the downgrade path unverified.- During downgrade we
> remove a Controller Service, as it's done for normal Controller Services.
> Two further tests pass, showing the NiFi upgrade and a plain NiFi restart are
> unaffected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
