[
https://issues.apache.org/jira/browse/NIP-29?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIP-29:
--------------------------
Description:
h1. Motivation
Since the introduction of Connectors (NIP-11) we’re already seeing a common
pattern emerge: existing versioned flows that are now being packaged as
Connectors.
Prior to Connectors, when a user has a flow that they wanted to deploy in
multiple scenarios, the recommendation was to save the flow as a Versioned Flow
using a Registry Client (whether that be NiFi Registry, Github, etc.) Now that
we’ve introduce Connectors, these provide a more powerful construct for many
users. But flows that have been running for some time have already amassed
state, such as which files have been listed from S3, etc. There’s no real way
to migrate existing state from a versioned flow to a new Connector. The only
option would be start ingest all over. For a source like S3, this could equate
to terabytes of data that would have to be ingested again. Additionally,
there’s significant overhead in reconfiguring the Connector to work in the same
way as the versioned flow.
h1. Proposed Solution
The proposal is to introduce two new methods to the Connector interface:
{code:java}
public default boolean isMigrationAllowed(VersionedExternalFlow versionedFlow) {
return false;
}
public default void migrate(VersionedExternalFlow versionedFlow,
ConnectorAssetFactory assetFactory) {
throw new UnsupportedOperationException("This Connector does not support
migrating configuration and state from the provided flow.");
}
{code}
Because the VersionedExternalFlow may have references to Assets, we should make
that seamless for users as well by introducing a new ConnectorAssetFactory to
ConnectorInitializationContext:
{code:java}
public interface ConnectorInitializationContext {
public ConnectorAssetFactory getAssetFactory();
}
public interface ConnectorAssetFactory {
/**
* Checks if the given VersionedAsset is locally available. If it is not,
returns an empty Optional. If the asset
* is available, creates a duplicate of the Asset that is accessible by the
Connector and returns
* an Optional that wraps a reference to that Asset so that the Connector can
be configured
* to reference it.
*/
public Optional<AssetReference> createAsset(VersionedAsset versionedAsset);
}
{code}
This allows a Connector to be created and then indicate whether or not its
configuration and assets can be migrated from a given Versioned Flow. By
default, connectors will not support migration from any versioned flows, but
they can do so by overriding these two methods.
If migrating a versioned for to a Connector on the same NiFi instance, those
Assets that are referenced by the versioned flow can automatically be copied
over for use in the Connector as well.
This means that for any Connector developer, we have the ability to make
migration from a Versioned Flow to the Connector pretty painless and seamless
for users. Secrets will still require configuring the Connector because Secrets
come from Secret Manager instead of local parameters, but everything else
should be able to be migrated very smoothly.
h1. REST API
In order for this to work, we will need a few additional REST APIs to be added
as well. Namely, we will need an endpoint for triggering the migration from a
Versioned Flow. Ideally, we will also have an endpoint that returns to us all
of the Versioned Flows on the canvas that are compatible for migration with a
given Connector as well. This will make it easy to offer in the UI the ability
to just say choose a Versioned Flow from the list instead of having to export
the flow as a Versioned Flow, and then go and upload that Versioned Flow to the
Connector.
Note that these APIs are conceptual and alone may not be fully sufficient for
the implementation. The actual API changes will, of course, go through the
normal RTC process.
was:
h1. Motivation
Since the introduction of Connectors (NIP-11) we’re already seeing a common
pattern emerge: existing versioned flows that are now being packaged as
Connectors.
Prior to Connectors, when a user has a flow that they wanted to deploy in
multiple scenarios, the recommendation was to save the flow as a Versioned Flow
using a Registry Client (whether that be NiFi Registry, Github, etc.) Now that
we’ve introduce Connectors, these provide a more powerful construct for many
users. But flows that have been running for some time have already amassed
state, such as which files have been listed from S3, etc. There’s no real way
to migrate existing state from a versioned flow to a new Connector. The only
option would be start ingest all over. For a source like S3, this could equate
to terabytes of data that would have to be ingested again. Additionally,
there’s significant overhead in reconfiguring the Connector to work in the same
way as the versioned flow.
h1. Proposed Solution
The proposal is to introduce two new methods to the Connector interface:
{code}
public default boolean isMigrationAllowed(VersionedExternalFlow versionedFlow) {
return false;
}
public default void migrate(VersionedExternalFlow versionedFlow,
ConnectorAssetFactory assetFactory) {
throw new UnsupportedOperationException("This Connector does not support
migrating configuration and state from the provided flow.");
}
{code}
Because the VersionedExternalFlow may have references to Assets, we should make
that seamless for users as well by introducing a new ConnectorAssetFactory to
ConnectorInitializationContext:
{code}
public interface ConnectorInitializationContext {
public ConnectorAssetFactory getAssetFactory();
}
public interface ConnectorAssetFactory {
/**
* Checks if the given VersionedAsset is locally available. If it is not,
returns an empty Optional. If the asset
* is available, creates a duplicate of the Asset that is accessible by the
Connector and returns
* an Optional that wraps a reference to that Asset so that the Connector can
be configured
* to reference it.
*/
public Optional<AssetReference> createAsset(VersionedAsset versionedAsset);
}
{code}
This allows a Connector to be created and then indicate whether or not its
configuration and assets can be migrated from a given Versioned Flow. By
default, connectors will not support migration from any versioned flows, but
they can do so by overriding these two methods.
If migrating a versioned for to a Connector on the same NiFi instance, those
Assets that are referenced by the versioned flow can automatically be copied
over for use in the Connector as well.
This means that for any Connector developer, we have the ability to make
migration from a Versioned Flow to the Connector pretty painless and seamless
for users. Secrets will still require configuring the Connector because Secrets
come from Secret Manager instead of local parameters, but everything else
should be able to be migrated very smoothly.
h1. REST API
In order for this to work, we will need a few additional REST APIs to be added
as well. Namely, we will need an endpoint for triggering the migration from a
Versioned Flow. Ideally, we will also have an endpoint that returns to us all
of the Versioned Flows on the canvas that are compatible for migration with a
given Connector as well. This will make it easy to offer in the UI the ability
to just say choose a Versioned Flow from the list instead of having to export
the flow as a Versioned Flow, and then go and upload that Versioned Flow to the
Connector.
> Versioned Flow to Connector Migration
> -------------------------------------
>
> Key: NIP-29
> URL: https://issues.apache.org/jira/browse/NIP-29
> Project: NiFi Improvement Proposal
> Issue Type: Improvement
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
>
> h1. Motivation
> Since the introduction of Connectors (NIP-11) we’re already seeing a common
> pattern emerge: existing versioned flows that are now being packaged as
> Connectors.
> Prior to Connectors, when a user has a flow that they wanted to deploy in
> multiple scenarios, the recommendation was to save the flow as a Versioned
> Flow using a Registry Client (whether that be NiFi Registry, Github, etc.)
> Now that we’ve introduce Connectors, these provide a more powerful construct
> for many users. But flows that have been running for some time have already
> amassed state, such as which files have been listed from S3, etc. There’s no
> real way to migrate existing state from a versioned flow to a new Connector.
> The only option would be start ingest all over. For a source like S3, this
> could equate to terabytes of data that would have to be ingested again.
> Additionally, there’s significant overhead in reconfiguring the Connector to
> work in the same way as the versioned flow.
> h1. Proposed Solution
> The proposal is to introduce two new methods to the Connector interface:
> {code:java}
> public default boolean isMigrationAllowed(VersionedExternalFlow
> versionedFlow) {
> return false;
> }
> public default void migrate(VersionedExternalFlow versionedFlow,
> ConnectorAssetFactory assetFactory) {
> throw new UnsupportedOperationException("This Connector does not support
> migrating configuration and state from the provided flow.");
> }
> {code}
> Because the VersionedExternalFlow may have references to Assets, we should
> make that seamless for users as well by introducing a new
> ConnectorAssetFactory to ConnectorInitializationContext:
> {code:java}
> public interface ConnectorInitializationContext {
> public ConnectorAssetFactory getAssetFactory();
> }
> public interface ConnectorAssetFactory {
> /**
> * Checks if the given VersionedAsset is locally available. If it is not,
> returns an empty Optional. If the asset
> * is available, creates a duplicate of the Asset that is accessible by the
> Connector and returns
> * an Optional that wraps a reference to that Asset so that the Connector
> can be configured
> * to reference it.
> */
> public Optional<AssetReference> createAsset(VersionedAsset versionedAsset);
> }
> {code}
> This allows a Connector to be created and then indicate whether or not its
> configuration and assets can be migrated from a given Versioned Flow. By
> default, connectors will not support migration from any versioned flows, but
> they can do so by overriding these two methods.
> If migrating a versioned for to a Connector on the same NiFi instance, those
> Assets that are referenced by the versioned flow can automatically be copied
> over for use in the Connector as well.
> This means that for any Connector developer, we have the ability to make
> migration from a Versioned Flow to the Connector pretty painless and seamless
> for users. Secrets will still require configuring the Connector because
> Secrets come from Secret Manager instead of local parameters, but everything
> else should be able to be migrated very smoothly.
> h1. REST API
> In order for this to work, we will need a few additional REST APIs to be
> added as well. Namely, we will need an endpoint for triggering the migration
> from a Versioned Flow. Ideally, we will also have an endpoint that returns to
> us all of the Versioned Flows on the canvas that are compatible for migration
> with a given Connector as well. This will make it easy to offer in the UI the
> ability to just say choose a Versioned Flow from the list instead of having
> to export the flow as a Versioned Flow, and then go and upload that Versioned
> Flow to the Connector.
> Note that these APIs are conceptual and alone may not be fully sufficient for
> the implementation. The actual API changes will, of course, go through the
> normal RTC process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
