[
https://issues.apache.org/jira/browse/NIP-29?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne reassigned NIP-29:
-----------------------------
Assignee: Mark Payne
> Versioned Flow to Connector Migration
> -------------------------------------
>
> Key: NIP-29
> URL: https://issues.apache.org/jira/browse/NIP-29
> Project: NiFi Improvement Proposal
> Issue Type: Improvement
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
>
> h1. Motivation
> Since the introduction of Connectors (NIP-11) we’re already seeing a common
> pattern emerge: existing versioned flows that are now being packaged as
> Connectors.
> Prior to Connectors, when a user has a flow that they wanted to deploy in
> multiple scenarios, the recommendation was to save the flow as a Versioned
> Flow using a Registry Client (whether that be NiFi Registry, Github, etc.)
> Now that we’ve introduce Connectors, these provide a more powerful construct
> for many users. But flows that have been running for some time have already
> amassed state, such as which files have been listed from S3, etc. There’s no
> real way to migrate existing state from a versioned flow to a new Connector.
> The only option would be start ingest all over. For a source like S3, this
> could equate to terabytes of data that would have to be ingested again.
> Additionally, there’s significant overhead in reconfiguring the Connector to
> work in the same way as the versioned flow.
> h1. Proposed Solution
> The proposal is to introduce two new methods to the Connector interface:
> {code}
> public default boolean isMigrationAllowed(VersionedExternalFlow
> versionedFlow) {
> return false;
> }
> public default void migrate(VersionedExternalFlow versionedFlow,
> ConnectorAssetFactory assetFactory) {
> throw new UnsupportedOperationException("This Connector does not support
> migrating configuration and state from the provided flow.");
> }
> {code}
> Because the VersionedExternalFlow may have references to Assets, we should
> make that seamless for users as well by introducing a new
> ConnectorAssetFactory to ConnectorInitializationContext:
> {code}
> public interface ConnectorInitializationContext {
> public ConnectorAssetFactory getAssetFactory();
> }
> public interface ConnectorAssetFactory {
> /**
> * Checks if the given VersionedAsset is locally available. If it is not,
> returns an empty Optional. If the asset
> * is available, creates a duplicate of the Asset that is accessible by the
> Connector and returns
> * an Optional that wraps a reference to that Asset so that the Connector
> can be configured
> * to reference it.
> */
> public Optional<AssetReference> createAsset(VersionedAsset versionedAsset);
> }
> {code}
> This allows a Connector to be created and then indicate whether or not its
> configuration and assets can be migrated from a given Versioned Flow. By
> default, connectors will not support migration from any versioned flows, but
> they can do so by overriding these two methods.
> If migrating a versioned for to a Connector on the same NiFi instance, those
> Assets that are referenced by the versioned flow can automatically be copied
> over for use in the Connector as well.
> This means that for any Connector developer, we have the ability to make
> migration from a Versioned Flow to the Connector pretty painless and seamless
> for users. Secrets will still require configuring the Connector because
> Secrets come from Secret Manager instead of local parameters, but everything
> else should be able to be migrated very smoothly.
> h1. REST API
> In order for this to work, we will need a few additional REST APIs to be
> added as well. Namely, we will need an endpoint for triggering the migration
> from a Versioned Flow. Ideally, we will also have an endpoint that returns to
> us all of the Versioned Flows on the canvas that are compatible for migration
> with a given Connector as well. This will make it easy to offer in the UI the
> ability to just say choose a Versioned Flow from the list instead of having
> to export the flow as a Versioned Flow, and then go and upload that Versioned
> Flow to the Connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
