Mark Payne created NIP-29:
-----------------------------
Summary: Versioned Flow to Connector Migration
Key: NIP-29
URL: https://issues.apache.org/jira/browse/NIP-29
Project: NiFi Improvement Proposal
Issue Type: Improvement
Reporter: Mark Payne
h1. Motivation
Since the introduction of Connectors (NIP-11) we’re already seeing a common
pattern emerge: existing versioned flows that are now being packaged as
Connectors.
Prior to Connectors, when a user has a flow that they wanted to deploy in
multiple scenarios, the recommendation was to save the flow as a Versioned Flow
using a Registry Client (whether that be NiFi Registry, Github, etc.) Now that
we’ve introduce Connectors, these provide a more powerful construct for many
users. But flows that have been running for some time have already amassed
state, such as which files have been listed from S3, etc. There’s no real way
to migrate existing state from a versioned flow to a new Connector. The only
option would be start ingest all over. For a source like S3, this could equate
to terabytes of data that would have to be ingested again. Additionally,
there’s significant overhead in reconfiguring the Connector to work in the same
way as the versioned flow.
h1. Proposed Solution
The proposal is to introduce two new methods to the Connector interface:
{{public default boolean isMigrationAllowed(VersionedExternalFlow
versionedFlow) {}}
{{ return false;}}
{{}}}
{{public default void migrate(VersionedExternalFlow versionedFlow,
ConnectorAssetFactory assetFactory) {}}
{{ throw new UnsupportedOperationException("This Connector does not support
migrating configuration and state from the provided flow.");}}
{{}}}
{{ }}
Because the VersionedExternalFlow may have references to Assets, we should make
that seamless for users as well by introducing a new ConnectorAssetFactory to
ConnectorInitializationContext:
{{{}public interface ConnectorInitializationContext{}}}{{{}{{}}}
{{ public ConnectorAssetFactory getAssetFactory();}}
{{ }}}
{{}}
{{{}public interface ConnectorAssetFactory{}}}{{{}{{}}}
{{ /**}}
{{ * Checks if the given VersionedAsset is locally available. If it is not,
returns an empty Optional. If the asset}}
{{ * is available, creates a duplicate of the Asset that is accessible by
the Connector and returns}}
{{ * an Optional that wraps a reference to that Asset so that the Connector
can be configured}}
{{ * to reference it.}}
{{ */}}
{{ public Optional<AssetReference> createAsset(VersionedAsset
versionedAsset);}}
{{ }}}
This allows a Connector to be created and then indicate whether or not its
configuration and assets can be migrated from a given Versioned Flow. By
default, connectors will not support migration from any versioned flows, but
they can do so by overriding these two methods.
If migrating a versioned for to a Connector on the same NiFi instance, those
Assets that are referenced by the versioned flow can automatically be copied
over for use in the Connector as well.
This means that for any Connector developer, we have the ability to make
migration from a Versioned Flow to the Connector pretty painless and seamless
for users. Secrets will still require configuring the Connector because Secrets
come from Secret Manager instead of local parameters, but everything else
should be able to be migrated very smoothly.
h1. REST API
In order for this to work, we will need a few additional REST APIs to be added
as well. Namely, we will need an endpoint for triggering the migration from a
Versioned Flow. Ideally, we will also have an endpoint that returns to us all
of the Versioned Flows on the canvas that are compatible for migration with a
given Connector as well. This will make it easy to offer in the UI the ability
to just say choose a Versioned Flow from the list instead of having to export
the flow as a Versioned Flow, and then go and upload that Versioned Flow to the
Connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
