[
https://issues.apache.org/jira/browse/NIP-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18065184#comment-18065184
]
Pierre Villard commented on NIP-4:
----------------------------------
I want to get back to this work. Based on the provided feedback above, the
initial approach won't be providing the Extensions Registry Clients as a first
class citizen component type with instantiation and configuration via the NiFi
UI like we have with the Flow Registry Clients today. Instead the first
implementation will only support a nifi.properties based configuration. Then,
based on usage and feedback this could be later evolved into a full
implementation where extension registry clients could be configured directly
into controller settings via a dedicated tab in the UI. Having NiFi bringing
NARs from external locations is something that should be only available to
admins and this should be done with proper guardrails. This is also the reason
why I have submitted NIFI-15675 to have NAR signing and then expose the
configuration to only accept properly signed NARs into NiFi.
> Extension Registry Client
> -------------------------
>
> Key: NIP-4
> URL: https://issues.apache.org/jira/browse/NIP-4
> Project: NiFi Improvement Proposal
> Issue Type: New Feature
> Reporter: Pierre Villard
> Priority: Major
>
> *Motivation and description*
> The goal of this NIP is to describe the addition of a new extension point:
> extension registry client. The goal of an extension registry client is to
> have a "pull" mechanism for NiFi to pull extensions (NAR files) into NiFi so
> that the components can be used in NiFi for designing and running flows.
> Main features :
> * the extension registry clients would check, at a specified frequency, for
> available NARs in external locations. The list of available components would
> be retrieved by looking at the manifest files exposed by the external
> locations.
> * when listing components in NiFi (processors, controller services, etc),
> the list would be a merge of what is presently available on the NiFi nodes as
> well as what is theoretically available through the extension registry
> clients. If a user starts using a component that is not locally available,
> NiFi would download the required NAR(s) to have the component in NiFi. This
> would work similarly to how it works with Python components where things are
> loaded in the background. If things cannot be loaded, the component would be
> ghosted.
> * when importing a flow definition (flow definition upload, flow registry
> clients, etc), the same behaviour would apply for components that are defined
> in the flow: if the instantiated flow references a bundle that is not locally
> available but available remotely, we would download the required NARs
> Extension registry clients will be defined at controller level similarly to
> what is available today with registry clients, reporting tasks, parameter
> providers, etc. The same level of permissions will be required and it is
> expected that only NiFi admin would configure such a client against trusted
> external locations.
> *Advantages*
> * by doing this, it would be possible to ship a NiFi runtime with a minimal
> size and only download the NARs that are required for running the flow. That
> would make auto scaling more efficient and considerably reduce the container
> image size.
> * this would provide a great option to raise awareness about components that
> are maintained by the community but not made available in the convenience
> binary.
> * this would provide a nice opportunity for third parties to share specific
> components with the community while not pushing the responsibility of
> maintaining those on the community.
> *Scope*
> A large number of changes are expected but nothing that would not be backward
> compatible. This NIP will likely need to be updated for exhaustivity while
> implementing the overall capability. It is hard to anticipate the full extent
> of the changes before actual POC implementation.
> In nifi-api, the expected changes are:
> * new classes around ExtensionRegistryClient
> * additional ComponentType
> * the Bundle object will have a boolean isRemote to determine if a bundle is
> locally available or remotely available for NiFi
> The new APIs for Extension Registry Clients would be:
> * InputStream getNARFile(ExtensionRegistryClientConfigurationContext
> context, Bundle bundle) throws ExtensionRegistryException, IOException;
> * InputStream getManifestFile(ExtensionRegistryClientConfigurationContext
> context, Bundle bundle) throws ExtensionRegistryException, IOException;
> * Set<Bundle> listBundles(ExtensionRegistryClientConfigurationContext
> context) throws ExtensionRegistryException, IOException;
> As part of this change, we would mark as deprecated the existing
> ExternalResourceProvider concept and all of its implementations. All of this
> would be removed in NiFi 3.
> *Not in Scope*
> * Support for NAR including Python components is not in scope at this stage.
> This will require some discussions regarding the extensions manifest for such
> components.
> * For the initial version, we do not support the case where NiFi would be
> initially provisioned with a flow.json.gz containing extension registry
> clients as well as components in the flow definition that should be fetched
> from remote locations. Support can later be added by this case by initially
> ghosting the components and after the initial startup of NiFi, check the
> ghost components and trigger the download of remote extensions where
> applicable.
> *Verification*
> Implementing a couple of extension providers will help verifying the proper
> implementation of the feature. In particular some interesting options that
> are considered:
> * Local file system implementation
> * Nexus-like implementation
> * AWS S3 implementation
> Anything that is not Nexus-like with well defined APIs will need to adhere to
> a specific convention allowing for discoverability of the components as well
> as retrieval of the manifest files.
> In order to provide proper behaviour, it is assumed and required for the
> external stores to have the manifest files published along the NAR files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
