Mark Payne created NIP-38:
-----------------------------
Summary: Pluggable Flow Persistence and Audit Storage
Key: NIP-38
URL: https://issues.apache.org/jira/browse/NIP-38
Project: NiFi Improvement Proposal
Issue Type: Improvement
Reporter: Mark Payne
h2. Motivation
Two parts of the NiFi framework are hardcoded to read and write local files,
with no way to redirect that storage elsewhere.
The first is the dataflow itself, which is always persisted to a local file as
GZIP-compressed JSON ({{{}flow.json.gz{}}}). Some deployments would benefit
from the ability to store the flow somewhere other than a local file, and there
is currently no extension point that lets the mechanism for loading and saving
the flow be replaced. This coupling also appears in components that only need
to read a small part of the flow. For example, {{FileAccessPolicyProvider}}
parses {{flow.json.gz}} directly to obtain the root process group identifier
when seeding initial access policies.
The second is the flow-change history (the audit trail of who changed what in
the flow), which is written to an embedded database under
{{{}nifi.database.directory{}}}. This history is stored by the
{{{}AuditService{}}}. An {{AuditService}} interface already exists, but the
framework constructs the {{EntityStoreAuditService}} implementation directly,
so there is no way to store this history anywhere other than the local embedded
database.
This NIP makes both storage mechanisms pluggable, using the same extension
pattern for each.
h2. Proposed Solution
h3. Flow Persistence Provider
Introduce a {{FlowPersistenceProvider}} extension point in
{{{}nifi-framework-api{}}}. A single provider is selected per instance and is
responsible for loading and saving the dataflow. The provider should operate on
{{{}VersionedDataflow{}}}; the encoding of the flow and the mechanism used to
store it are entirely the provider's concern.
{code:java}
/**
* Persists and restores the dataflow for a NiFi instance. A single provider is
selected per
* instance and is responsible for durably storing the flow so that it can be
recovered when
* NiFi restarts. The encoding of the flow and the mechanism used to store it
are entirely the
* provider's concern.
*/
public interface FlowPersistenceProvider {
/**
* Called once after instantiation to provide the provider with its
configuration.
*/
void initialize(FlowPersistenceProviderInitializationContext
initializationContext) throws IOException;
/**
* Loads the currently persisted flow, or an empty Optional when no flow
has been persisted.
*/
Optional<VersionedDataflow> loadFlow() throws IOException;
/**
* Persists the flow described by the given context, replacing any
previously persisted flow.
*/
void saveFlow(FlowPersistenceContext context) throws IOException;
}
{code}
{{saveFlow}} accepts a {{FlowPersistenceContext}} rather than a bare
{{VersionedDataflow}} so that additional detail about the save — for example a
human-readable description of what changed — can be added later without
breaking existing providers.
{{VersionedDataflow}} and {{VersionedFlowEncodingVersion}} move from
{{nifi-framework-core-api}} into {{nifi-framework-api}} so the extension point
can expose a typed API.
A default {{StandardFlowPersistenceProvider}} in {{nifi-framework-core}}
persists the flow as GZIP-compressed JSON to the configured flow configuration
file.
h3. Audit Service
Make the {{AuditService}} a selectable extension point. The interface moves
from {{nifi-administration}} into {{nifi-framework-api}} so that
implementations can be delivered in NARs, and it gains a default {{initialize}}
method that receives the implementation's configuration.
{code:java}
/**
* Called once after instantiation to provide the Audit Service with its
configuration. The
* default implementation does nothing so that existing implementations
continue to work
* unchanged.
*/
default void initialize(AuditServiceInitializationContext
initializationContext) throws IOException {
}
{code}
The implementation to use is selected in {{nifi.properties}} and defaults to
{{{}EntityStoreAuditService{}}}, which stores the flow-change history in the
existing embedded database. The directory the default implementation uses is
supplied through its initialization context, so the storage location is
configuration rather than a hardcoded file reference.
h2. Framework Changes
* New {{FlowPersistenceProvider}} extension point and supporting context types
in {{{}nifi-framework-api{}}}.
* {{VersionedDataflow}} and {{VersionedFlowEncodingVersion}} move from
{{nifi-framework-core-api}} to {{{}nifi-framework-api{}}}.
* Default {{StandardFlowPersistenceProvider}} plus its extension-discovery and
dependency-injection wiring in {{{}nifi-framework-core{}}}.
* Every point at which the framework saves the flow, and every point at which
it restores the flow, goes through the provider.
* {{FileAccessPolicyProvider}} obtains the root process group identifier from
the provider rather than reading {{flow.json.gz}} directly.
* The previous {{FlowConfigurationDAO}} / {{StandardFlowConfigurationDAO}} /
{{FlowParser}} / {{FlowInfo}} code path is removed.
* {{AuditService}} (and its supporting types {{{}History{}}},
{{{}HistoryQuery{}}}, {{{}PreviousValue{}}}) move from {{nifi-administration}}
to {{{}nifi-framework-api{}}}, and the interface gains a default {{initialize}}
method plus an {{{}AuditServiceInitializationContext{}}}.
* The {{AuditService}} implementation is created through extension-discovery
and dependency-injection wiring in {{nifi-framework-core}} instead of being
constructed directly.
* {{EntityStoreAuditService}} reads its storage directory from its
initialization context rather than from a hardcoded location.
h2. nifi.properties Configuration
* {{nifi.flow.persistence.provider.implementation}} — fully qualified class
name of the {{FlowPersistenceProvider}} to use. Default:
{{{}org.apache.nifi.controller.flow.StandardFlowPersistenceProvider{}}}.
* {{nifi.flow.persistence.provider.properties.<name>}} — provider-specific
configuration, passed to the provider through its initialization context.
* {{nifi.audit.service.implementation}} — fully qualified class name of the
{{AuditService}} to use. Default:
{{{}org.apache.nifi.admin.service.EntityStoreAuditService{}}}.
* {{nifi.audit.service.properties.<name>}} — implementation-specific
configuration, passed to the Audit Service through its initialization context.
The existing {{{}nifi.flow.configuration.file{}}},
{{{}nifi.flow.configuration.archive.*{}}}, and {{nifi.database.directory}}
properties are honored as fallbacks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
