Mark Payne created NIP-51:
-----------------------------
Summary: Allow Draining of a Process Group
Key: NIP-51
URL: https://issues.apache.org/jira/browse/NIP-51
Project: NiFi Improvement Proposal
Issue Type: Improvement
Reporter: Mark Payne
h2. Motivation
There are several times when a user needs all of the data that is already
queued in a Process Group to finish processing, without letting any new data in:
* *Moving a flow to another NiFi instance.* NiFi can now export a flow along
with the state of its components and import it elsewhere with that state
(NIFI-15671). Queued FlowFiles are not part of the export, so the queues need
to be empty first.
* *Upgrading a Versioned Flow.* A new version may change or remove the
components that hold queued data. Emptying the group first means that the data
is never processed by a flow that it was not built for.
* *Suspending a flow for a period of time.* Letting queued data finish first
means it is not left waiting until the flow is started again.
Today the user has to find and stop every component that brings data into the
group, make sure everything else is running, and then watch the queues until
they are empty. Furthermore, it can take a long time for some processors such
as MergeContent to complete because we have to wait for the Max Bin Age, etc.
Recent additions already allow stopping all source components. This is not
intended to replace that but to complement it, as there are reasons for
stopping the source components other than necessarily a full drain.
NiFi already supports draining a Connector. This NIP proposes the same ability
for any Process Group on the canvas.
h2. Proposed Solution
A user can choose "Drain" for a Process Group. NiFi then:
* Stops the source Processors (Processors with no incoming connections) in the
group and all of its child groups, and the Input Ports of the drained group
itself.
* Starts all other Processors, Output Ports and Remote Process Groups, and
enables the Controller Services that they reference.
* Reports progress as the number and size of FlowFiles still queued, compared
to when the drain began.
* Lists the components that keep the drain from finishing because they are
invalid or disabled and have data queued.
The user can also cancel a drain at any time. While a drain is in progress,
users cannot start the source components or stop the other components in the
group. Drains are not remembered across a restart.
Some Processors may take a very long time to pass FlowFiles along, or may never
do so, once new data stops arriving. MergeContent and MergeRecord configured
without a Max Bin Age are good examples. NiFi will notify Processors when a
drain begins and ends, so that they can process their data more aggressively
during a drain.
h2. NiFi API Changes
Two new lifecycle annotations, {{@OnDrainInitiated}} and
{{{}@OnDrainCompleted{}}}, for Processors and Controller Services that need to
behave differently while their Process Group is being drained. Methods may take
no arguments, a {{ProcessContext}} (Processors), or a {{ConfigurationContext}}
(Controller Services). This change is purely additive. Components that do not
use the annotations are not affected.
{{TestRunner}} in nifi-mock gains {{initiateDrain()}} and {{completeDrain()}}
so that extension developers can test this behavior.
h2. REST API Changes
New asynchronous endpoints will be added for creating, fetching, and deleting a
drian.
Beginning a drain requires WRITE permission on the Process Group and on
everything in it. In a cluster, all nodes must be connected.
{{ProcessGroupDTO}} gains a read-only {{drainRequestId}} field.
h2. UI Changes
The Process Group context menu gains "Drain". A dialog shows a progress bar and
a Cancel button. It also lists any blocking components with the reason they are
blocking.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
