Mark Payne created NIP-44:
-----------------------------
Summary: Allow Stateless groups to keep FlowFile content in memory
Key: NIP-44
URL: https://issues.apache.org/jira/browse/NIP-44
Project: NiFi Improvement Proposal
Issue Type: New Feature
Reporter: Mark Payne
h1. Background
When Stateless NiFi is used directly via cli or embedded, it is typically run
with an in-memory (on-heap) Content Repository. This makes a lot of sense
because it's much more performance than writing to disk and it generally 'owns'
the full JVM. No risk of using a lot of heap and causing completely separate
flows to be memory constrained.
However, when we introduced Stateless Process Groups in 'NiFi proper' we did
not set it up this way. We set it up to write content directly to the
configured Content Repository, even though the content claim will have no
references and will be destroyed upon restart of NiFi. We did this out of
concerns for stability. If you have a single Stateless group that buffers a
couple GB of FlowFile content into heap on a fairly heap-constrained JVM you
can cause OOME, etc. that effects all flows.
I believe writing to the Content Repository directly is the correct default.
However, there are cases where keeping the FlowFile content in memory and
on-heap still makes sense even for a Stateless Process Group. Two cases in
particular:
# The flow is designed such that it only ever keeps small a small amount of
data in-flight at a given time. If it never will buffer more than 5-10 MB of
data it's still quite safe.
# Environments where the only thing running is a Stateless Group. With the
prevalence of running NiFi on Kubernetes now, we see many users create a lot of
tiny NiFi instances instead of a few huge instances. In this paradigm, it's far
more common to have a flow that only runs a Stateless Group or where the
Stateless Group is the majority of the processing logic.
In order to accommodate these cases, we should allow buffering the FlowFile
content in memory/heap.
h1. Proposal
My proposal, then, is to allow a Process Group, when its Execution Engine is
configured for Stateless, to also be able to configure whether FlowFile content
is written to the Content Repository or buffered in memory.
This would result in a new configuration element in the UI, and the default
will write to the Content Repository, but it would become configurable.
An important caveat is that we should NOT allow it to be configured differently
from one Process Group to its child. This is similar to how we forbid Group A
from having an Execution Engine of Stateless while its child group has an
Execution Engine of Standard.
h1. API Changes
This will require changes to typical REST endpoint / flow configuration
objects: ProcessGroupDTO, VersionedProcessGroup, ProcessGroup, endpoint
mergers, etc. in order to provide the plumbing.
h1. Framework Changes
This will also require updates to the framework for this to be implemented, but
the Content Repository implementation already exists. It may require moving it
into a different maven module that can be more readily shared, but I don't
expect this to require significant machinery otherwise.
There will also be necessary updates to the flow inheritance modules to ensure
that we apply updates appropriately.
h1. Documentation Changes
We will need to update the user guide to explain this new capability and the
tradeoffs of in-memory vs. Content Repository.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
