[
https://issues.apache.org/jira/browse/NIP-44?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109942#comment-18109942
]
Mark Payne edited comment on NIP-44 at 8/31/26 4:04 PM:
--------------------------------------------------------
Vote passed; NIP ratified
Vote thread: https://lists.apache.org/thread/3skm11z4ybrnoglbo93kkkfrrfhbp8cc
was (Author: markap14):
Vote passed; NIP ratified
> Allow Stateless groups to keep FlowFile content in memory
> ---------------------------------------------------------
>
> Key: NIP-44
> URL: https://issues.apache.org/jira/browse/NIP-44
> Project: NiFi Improvement Proposal
> Issue Type: New Feature
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
>
> h1. Background
> When Stateless NiFi is used directly via cli or embedded, it is typically run
> with an in-memory (on-heap) Content Repository. This makes a lot of sense
> because it's much more performance than writing to disk and it generally
> 'owns' the full JVM. No risk of using a lot of heap and causing completely
> separate flows to be memory constrained.
> However, when we introduced Stateless Process Groups in 'NiFi proper' we did
> not set it up this way. We set it up to write content directly to the
> configured Content Repository, even though the content claim will have no
> references and will be destroyed upon restart of NiFi. We did this out of
> concerns for stability. If you have a single Stateless group that buffers a
> couple GB of FlowFile content into heap on a fairly heap-constrained JVM you
> can cause OOME, etc. that effects all flows.
> I believe writing to the Content Repository directly is the correct default.
> However, there are cases where keeping the FlowFile content in memory and
> on-heap still makes sense even for a Stateless Process Group. Two cases in
> particular:
> # The flow is designed such that it only ever keeps small a small amount of
> data in-flight at a given time. If it never will buffer more than 5-10 MB of
> data it's still quite safe.
> # Environments where the only thing running is a Stateless Group. With the
> prevalence of running NiFi on Kubernetes now, we see many users create a lot
> of tiny NiFi instances instead of a few huge instances. In this paradigm,
> it's far more common to have a flow that only runs a Stateless Group or where
> the Stateless Group is the majority of the processing logic.
> In order to accommodate these cases, we should allow buffering the FlowFile
> content in memory/heap.
> h1. Proposal
> My proposal, then, is to allow a Process Group, when its Execution Engine is
> configured for Stateless, to also be able to configure whether FlowFile
> content is written to the Content Repository or buffered in memory.
> This would result in a new configuration element in the UI, and the default
> will write to the Content Repository, but it would become configurable.
> An important caveat is that we should NOT allow it to be configured
> differently from one Process Group to its child. This is similar to how we
> forbid Group A from having an Execution Engine of Stateless while its child
> group has an Execution Engine of Standard.
> h1. API Changes
> This will require changes to typical REST endpoint / flow configuration
> objects: ProcessGroupDTO, VersionedProcessGroup, ProcessGroup, endpoint
> mergers, etc. in order to provide the plumbing.
> h1. Framework Changes
> This will also require updates to the framework for this to be implemented,
> but the Content Repository implementation already exists. It may require
> moving it into a different maven module that can be more readily shared, but
> I don't expect this to require significant machinery otherwise.
> There will also be necessary updates to the flow inheritance modules to
> ensure that we apply updates appropriately.
> h1. Documentation Changes
> We will need to update the user guide to explain this new capability and the
> tradeoffs of in-memory vs. Content Repository.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
