Eric Secules created NIFI-8603: ---------------------------------- Summary: Use non-heap collections in StandardProcessSession Key: NIFI-8603 URL: https://issues.apache.org/jira/browse/NIFI-8603 Project: Apache NiFi Issue Type: New Feature Components: Core Framework Affects Versions: 1.13.2 Reporter: Eric Secules
*Use Case:* A processor is splitting one flowfile into thousands or millions of flowfiles. It's preferred to keep all those flowfiles in the process session in case there is some error and we need to roll back. However, the process session stores the FlowFiles in Maps on the heap while the session is waiting to commit. One workaround is to commit the session early while processing splits and require the user to use wait/notify to hold the split portions until the whole input file is successfully split. This takes advantage of the fact that NiFi queues are swapped to disk after they reach a certain size. It would be great if the process session had a similar behaviour to NiFi queues between processors so that it is able to hold a large amount of flowfiles in the session without consuming the whole heap. *Acceptance Criteria:* * NiFi StandardProcessSession can handle arbitrarily large transactions by holding flowfiles off-heap or partially-off heap -- This message was sent by Atlassian Jira (v8.3.4#803005)