Mark Payne created NIFI-16200:
---------------------------------
Summary: Improve resilience against large/numerous FlowFile
attributes
Key: NIFI-16200
URL: https://issues.apache.org/jira/browse/NIFI-16200
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
h2. Motivation
FlowFile attributes are intended to be small metadata (typically on the order
of hundreds of characters). In practice, flows often place large payloads in
attributes. Attributes are held in memory on each FlowFile and all attributes
are rewritten to the FlowFile Repository on every update that includes any
attribute change. Large attributes therefore increase write cost, inflate the
FlowFile Repository write-ahead log (WAL) between checkpoints, and can exhaust
JVM heap before today's count-based swap thresholds engage.
Today, swap is driven by a globally configured FlowFile count
({{{}nifi.queue.swap.threshold{}}}, default 20,000) with a fixed batch size of
10,000 FlowFiles written per swap file.
FlowFile Repository checkpointing is time-based (default 20 seconds). Under
heavy attribute write load, the active journal can grow large enough to exhaust
disk before the next scheduled checkpoint.
As a result, a huge number of attributes, or very large attributes, cause
issues with both NiFi's JVM heap as well as disk contention contention and
storage space exhaustion.
h2. Proposed Solution
Two related changes:
# *Size-triggered FlowFile Repository checkpointing.* Keep the existing
time-based checkpoint interval. Add a configurable maximum size for the active
WAL journal. When that size is reached, force a checkpoint instead of waiting
for the interval, then begin a new journal and discard the prior
snapshot/journals as today.
# *Per-queue, heap-size-based swapping.* Replace globally configured
count-based swap thresholds with a per-connection budget for estimated in-heap
FlowFile metadata (record overhead plus attribute key/value lengths). Default
global budget: 10 MB of metadata in the active queue before overflow begins.
Overflow uses the existing secondary in-heap queue; when that queue holds about
1 MB of estimated metadata, write a swap file and drop those FlowFiles from
heap. Each connection may override the budget (same configuration pattern as
backpressure object/data-size thresholds). A global default remains in
{{{}nifi.properties{}}}.
h2. NiFi API Changes
* {{VersionedConnection}} gains a field for the per-connection metadata heap
swap budget (data-size string, consistent with backpressure data-size fields).
h2. REST API Changes
* {{ConnectionDTO}} (and related create/update paths) expose the
per-connection metadata heap swap budget.
h2. Framework Changes
* Estimate FlowFile metadata heap size from the FlowFile record structure and
attribute key/value lengths.
* Drive active-queue overflow and swap-file batching from estimated metadata
bytes in addition to FlowFile counts (including clustered queue partitions).
* FlowFile Repository: force checkpoint when the active journal reaches the
configured size threshold, in addition to the existing time-based schedule and
the existing post-recovery checkpoint on startup.
h2. UI Changes
* Create Connection and Edit Connection gain a field for the metadata heap
swap budget (alongside FlowFile expiration and backpressure).
h2. nifi.properties Configuration
* New property for the default per-queue metadata heap swap budget (default
{{{}10 MB{}}}). This supersedes {{nifi.queue.swap.threshold}} as the primary
swap control; the count-based property is removed.
* New property for the maximum FlowFile Repository WAL journal size that
forces a checkpoint.
This set of changes provide significant protection to NiFi's JVM heap, as well
as storage constraints. Additionally, it does so in a way that provides
queue-specific overrides so that if there are specific queues that do need to
support larger heap utilization without swapping, that can be accommodated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
