[
https://issues.apache.org/jira/browse/NIFI-16200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne resolved NIFI-16200.
-------------------------------
Resolution: Duplicate
> Improve resilience against large/numerous FlowFile attributes
> -------------------------------------------------------------
>
> Key: NIFI-16200
> URL: https://issues.apache.org/jira/browse/NIFI-16200
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
>
> h2. Motivation
> FlowFile attributes are intended to be small metadata (typically on the order
> of hundreds of characters). In practice, flows often place large payloads in
> attributes. Attributes are held in memory on each FlowFile and all attributes
> are rewritten to the FlowFile Repository on every update that includes any
> attribute change. Large attributes therefore increase write cost, inflate the
> FlowFile Repository write-ahead log (WAL) between checkpoints, and can
> exhaust JVM heap before today's count-based swap thresholds engage.
> Today, swap is driven by a globally configured FlowFile count
> ({{{}nifi.queue.swap.threshold{}}}, default 20,000) with a fixed batch size
> of 10,000 FlowFiles written per swap file.
> FlowFile Repository checkpointing is time-based (default 20 seconds). Under
> heavy attribute write load, the active journal can grow large enough to
> exhaust disk before the next scheduled checkpoint.
> As a result, a huge number of attributes, or very large attributes, cause
> issues with both NiFi's JVM heap as well as disk contention contention and
> storage space exhaustion.
> h2. Proposed Solution
> Two related changes:
> # *Size-triggered FlowFile Repository checkpointing.* Keep the existing
> time-based checkpoint interval. Add a configurable maximum size for the
> active WAL journal. When that size is reached, force a checkpoint instead of
> waiting for the interval, then begin a new journal and discard the prior
> snapshot/journals as today.
> # *Per-queue, heap-size-based swapping.* Replace globally configured
> count-based swap thresholds with a per-connection budget for estimated
> in-heap FlowFile metadata (record overhead plus attribute key/value lengths).
> Default global budget: 10 MB of metadata in the active queue before overflow
> begins. Overflow uses the existing secondary in-heap queue; when that queue
> holds about 1 MB of estimated metadata, write a swap file and drop those
> FlowFiles from heap. Each connection may override the budget (same
> configuration pattern as backpressure object/data-size thresholds). A global
> default remains in {{{}nifi.properties{}}}.
> h2. NiFi API Changes
> * {{VersionedConnection}} gains a field for the per-connection metadata heap
> swap budget (data-size string, consistent with backpressure data-size fields).
> h2. REST API Changes
> * {{ConnectionDTO}} (and related create/update paths) expose the
> per-connection metadata heap swap budget.
> h2. Framework Changes
> * Estimate FlowFile metadata heap size from the FlowFile record structure
> and attribute key/value lengths.
> * Drive active-queue overflow and swap-file batching from estimated metadata
> bytes in addition to FlowFile counts (including clustered queue partitions).
> * FlowFile Repository: force checkpoint when the active journal reaches the
> configured size threshold, in addition to the existing time-based schedule
> and the existing post-recovery checkpoint on startup.
> h2. UI Changes
> * Create Connection and Edit Connection gain a field for the metadata heap
> swap budget (alongside FlowFile expiration and backpressure).
> h2. nifi.properties Configuration
> * New property for the default per-queue metadata heap swap budget (default
> {{{}10 MB{}}}). This supersedes {{nifi.queue.swap.threshold}} as the primary
> swap control; the count-based property is removed.
> * New property for the maximum FlowFile Repository WAL journal size that
> forces a checkpoint.
> This set of changes provide significant protection to NiFi's JVM heap, as
> well as storage constraints. Additionally, it does so in a way that provides
> queue-specific overrides so that if there are specific queues that do need to
> support larger heap utilization without swapping, that can be accommodated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
