Joe Witt created NIFI-16322:
-------------------------------
Summary: GenerateFlowFile should stream File Size instead of
allocating a byte array
Key: NIFI-16322
URL: https://issues.apache.org/jira/browse/NIFI-16322
Project: Apache NiFi
Issue Type: Improvement
Reporter: Joe Witt
Assignee: Joe Witt
GenerateFlowFile builds the entire File Size in a heap byte[] before
session.write.
generateData() does asDataSize(DataUnit.B).intValue() then new byte[byteCount].
Double.intValue() saturates at Integer.MAX_VALUE, so a legal value such as 3 GB
is
silently generated as 2,147,483,647 bytes. DATA_SIZE_VALIDATOR has no upper
bound.
The array is then written to the content repository, so the payload exists
twice.
Unique FlowFiles = false builds that array in @OnScheduled and holds it for the
run. Unique FlowFiles = true allocates a new array per FlowFile in the batch.
OutOfMemoryError is uncaught; the processor only has success.
There is no protocol reason for a contiguous buffer. Generate random or text
bytes into a small buffer and stream them through session.write until File Size
bytes are written. Unique = false can reuse a seed or a modest buffer so
FlowFiles
still share the same content without caching a full-size array.
GenerateFlowFile has no @SystemResourceConsideration(MEMORY) today. Do not add
one that claims the whole File Size lives on heap. After streaming, leave the
annotation off.
Expected:
- File Size is written in chunks via session.write; do not hold a byte[] of
File Size.
- Values greater than Integer.MAX_VALUE bytes are invalid or rejected without
allocating.
- Do not truncate 3 GB to 2 GiB - 1.
- Unique FlowFiles and Batch Size behavior stay the same.
- Custom Text is unchanged (File Size is already ignored on that path).
- Small sizes (default 0B, existing tests) still produce the exact length.
Tests (surefire TestGenerateFlowFile, no multi-GB fixture):
- Size within cap produces a FlowFile of that exact length (unique and
non-unique).
- Size above Integer.MAX_VALUE is invalid at configuration or fails without
NegativeArraySizeException / a huge array.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
